hello my name is vanessa leguin i am a phd student between edf the french electricity company and the cnarm supervised by professor nicolato i'm glad to present our work on solar irradiance forecasting with fisheye images at the cvpr omnicv workshop photovoltaic power generation is a promising source of vulnerable energy with std growth however its intermittent nature is still a challenge for its integration at scale in the electricity grid accurate solar energy forecasts are thus crucial we present here the different tools for producing solar energy forecasts with respect to the forecasting horizon for their head and longer horizons which additionally rely on weather forecasts and historical models for intraday satellite images are used and more recently for very short turns horizon and at local scale a lot of rocks are focused on fisher images pointed throughout the sky to extrapolate the future cloud motion edf has conducted a meteorological campaign on la reno island fisher images have collected images every 10 seconds at the same time pyranometers have measured the components of solar irrigence the dataset collected consists in several millions images labeled by the solar irradiance the machine learning problem here consists in estimating and forecasting solar eugenes based on cameras only since cameras are much cheaper than high grade pyranometers the traditional forecasting method at egf consists in the following steps first fisheye cameras calibration with calibration patterns and the ochem calip toolbox second projection of the hemispherical images on a plane at a given altitude then we compute the optical flow between two images to get the cloud motion and finally we compute the future image by warping this motion into the future the future imagines is obtained with a segmentation method on the image and standard machine learning techniques in this context deep learning is an appealing solution for learning directly from large data sets of raw fisheye images previous works have shown that deep neural networks can achieve promising results for both the irrigant estimation and the reagents forecasting task with conv lstms still extrapolating cloud motion is a very hard weather forecasting problem in this work we incorporate prior physical knowledge on the cloud motion to regularize deep neural networks our model for solar energy forecasting is based on a deep video prediction model that incorporates physical knowledge we present this feed net model at the cppr conference the main idea is to leverage physical dynamics with learned partial differential equations but since physics is often insufficient to fully describe generic videos as in this moving yeminis example we learned a data-driven residual branch responsible for capturing complementary information in this example physics captures cost-moving segmentation masks and the residual captures the fine appearance of digits fitnet is a two branch architecture that is entangled physical from residual dynamics in latent space physics islam with a recurrent neural network cell called fissile whereas a complement is learned by a data-driven conv lstm when unfolded in time this forms a sequence to sequence architecture for video prediction physical cell is called physical it is an atomic cell for building physically constrained prediction sea cell learns species dynamics in latent space with a differential operators approximated with convolutions this cell follows a two-step scheme first a normal physical prediction step in nation space and second a correction step with input images this scheme can be seen as a kind of data assimilation technique now we present our contribution to the solar elegance forecasting problem we propose fitness jewel it is a slightly improved version of feed net with specific encoders and decoders for both branches we use this model to encode the sequence of fisheye images up to tide t into a context the vector c this context this vector summarizes all the previous seconds of images we then plug a cnn to predict the five-minute edge for the image and mlp for the feature irregular value we obtain better result with when predicting 20 the future image and the irregions than the versions only due to the improved supervision we show here qualitative result on the left figure with a five minute prediction on a particular day we we can see that our model accurately predicts the sharp variation of irritants on time and then on the right you can see the table that show qualitative quantitative results with respect to competitive baselines a stm a thread rnn and fitnet we see that our financial model which is the best performances with respect to all baseline both in terms of image prediction and in terms of emergence prediction we also see here some qualitative video prediction results the model takes five minutes of best images and predicts the five the five following images we can see that this problem is very challenging because in five minutes there is a large cloud motion the two clouds that are depicted in blue and green are moving closer and then finally merging in the into the yellow cloud and even if the feed natural predictions are a little bit blurry we observe that the model is able to predict this behavior so it's confirmed the benefits of critically constrained prediction and here we can also we can also see that the the financial model projects much sharper images than the baseline lst as perspective after this work we could use for example more specific physical models by modeling more finely the physical atmospherical phenomena extension to probabilistic forecasting could also be useful for assisting electricity grid manager with uncertainty estimates and finally a future direction is to consider the cooperation between a network or fisheye cameras over a small territory thank you for your attention