The world is full of pundits and predictions, across every industry and every walk of life, with analysts constantly debating the merits of different models for everything from manufacturing output to holiday shopping trends.
The energy sector is no different and, in fact, represents one of the most prominent fields for projections, since demand estimates play a crucial role in guiding production. Genscape itself uses a unique methodology to project electricity demand with remarkable accuracy. By taking into account a wide range of meteorological and economic data combined with expert analysis, their forecasts can reach as high as 99.8 percent accurate.
Over years of forecasting, Genscape's analysts have proven their accuracy to a remarkable degree. The company calculates the mean average percent error for each of its forecasts, and has almost universally outperformed the figures forecast by the system operators. As the charts below illustrate, over the May 2012 to October 2012 time period, which includes the critical summer period, Genscape's demand forecast are more accurate than the system operators forecast in every region over every time period except for one case. From May to October 2012, error rates for the current day (known as the Bal Day, hereafter BD) ranged from no more than 2.2 percent for ERCOT to as low as 1.31 percent for MISO. The operators meanwhile never managed lower than 1.72 percent, and ERCOT reached as high as 2.66 percent.
The differences could be just as stark for next day projections. Genscape managed a mean error of only 1.67 percent for CAISO, compared to 2.6 percent from the operator, and only MISO boasted a slight advantage over the company's numbers.
Demand projections in the energy sector primarily help to prevent shortages and oversupply, both major concerns for the power industry in particular. Too little energy can mean a sudden loss of power and too much can damage the infrastructure of the electrical grid. In addition, errors in demand forecasts by grid operators and others can be one of the big drivers creating trading opportunities. When grid operators have high error in their demand forecasts, it can cause large swings in real-time prices and create trading opportunities as grid operators react to quickly bring on-line new power plants or take unnecessary generators off-line.
Each of the various grid operators releases their own estimates each day on how much electricity will be consumed within their service areas, as do a variety of research and analysis firms as well as energy traders. These models generally stem from regressions of meteorological data, primarily relying on recent economic trends and a daily forecast of key weather indicators like temperature and cloud cover.
Taking a different tack
Immediately from here, Genscape's process differs markedly. The company's analysts all work from a baseline raw projection produced by a neural network model that has proven highly effective for more than a decade.
Carl Bomgardner, who developed the earliest version of the model to estimate power demand in New England in 2000, explains that, though Genscape's approach remains unique within the industry, he relied upon a fairly standard approach to neural networks. Carl and his team "train" the model each day, providing a huge array of inputs that the model uses to construct a function through back-error propagation.
These inputs cover many of the same factors covered by other statistical models such as temperature, humidity, cloud cover, day of the week and whether there are any holidays, as well as a number of derived statistics such as heating and cooling degrees. The weather data being used by the model stretch back as far as 1999, providing a huge basis from which to predict future trends. But Genscape's analysts also seek to ensure that the data being used is actually relevant to the current conditions, so the model is trained exclusively on numbers from a period within 45 days in either direction of the forecast date, updating the system each day at 3:00 in the morning.
"But before it does that, it does a backcast," notes Bomgardner. "Which basically, we're writing into our database, now that we have observed weather for yesterday, we run the models against observed weather and then we can look at the difference between what the models predicted using observed weather versus forecasts, for us to be able to do error tracking. So every day we do that, one more day we add to the database. Then it trains the models, and then a few hours later we run the forecast."
A single model, however, could never accomplish the exacting accuracy needed to predict power demand hour-by-hour for days and weeks in advance. So Genscape constructs 72 separate models, one for each hour of the day for the bal day, next day and any days past that.
"The trick to getting that to work," says Bomgardner, "is obviously how do you not have radical jumps from one hour to the next, because you've got separate models.."
Beyond 15 days out the model starts to suffer from limited accuracy in weather forecasts, but even past that time period the approach still allows for the use of climatological data, out as far as 60 days. Each of these forecasts is produced once around 5:00 a.m., with notifications going out to analysts if the weather makes any major deviations from predictions. The analysts can then re-run projections as they want over the course of the day, until the backcast the next day and the process begins again.
Genscape's demand projection team takes this concept and builds on it even more, however, with an unprecedented level of expert analysis of statistical models. Each day, the company's team pulls out a raw projection of power demand similar to those created by the grid operators and other sources.
Where traditional models rely solely on identifying trends from historical data and applying it to current weather, Genscape uses this information primarily as a guideline for experts' personal judgments. The company employs a team of professional meteorologists who make use of a tool referred to as the "nearest weather curve." This application allows the analysts to compare the demand curve projected by their raw model, using only the basic inputs that other forecasters rely upon, against the actual power demand of another, similar weather day.
"We can theoretically pick up any unlimited number of days from 2005 to the day before the current one which, based on the forecast weather that goes into the model, will provide us with a demand profile," explains Pedro Mulero, Genscape's chief meteorologist and the head of the power demand team. "We're going to look back in history and see what was the demand on a day that was similar, from a forecast weather standpoint, to today, tomorrow or any forecast day ahead... Our process is heavily dependent on our own expertise and our own knowledge of the weather and for the day and how that will impact demand," Mulero notes. "We have this flexibility in adjusting any part of the demand curve that we want."
From the very start, using the nearest weather curve is an extremely subjective process. While Genscape's analysts can pick from a wide range of objectively similar days, it remains up to the power team to determine what day best reflects the relevant aspects of the forecast day. That means not only matching temperature, cloud cover, wind patterns and other meteorological phenomena, but a variety of factors completely unrelated to the weather. Weekends are the most obvious example, and one of the simpler problems. But holidays and other less frequent events can have a major impact on power demand, and that can make it hard to find a comparable weather day with the same consumption profile.
Tools for every scale
Despite these challenges, the nearest weather curve offers Genscape's analysts a convenient look into the broad patterns of power demand in each region, allowing them to nudge projections up and down to better fit the type of curve seen in the area.
From there though, it comes down to identifying the small-scale local weather events and translating these features, such as storms, cold fronts, sea breeze, and cloud cover, into changing demand.
Genscape's demand forecasters use a variety of different weather forecasting models, from the Global Forecasting System (GFS) to the European Center for Medium Range Weather Forecasting (ECMWF). But the more granular North American Mesoscale (NAM) model from the National Centers for Environmental Prediction (NCEP) and the High-Resolution Rapid Refresh (HRRR) model from the National Oceanic and Atmospheric Administration (NOAA) are key to pinpointing the peaks and troughs as weather patterns shift throughout the day - details that get entirely missed by grid operators. In the short term, especially for the BD outlook, surface weather observations and satellite imagery offer valuable clues as to why demand may be tracking over/under forecast and what the latest weather trends portend for demand during the next three, six, or 12 hours.
"The power that we have lies in our flexibility, which is maximized when we want to take into account smaller scale features that will impact demand," says Mulero. "If we're looking at a summer day in the Mid-Atlantic and you think there's going to be a good chance of afternoon thunderstorms, say somewhere between 3 p.m. and 5 p.m., right around the peak, so what we will do … we are able to look at a curve and say, 'Wait a minute, there's something that doesn't look quite right here. This peak is going to be much weaker when these storms impact the Philadelphia area, … in fact, the peak may not even happen in the late afternoon.' ... We might have huge discrepancies between us and the [grid operators], but we have the flexibility of picking up two, three, four, five hours when we think there's going to be a big event and just drag the curve downward."
These kinds of updates happen anywhere from two to six times each day depending on how closely actual weather patterns and actual demand are matching up with the earlier forecasts, with the team of meteorologists tracking the progress of their predictions throughout the day. Each morning, they come in and assess how closely their estimates followed actual demand on the previous day.
On the rare occasions when some blip pushed projections off, they sift through data to pin down where things veered off course and outline how the problem arose. Aside from giving customers greater confidence in the projections, this also gives the analysts one more experience to draw on for future estimates.
"It's kind of interesting, because each one of us on the demand forecasting team learns something new almost every day," says Mulero. "That is a very critical part of the process. Demand is very non-linear and there are things that from a weather/non-weather perspective, we must learn from in order to keep building on our success."
Making a power expert out of a meteorologist
And it is by no means a simple task to create meteorological team capable of pulling together the different threads of what goes into power demand to help translate these weather forecasts.
The group that Mulero leads brings together meteorologists with backgrounds ranging from predicting the output of wind farms in the Pacific Northwest to purely weather forecasting. This diversity forces them to learn the intricacies of how the weather patterns they recognize can impact demand in different ways, a skill Mulero notes meteorologists rarely, if ever, encounter in their training.
"We have all kinds of different backgrounds, but at the same time most meteorologists out there, they're not exposed to forecasting demand," explains Mulero. "We forecast temperatures, we look at weather and why things happen the way they do."
To help supplement this wealth of meteorological expertise, Genscape also brings in analysts with a deep wealth of knowledge about the power sector, helping to give the weather forecasting team a filter to pass their projections through before they are finalized. The dialogue between these two groups, as they consider large- and small-scale weather shifts across the country, helps to bring out when, where and how the weather will be impacting demand.
"People with no experience whatsoever in the field, in the business, will have a little bit of a harder time understanding some of the non-weather factors when it comes to driving demand," notes Mulero. "But I think experience really is the key too … and the fact that we're a good team. We discuss demand pretty much every day, every hour. And we try to always analyze why things came in higher or lower than expected. Why did demand, why did the shape of the curve behave in this fashion? It's something that experience has given us a lot of tools to understand and forecast, but at the same time, there are times when we always find something new."
The process of updating demand forecasts on a continual basis, noting throughout the day how the numbers react when the weather inevitably throws a wrinkle into the forecast, gives Genscape a unique kind of experience in predicting power demand.
Separating from the crowd
The trend in many industries has been to increase transparency in how data are gathered, estimates generated and results reported. The unfortunate side effect of this move is the often misguided assumption that all expertise can be broken down into charts and figures to be reported along with projections.
But no model will ever predict power demand with perfect accuracy and reliability, no matter the level of detail included in its inputs or the sophistication of its analysis. Genscape addresses the shortcomings of a purely quantitative approach by giving experts within the field direct influence over its projections.
Starting from the most accurate and responsive models possible provides a reliable baseline. But the involvement of knowledgeable minds capable of more advanced learning and holistic thinking provides a distinction from existing industry projections that proves valuable for anyone with a stake in the movement of power prices.
Meanwhile, the company maintains the concept of transparency as well by providing unprecedented access to the methods and even personnel behind this process. Not only can Genscape promise more accurate estimates, but it can better explain where they come from and how else the numbers could play out.