Travis Whalen
History and folklore abound with stories of oracles and prognosticators that shaped the course of civilization. The cryptic words of the Oracle at Delphi sent Greek kings on to both legendary conquests and ignominious defeats, and one 16th century monk has parts of the Catholic world expecting the new Pope to be the very last. Today’s prophets in the utility sector set their sights a bit lower, content to predict how quickly customers might adopt electric vehicles and when annual peaks might shift from summer to winter. But have the standards and expectations for the modern prophets changed that much with the times? In many cases before regulators today, the challenge in load forecasting is as much about educating the public about what forecasts are intended to accomplish as it is about describing the forecasts themselves.
Classical View of Accuracy
As often as not, ancient prophets’ predictions were ambiguous enough that they’d find a skeptical audience in most public utilities commissions. And not many regulators will accept tea leaves as admissible filings regardless. But for veteran load forecasters in the utility sector, the metrics upon which their work has been judged would be familiar to anyone with a grounding in statistics: R squared, mean squared error, mean absolute percent error, and similar metrics make regular appearances in rate cases and similar filings.
At their core, these metrics are more about the past than the future. They explain the strength of model drivers’ historical relationships to power or gas consumption. The reliability of any forecast judged on these metrics hinges on the stability of those dynamics, as well as the reliability of the forecast of those underlying drivers.
None of these constraints make these bad metrics to judge a forecast, which is of course why they represent the standard. In an environment of stable market dynamics, a solid set of historical relationships set a reasonable baseline for setting rates; planning distribution, transmission, and generation investments; and all the other long-term planning functions that utilities require. As those relationships change, however, these distinct needs can become increasingly divorced from each other. And forecasting for each purpose should likely follow suit.
Failure in Accuracy and Success in Error
While all manner of ancient heroes may misunderstand their prophecies, the assumption is usually that these predictions are either right or they’re wrong in the end. But load forecasters can easily find themselves in a situation where their forecast was, by most metrics, correct, and nonetheless fail to meet their needs.
As a scenario, let’s say that a region is rapidly electrifying its heating. This regional electrification results in a significant increase in sales, a shift to winter peaking, and increased need for thermal generation available in evening and morning hours. A conventional forecast that layers in the increased load from electric heating and correctly predicts the level of technological adoption might correctly predict the impact on sales and the system’s aggregate load shape. It might even accurately capture the collective impact across the system’s distribution system. Such an outcome would typically be regarded as highly successful.
Assume, however, that this scenario takes an older approach toward distribution forecasting and applies a broad strokes allocation of loads spread across the entire distribution system. Even correctly shaped and accurate in aggregate, an inappropriately wide spread of load additions could lead planners to expect far fewer constraints on the system than might ultimately arise, because load additions would be smoothly distributed rather than clustered, as can often occur with new technology adoption. Then, despite a highly accurate forecast on multiple levels, the utility finds itself having planned for far less investment than required and experiences severe constraints on some feeders, leading to unexpected risks to the system.
From the perspective of the distribution planners, a less accurate peak forecast that better anticipated the scale and location of potential constraints would have been more useful. Given long lead times within electrical equipment supply chains and the need to secure land for future development, even substantial inaccuracies in overall levels can be preferable if more directional guidance proves more accurate.
The Problem with Planning to be Right
Of course, the above scenario only presents a problem because it assumes a forecast that fails on a more challenging measure of accuracy. Presuming a forecast that meets the needs of distribution planning just as well as generation planning resolves that concern. Yet even then, for many utilities this more flexible forecast presents new problems—antithetical risks posed to gas and electrical systems, when risks to one system are directly opposed to the risks on the other.
Across both gas and electrical utilities, planning has always built upon inherently conservative assumptions intended to ensure that the lights stay on and houses stay warm—from planning to be able to provide gas in outlier low temperatures to accounting for possibly aggressive adoption rates for electric vehicles. In a status quo forecasting environment, any differences in those assumptions need never come in conflict. But as the industry begins to grapple with the growing pressure to electrify more gas appliances and equipment, those two distinct worlds are forced to play a game of tug of war for limited capital resources. In either case, the consequence of misallocating between the needs of the gas and electric systems can be disastrous, potentially putting reliability at risk during the periods of greatest need.
Given the established practice in many utilities of planning systems around the most extreme weather over the prior 30 years, it could be just as plausible to assume a 1-in-30 outcome for the pace of electrification from either perspective. That could mean dramatically lower adoption assumed in gas planning and dramatically higher adoption assumed in electric planning. That’s far more conservative than many regulators might want to pay to support, and yet targeting planning on the presumption of an accurate forecast puts the risks of market changes entirely at odds with the risks assumed from weather and other more conventional factors. In essence, these changes to the market put utilities’ own definitions of accuracy at odds with each other and force a consideration of that core question of what accuracy means directly within each filing.
Crystal Balls over Chicken Bones
How then can a forecast meet all these competing requirements, often while expecting a level of consistency between them that may not actually be possible or preferable? One effort being investigated by Xcel’s Integrated System Planning teams is a tool intended to pull more of these disparate forecasting efforts within a single framework.
Old school forecasting processes often entail multiple handoffs, multiple teams managing assumptions across platforms, and at times the rough adaptation of processes intended for a specific application to serve a more general (or differently specific) application. The more of these applications that can be housed under a single roof, the more reliably different views of a forecast can be produced with common assumptions or, where differences are required, those differences can be clearly and explicitly tracked.
Building this type of platform requires a wide variety of tools and models, but above all it requires a reconciliation of data and assumptions. Common understandings of grid topology, load shapes, and normal weather calculations can all change substantially depending on need, but too often these disconnects grow in isolation beyond any actual requirements. So long as any differences in true requirements are documented and stored within the same forecasting process, their incorporation becomes an intentional insight rather than a sloppy oversight.
Forgive the strained fortune-telling metaphor, but the aim should be a crystal ball that each group can look at from their own angle, yet still see everything going on inside.
It’s About the Journey, Not the Destination
Like every great prophet, today’s forecasters will always be judged by whether they were right or wrong. Hopefully with as much leeway as the likes of Nostradamus, but it’s best not to be overly optimistic. Instead, the key will increasingly be to proactively define what “right” means in a given context and where it’s reasonable to be wrong. But pulling the spotlight away from the simpler measures of outcome doesn’t mean the process gets easier. Instead, the burden simply shifts to shining the flashlight more closely on the grittier details of process. Better explanations of why and how a forecast come to be may lack the mystique of the mathematical oracle, but they should provide a better foundation of trust in a changing world.
Travis Whalen
Manager of Load Forecasting
Xcel Energy
Leave a Reply