The Solar Forecast Arbiter

Renewable energy forecasts are important tools for utilities and system operators to predict power output and total production for weather-driven generation sources such as solar and wind. However, validation and analysis of these forecasts are typically done using ad hoc methods built by either forecast providers or end users, and are often proprietary, not transparent, and not standardized. This has resulted in non-repeatable results and disputes over validity. It has also led to a focus on bulk metrics, which, while easy to track, provide only limited information about a forecast’s usefulness, as they represent average values over a longer period and fail to capture specific values during shorter, discrete events—whose prediction is of particular value to the end user. As discussed in the blog post, “Reflections on 10 years of ESIG Forecasting Workshops,” this inability to specifically forecast events such as the ramping up or down of solar or wind generation makes it difficult to quantify improvements in forecasting or compare among forecasts performed by different entities over time.

The Solar Forecast Arbiter (SFA) changes this, providing a standardized, open source, extensible platform with a diverse suite of metrics that can be tailored to different needs. Below we outline the SFA’s features and invite you to log in, explore, and offer your feedback. You can register for a free, no-obligation account here. We are also looking for end users who want to conduct an operational forecast trial that we hope can start soon after this post is published in Fall 2020—please see below.

An Open Source Evaluation Framework for Solar Forecasting

The SFA provides solar forecast providers and end users with a standardized and objective platform that can be used to evaluate and compare forecasts. Users will be able to assess multiple vendor forecasts, make comparisons among their own forecasts, and track changes in accuracy. For forecast providers, this standardization will allow the use of a single framework for all trial forecasts, removing a need to rebuild the system based on the requests of each potential customer.

Funded as part of the DOE Solar Forecasting 2 program (but built with extensibility to wind forecasting also in mind), the SFA is an open source tool that enables evaluations of solar irradiance, solar power, and net load forecasts that are impartial, repeatable, and auditable. The SFA is geared toward evaluating the quality and value of deterministic, probabilistic, and event forecast products of solar power and irradiance, and includes automatable analysis of multiple operational forecasts and management of multi-vendor operational forecast trials. The SFA assesses a renewable energy forecast and provides an objective score using a broad suite of metrics, and provides end-to-end functionality for all aspects of end-user forecast validation.

See figure 1 and a short movie providing additional details.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 1. Overview of SFA features and functionality, with example output.

Data Management

The heart of the SFA is a database and data rights management system that allows users to define and store site metadata, site observations, and site forecasts in a secure and private way. For example, a solar plant owner or off-taker may define plant metadata (e.g., location, tilt, and DC and AC capacity), define measurement metadata (e.g., type of variable, interval label, interval length), and upload measurement data in CSV or JSON format. They can share access to these data with others inside their organization, with partner organizations, and with forecast vendors. Forecast vendors, in turn, may download the quality-controlled measurement data and share forecasts with the plant owner. Each data owner retains full control to limit access or remove data at any time.

A data use agreement governs how data may and may not be used on the SFA, and participating organizations must sign the agreement to access the tool’s full functionality. This legal agreement and our data policies provide generous protections to data owners while limiting access by SFA administrators to that needed for technical troubleshooting.

Validation and Analysis

The standardization of trials as done by the SFA has tremendous value to vendors and end users, saving time and effort and providing unbiased results. The SFA also adds value to solar forecasting through its much broader suite of metrics than those typically calculated during trials, including a cost metric.

Analysis reports can be produced for deterministic, probabilistic, or event-based forecasts. The user selects date ranges, metrics, time categories (e.g., total, day, hour), and quality control flags, and indicates how missing or invalid data are to be handled. The reports produced are web-based with interactive graphics and may be downloaded in HTML or PDF format. (The link above opens a report in the SFA and requires one to log in first. An option to create an account is provided at the bottom of the login window or one can follow the instructions here.)  All of the SFA functionality, including data upload and report generation, can be automated using the RESTful API.

To ensure clarity around the desired results of a multi-vendor operational forecast trial, the utility and the forecast vendors must agree, prior to beginning the trial, on what exactly will be evaluated and how. Multi-vendor operational forecast trials—one type of the SFA’s real-time trials—have all of the constraints of an operational system. This includes handling the possibility of observational data outages (using options that can be selected by the user) and forecast delivery delays (by requiring forecasts to be delivered by specific cut-off times and allowing the user to select options for handling forecasts that are missing or late). Multi-vendor trials also allow anonymization of the forecast providers.

Since forecast validations are only as good as the quality of the ground truth data that are used, the SFA provides quality control of data uploaded to it. Figure 2 shows an example of quality-controlled observation data. (Clicking on the figure opens the observation page in the SFA and requires a account.)

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 2: Example of the SFA’s interface for selecting and displaying observation data, including quality-control flags.

SFA trials may include reference forecasts, supplied by either the SFA or the end user. Reference forecasts provided by the SFA are based on easily implemented, freely available forecasting solutions and use public reference data from hundreds of solar irradiance, power, and net-load measurements. These reference forecasts allow end users to compare vendors’ forecast trials to forecasts based on freely available forecasting solutions, helping end users to determine the degree of value added in the proprietary forecasts.

Invitation for Community Engagement

The SFA represents a milestone for improving renewable energy forecasts and is ready for you to use. There are several ways that you can get involved:

• The public observations and reference forecasts provide a way to explore the SFA without any need to sign the data use agreement. Once you have created an account, you may access the live SFA user interface and begin by exploring the detailed “Getting Started” documentation.
• You may also join the Stakeholder Committee email list, which will keep you apprised of project development and provide you with opportunities to inform it. The origins of the Stakeholder Committee date to 2018, when a diverse group of people from the forecast user and provider communities met following the 2018 ESIG Forecasting Workshop and produced user stories describing the desired uses of a forecast validation platform. These stories were synthesized into use cases for implementation, and the group has continued through the development phase of the SFA.
• All of the software is implemented in open source packages available on GitHub, and the community is encouraged to view, audit, and improve the code.

Lastly, and importantly, we are looking for end users interested in conducting an operational forecast trial that could start soon. Many forecast vendors have indicated their support and readiness to utilize the SFA for such a trial. Early adopters will receive the benefit of extra guidance from the developers. If you are interested in potentially participating, please send an email to info@solarforecastarbiter.org.

Acknowledgments

The SFA is funded by the U.S. Department of Energy Solar Energy Technologies Office under award number DE-EE0008214.

The SFA team thanks all of the stakeholders that have contributed ideas and reviews to the project.

 

Justin Sharp, Principal, Sharply Focused
Will Holmgren, Assistant Research Professor, University of Arizona