At risk of stating the obvious, we need to lower emissions, not only from the electric power sector, but also from transportation and other energy-consuming sectors. Agreement on this is evident in the number of states, regions, and countries that are making ambitious commitments to decarbonize. However, having the societal and political will to promote sustainability typically does not account for, or even understand, the significance—the sacrifice implied—by these commitments. The challenges of this transition loom large as we move to quickly replace the legacy generation fleet with renewable resources that are largely dependent on weather patterns. Studies show that installing significant renewable generation capacity is possible, but the associated flexibility required to ensure reliability in a system with significant variable generation is not yet available.
C. Lindsay Anderson
Is there optimism for achieving a successful system evolution? There is no shortage of discussion of the need for flexibility, both within the electric power system and in coupling with other infrastructure systems. Indeed, recent ESIG blog posts have made an excellent case for the value and capability of buildings and other distributed energy resources to provide such flexibility. The demand side of the energy system is going to be a significant provider of flexibility, but leveraging this flexibility will require new modeling, data analysis, and operational models, not to mention willing homeowners and businesses. These new models are one step. We also require extensive knowledge of distributed energy resources’ locations and capabilities, and the development of cooperative decisionmaking across voltage levels in the power system. These requirements bring us to the power and elegance offered by co-optimization solutions.
The Use of Co-optimization to Model Increasingly Complex Power Systems
Judy Cardell
Co-optimization seeks to make decisions that are collectively optimal for coupled systems with distinct objectives. In the traditional power grid, the active, responsive, and controllable resources were few in number and centrally located at the high-voltage level, and traditional decision-making focused on these few, centralized generators. In our emerging sustainable system, active resources are numerous and distributed across voltage levels, and only some are responsive and controllable. Decisionmaking for this new system must account for both the capabilities and limitations of all these resources, yet traditional decision-making tools are not up to the task. But co-optimization is. Co-optimization effectively models and analyzes the entirety of this complex system, and lets operators share resources and capabilities across systems.
Is co-optimization a new tool for power systems? Not entirely. During the ESIG spring technical conference in 2021, a panel discussion highlighted co-optimization research. The panel “Co-optimization Techniques and Results included three speakers using co-simulation and co-optimization of integrated transmission and distribution systems, electric power and gas networks, and coupled planning and operations. Consensus is that we are not yet capable of detailed and dynamic system-wide co-optimization, but this is a potentially game-changing objective.
Currently, co-optimization is used in electric power markets for providing both energy and ancillary services. But now we are faced with an increasing need for making interdependent operational decisions. Widespread integration of new distributed energy resources can benefit both transmission and distribution operations, but these benefits will be maximized when they are fully coordinated across systems.
A Co-optimization Example
Consider the bi-level optimization of transmission and distribution, illustrated in the figure below. Historically, the main focus of distribution system operations has been to maintain voltage across the system as power is delivered to the end user, while the transmission system dispatches resources to provide the requisite power to each connected distribution system, at minimum cost. In the future when end users are capable and incentivized to respond to dynamic needs of the bulk power system, coordinated operations will maximize the efficacy of resources across the interconnected systems. Transmission operators will consider distribution system capabilities in unit commitment, dispatch, and reserve allocations. Distribution operators will consider the variability of renewables to plan operations and correctly value the services they can offer. This future will be facilitated by the use of co-optimization in the decision-making tools.
Figure: Stylized example of bi-level optimization of transmission and distribution for enhanced sharing of resources and flexibility.
Source: Cornell University
Challenges of Co-optimization
The challenge of co-optimization lies in the increasing number of decision variables, leading to computational complexity that becomes intractable on networks of realistic scale. Of course, one solution is more computing power, and certainly we can do more with clusters and parallelization. However, parallelization has limited efficacy on problems that are truly co-optimized.
Counter to these challenges, the benefits of co-optimization are becoming more widely recognized, and continuing advances have been made in algorithmic development, for example:
- Simulation-based optimization, which builds the physics of the underlying system into a simulation, within a heuristic-based optimization framework.
- Decomposition with iterative trading of information, to solve the systems separately with iterative feedback. This is a hybrid method that maintains some efficiency of independent problems while still representing the co-dependence of coordinated optimization.
Computational capability is not the only barrier to co-optimization. In the case of co-optimized transmission and distribution, regulations and markets play a significant role. Regulatory separation of systems means that transmission system operators do not currently have detailed awareness of distribution system conditions and decisions. Conversely, distribution systems (and demand response aggregators) face challenges in market participation. This means that for co-optimization to move out of academia and into industry operations, there must also be willing cooperation between transmission and distribution system participants.
The Way Forward
Significant and effective integration of renewable energy resources will be required for a sustainable energy system. Successful mitigation of uncertainty from renewables will be achieved by leveraging flexible loads and other distributed energy resources. Effective coordination of these resources can only be achieved with co-optimization within operations, supported by changes in regulations and markets.
We need to continually evolve new algorithms that can solve these problems on large-scale networks, with a massive number of variables, in reasonable computational time without neglecting the underlying system physics. The key will be hybrid algorithms that combine the physical reality of networks with the structural flexibility of heuristic and artificial intelligence methods. Development and validation of these algorithms will also require massive stores of data not currently available to (most) researchers. But the only way to make the most effective use of both supply and demand side resources is for these resources to be actively coordinated, and this will require the use of co-optimization. This is a significant research adventure, and when solved, will have a lasting impact on grid operations.
C. Lindsay Anderson, Cornell University, and Judy Cardell, Smith College
I agree that co-optimization is an important goal for industry practice but the current challenges are somewhat more prosaic. For many co-optimization challenges, we already have algorithms and tools that support practical deployment (e.g., gas-electric co-optimization, generation and transmission co-optimization, co-optimization of power with heat systems and hydro power, and so forth), but these are no yet in use. A bigger problem remains in the sharing of data between organizations. From the perspective of a power-system operator, it is still difficult to have access to the information on gas-pipeline systems, distribution systems, and other energy systems that have significant impacts on the reliability and economics of electrical power. In addition, change is hard, often motivated by crisis, and not sufficiently driven by appropriate planning. And then we have the challenge of who gets paid for what.
In broad terms, many of the challenges are rooted in regulatory and policy limitations. There are some obvious solutions (e.g., cost on carbon) that would make this easier. But even then there are thorny problems such as retail rate design, transmission right of way, nuclear licensing, … which get treated as third rails. As engineers, we can (hopefully) have an impact by showing the consequences of alternative policies, such as through accurate assessments of system reliability and cost/benefit.
Great blog post…thank you for this informative and interesting article.
This is a great place to point out to readers that co-optimization can also be applied at the planning level in the choice of transmission and renewable resource placement and type and this can potentially have a large impact on reducing VER uncertainty making the operational co-optimization problem simpler. The current methodology of using LCOE to place renewable resources will not work unless long term storage can be cost effectively deployed. Instead, renewable resource placement should consider the value of the resource from the perspective of providing energy when it is needed, versus providing as many MWh as cheaply as possible. Such an approach would open up opportunities to deploy renewables in places where capacity factors currently would be considered as untenable; there are plenty of places where there is strong correlation between wind and/or solar resource availability and periods of high load that are not being currently considered because spatial and temporal co-optimization of load, transmission and renewables is not currently done. Our sector needs to understand that most of the drivers of variability and uncertainty in load and renewables resources weather driven and follow well defined patterns. For instance, regions in the west experience high summer and winter loads, when downslope and gap winds come from the east leaving current wind farms east of the barriers generating almost no power, while robust generation is possible west of the same barrier but this resource is ignored because annual capacity factors are too low (for more see blog on this same forum: https://www.esig.energy/blog-crucial-need-incorporate-meteorology-renewable-energy-transition-part-1/). And because optimization during planning needs to be done infrequently, the problem is more computationally tractable.
Without attempting to oversimplify a complex issue, enabling autonomy on the part of energy consumers could provide some relief. Deployment of appliance control and metering technologies has demonstrated that consumer energy decisions can be shifted to a risk informed cost decision model. The present centralized planning urgency could be relieved by more affordable consumer storage.
Renewables are disaggregating our generation from a centralized “command and control” paradigm, and in the case of the recently evaluated ERCOT events of May and June are identifying gaps in our existing system models that make migration to more complex dynamic models problematic. As an educator it is increasingly difficult to attract students to a practice area that requires sophistication with higher level computing resources than are available to my (and many other) institutions.