May 2023: ISSUE 136
ELECTRICITY MARKET DESIGN IN THE NEW JAZZ AGE: THE DECENTRALIZED
COORDINATION OF FLEXIBLE DEMAND IN A RENEWABLES-DOMINATED POWER
SYSTEM
Michael Hogan
The power grid has traditionally been built and operated like a symphony orchestra, with a mix of supply-side resources
centrally assembled and directed by the system operator to meet demand. There is now growing recognition of the need to tap
into the latent flexibility of demand if we are to advance both electricity decarbonization and the electrification of energy services
in an affordable manner. The value of doing so will lie predominantly in avoiding uneconomic investment in higher-cost supply-
side alternatives.
The great majority of this low-cost, high-value potential will not lend itself to the old symphony orchestra model—the grid will
need to function more like a jazz ensemble, with different players improvising in response as less ‘directable’ supply ebbs and
flows. If flexible loads, especially large, inherently flexible new electrified end-uses like transport and heat, are encouraged to
consume electricity without regard for the impact of the timing of their consumption on marginal costs, the result will be greatly
inflated costs for all and significant lost opportunity. It is therefore necessary that any proposed market design be effective, day
in and day out, in eliciting a harmonious interplay between flexible loads and variable supply. The single pay-as-cleared,
marginal-cost-pricing wholesale energy market paradigm is uniquely well suited to inform the distributed, decentralized
coordination required for this to happen.
Be careful what you wish for
The European energy crisis has prompted a surge in critical attention to electricity market design. There is nothing new about
claims that the wholesale market design paradigm is unfit for purpose, and there are certainly issues to be addressed, but this
latest eruption is especially poorly grounded. The pain caused by the crisis is all too real. But blaming the electricity market
design for it is like blaming organic chemistry for climate change. The wholesale electricity market is simply telling us—all of us,
not just policymakers—what we very much need to know: the cost, considering our current and future sources of primary
energy, of extending our current level and patterns of consumption, or the savings to be had from consuming less and
consuming differently. It is also telling us, in unmistakable terms, how urgent it is that we ween ourselves from our dependency
on fossil fuels.
Mitigating the equity concerns posed by these very real costs is the purview of retail tariff regulation and social policy, not
wholesale market design. The parallel electricity transformations currently underway—rapid expansion of variable supply and
rapid electrification of energy services such as transport and heat—make access to that information for all who can act on it
more vital to the future of energy than it has ever been.
May 2023: ISSUE 136
In other words, any discussion of electricity market design must consider the central role flexible demand can and should play in
mitigating the enormous investment challenge we face. Proposals to ‘redesign the electricity market’ come and go, sometimes
intriguing, but they are nearly always focused on addressing the myriad challenges presented by these transitions with top-
down, supply-side prescriptions (generation, transmission, distribution), including huge investments in marginal assets that will
be little used. Very few address—or even acknowledge—the vital role to be played by an active and responsive demand side.
The importance of demand flexibility
A recent study published by the Pacific Northwest National Laboratory (PNNL) offers a glimpse of the costs of a myopically
supply-side approach, and the potential for flexible demand to substantially mitigate those costs. 1 The study modelled a stand-
alone grid roughly the size of the Texas market under ‘moderate’ (15 per cent of annual energy) and ‘high’ (40 per cent of
annual energy) variable resource penetration. It then compared wholesale system costs in each scenario under business as
usual (no change in demand patterns) and with distributed customer-controlled load flexibility in response to supply variation.
Under the ‘moderate’ scenario, which excluded electric vehicle (EV) loads, the study found bill reductions of 10 per cent for all
customers and 17 per cent for participating flexible customers. Savings were roughly 40 per cent greater in the ‘high’ scenario
(which assumed 33 per cent penetration of EVs). The study did not examine the potential reductions in needed expansion of the
distribution network, which typically represents roughly a third of retail bills. (This is being examined in a yet-to-be-published
PNNL study.) Consider that medium-term targets for both renewables and electrification are already higher in many countries
than the assumptions in the ‘high’ scenario.
The figure below (the ‘moderate’ scenario) shows that these cost reductions derive overwhelmingly from reductions in capital
costs. This results from a combination of more conventional reductions in peak load of 9–15 per cent with more transformational
reductions in daily load variation of 20–44 per cent, together significantly reducing the need for infrastructure investment (again,
excluding potential savings in distribution network expansion). Whilst many system operators focus principally on post-gate-
closure contingency services (which contribute less than 1 per cent of projected savings), the bulk of projected savings come
from increased asset utilization due to pre-gate-closure flexibility in response to expected variations in system conditions.
Figure 1: Potential cost savings from demand responding to prices, based on analysis of the amount and sources of
economic benefit to consumers on a hypothetical Texas-sized grid from deploying flexible loads in response to system
needs.
Note: Bars circled in red are investment cost savings. ‘Capacity payments’ is a proxy for generation investment. ‘Distribution hardware’
considers only substation upgrades. ‘O&M’ is operation and maintenance.
Source: Pacific Northwest National Laboratory.
1
Reeve, H., Widergren, S., Pratt, R., Bhattarai, B., Hanif, S., Bender, S., Hardy, T., and Pelton, M. (2022), The Distribution System Operator
with Transactive Study, Richland, WA: Pacific Northwest National Laboratory.
The views expressed here are those of the authors. They do not represent the views of the Oxford Institute for Energy Studies or any
of its Members nor the position of the present or previous employer, or funding body, of any of the authors . 2
May 2023: ISSUE 136
A small share of these savings can be accessed by enabling demand participation in wholesale markets (energy, ancillary
services, capacity) in competition with generation. However, most valuable daily demand flexibility cannot or will not participate
directly in these markets.2 As the PNNL study concludes, ‘there is a need for a solution that integrates the coordination of
demand flexibility into everyday grid operation, ensures it is automated, puts the customer in control of how much or little they
participate, and fairly compensates them for the level of flexibility they provide to the grid.’3
Moving on, keeping up
Many—especially incumbent industry players—dismiss the idea of a responsive demand side based on the presumption that
demand is nearly perfectly inelastic and cannot or should not be rationed with pricing. That presumption is increasingly
anachronistic.
Certainly, some loads are less elastic than others, and some end uses are more suited to adopting and responding to dynamic
pricing than others. But the fact is that the ‘value of lost load’ (VoLL) for many end-use services, especially large new transport
and heat loads, varies widely across time and location. Whilst the VoLL to interrupt lighting may be quite high, the VoLL to
interrupt EV charging and shift it to a different time is usually very low. Indeed, at, say, 02:00 it is, in nearly all cases, effectively
zero. Even across a more traditional suite of end uses, the potential for end users to respond is quite significant. 4
The presumption also ignores the rapidly decreasing cost, increasing ease, and increasing value of bidirectional demand
flexibility as we expand reliance on variable, capital-intensive and low-variable-cost production. Finally, it ignores clear evidence
that flexible loads will, in fact, respond to pricing if it fully values the real-time flexibility they can provide, and that they can
sustain that responsiveness indefinitely.
One recent demonstration of this new flexible demand reality is National Grid ESO’s (Electricity System Operator’s) CrowdFlex
project.5 Moving from flat tariffs to a set of time-of-use retail pricing offerings for a large sample of Octopus Energy and Ohme
customers over a six-month trial elicited load shifting away from high-price hours to low-price hours of 19–23 per cent among
EV-owning customers and 12 per cent among non-EV-owning customers. Those levels of responsiveness persisted unabated
for the duration of the trial. A more traditional option was also trialled with a large sample of customers, requiring customers to
enrol in advance for a one-time increase or decrease in demand over a two-hour period of ESO’s choosing. This elicited a much
larger response—a downturn of 41 per cent (non-EV owners) to 59 per cent (EV owners) and an upturn of 131 per cent (non-
EV) to 617 per cent (EV)—but one that is unlikely to be sustainable over any significant number of events. (The upturn results
point to the potential for flexible demand not only to reduce peak prices, but to lift market clearing prices during periods of high
renewable production, an important part of a much longer response to the highly contingent claims about the impact low-
variable-cost resources will have on prices under the current market design paradigm.)
A noteworthy aspect of the CrowdFlex results is the difference they illustrate between a traditional conception of demand
flexibility—a centrally directed imitation of a supply resource, which for reasons both administrative and behavioural is highly
limited in application—and the conception of demand flexibility that will be most valuable to a decarbonizing and electrifying
energy system—a highly decentralized, price-driven behavioural change by consumers acting independently, most likely via
automation or via an energy service provider, on a daily basis over an extended period of time. The difference in results may
reflect that on any given day the customer loads responding to time-of-use pricing form a shifting subset of those responding to
a one-off event, with the combination of diversity and agency contributing to a lower but sustainable and still very significant shift
in the pattern of demand.
There is long experience and a level of comfort in some regions with the former model of command-and-control demand
flexibility. In contrast, the electricity industry has traditionally dismissed the potential for the latter model of decentralized,
repeatable shifts in load curves by customers happy to be regularly flexible if the price is right. Initiatives like CrowdFlex are
demonstrating just how out-of-date this traditional viewpoint is. In doing so, they shine a bright light on the necessary (though
not sufficient) role of an underlying wholesale market price that reflects real short-term variation in the marginal cost of
2
Hogan, M. (2022), Tapping the Mother Lode: Employing Price-Responsive Demand to Reduce the Investment Challenge, Reston, VA: Energy
Systems Integration Group, Retail Pricing Task Force.
3
Reeve et al., (2022), The Distribution System Operator, Executive Summary page 3
4
See, for example, Alstone, P., Potter, J., Piette, M.A., Schwartz, P., Berger, M., Dunn, L., Smith, S., Sohn, M., Aghajanzadeh, A., Stensson, S.,
Szinai, J., and Walter, T. (2017), 2025 California Demand Response Potential Study—Final Report on Phase 2 Results, Berkeley, CA:
Lawrence Berkeley National Laboratory.
5
CrowdFlex—Phase 1 Report, 2021, Gallows Hill, Warwick, UK: National Grid ESO.
The views expressed here are those of the authors. They do not represent the views of the Oxford Institute for Energy Studies or any
of its Members nor the position of the present or previous employer, or funding body, of any of the authors . 3
May 2023: ISSUE 136
consuming electricity.6 Lest one be tempted to fall back on tired evasions like customers being uninterested in watching their
meter or in being directly exposed to wholesale prices, projects like CrowdFlex and many others showcase the innovation taking
place in offering consumers flexible energy management services that are convenient and financially compelling.
The deeply ingrained supply-side bias that leads to discounting of distributed demand flexibility in resource planning is one of
several contextual factors that create hidden barriers to a more active demand side, turning scepticism into a self-fulfilling
prophecy.7 Others include out-of-market ‘capacity remuneration mechanisms’ that over-procure the supply-side resources
capable of participating in them by implicitly valuing marginal capacity at levels many times higher than the value available to
energy market participants. These contextual barriers are addressable if the goal is to make the current market paradigm fit for
the future. In contrast, top-down, supply-side-focused market redesign proposals are only likely to exacerbate and further
institutionalize these impediments to an affordable energy system transformation.
A simple illustration
Consider a simplified example of the social welfare implications of neglecting development of distributed demand flexibility and
the short-term wholesale market pricing needed to inform it. The figures below depict a hypothetical system during a given
pricing interval, under two sets of assumptions. In this example the demand for energy has risen beyond the point where the
demand for reserves can be met by re-dispatch and is approaching the point at which the system operator would choose to
curtail firm load to maintain a minimum safe reserve level.
The first figure includes the familiar depiction of demand as nearly completely inelastic. It also depicts a supply curve that does
not fully incorporate the opportunity cost of being short of the desired level of reserves. The second figure improves both curves;
in this figure, the demand curve reflects the fact that while demand is relatively inelastic across a lower range of prices, the
higher the price rises, the more elastic demand becomes. The supply curve in this figure reflects an administrative intervention
(co-optimization of energy and reserves and/or an administrative reserve shortage price adder) to ensure the market price
reflects the full marginal cost of energy, including both the marginal generator’s variable cost of production and the opportunity
cost of further reducing reserve levels. As a result, the market clears at a lower level of demand, at a higher clearing price, and
with a gain in social welfare during this one interval of €387,780. The gain, of course, is the result of not consuming more
electricity in this interval at a cost increasingly exceeding its value.
Figure 2: Conventional supply and demand curves, with nearly complete demand inelasticity and supply that does not
fully account for the opportunity cost of a reserve shortfall
Source: Regulatory Assistance Project.
6
For an extended treatment of barriers and remedies, see Yule-Bennett, S., and Sunderland, L. (2022), ‘The joy of flex: Embracing household
demand-side flexibility as a power system resource for Europe’, Montpelier, Vermont: Regulatory Assistance Project.
7
For a more detailed discussion of contextual barriers, see Hogan (2022), Tapping the Mother Lode.
The views expressed here are those of the authors. They do not represent the views of the Oxford Institute for Energy Studies or any
of its Members nor the position of the present or previous employer, or funding body, of any of the authors . 4
May 2023: ISSUE 136
Figure 3: Supply and demand curves reflecting demand elasticity that increases with price and supply that reflects the
contribution of the opportunity cost of a reserve shortage to marginal cost
Source: Regulatory Assistance Project.
Note: The dotted line curves show the original, unimproved supply and demand curves; the area inside the triangle W represents the gain in
social welfare resulting from the correction. The black dotted lines denote the original market-clearing quantity (right) and the improved market-
clearing quantity (left).
Again, a very truncated version of this effect can be obtained with more traditional command-and-control peak-shaving
programs, but these capture none of the value of daily flexibility, nor do they capture any of the value of the obverse elasticity of
demand to very low prices.
Conclusion
An engaged and responsive demand side is crucial to the affordability—and thus the political sustainability—of the dual
transitions to decarbonization and electrification. The potential savings are predominantly in avoiding uneconomic capital
investment, and those savings are accessible via distributed, customer-driven interaction between flexible loads and the needs
of a system increasingly reliant on variable supply. It is a transition from a legacy power grid run like a symphony orchestra to a
21st century grid functioning more like a jazz ensemble. The single pay-as-cleared, marginal-cost-pricing wholesale energy
market provides the essential information needed to make beautiful music.
Whilst this article addresses specifically the imperative of flexible demand in market design, a brief word is needed regarding
concerns about driving low-carbon investment. There is no validity to claims that growth of renewables must lead to the inability
of spot market prices to support investment. (The subject of this article is an important part of the rebuttal of those arguments.)
Nonetheless, significant improvement in spot market price formation is possible and necessary. Investment also relies on
access to a range of forward risk management tools. These would normally emerge as a natural extension of buyer and seller
incentives in response to legitimate need for new investment (which can be shaped by policy). Yet there are valid concerns that
various factors have conspired to impede that normal market function.
Whilst capacity remuneration mechanisms have proven especially problematic, several promising ideas have been brought
forward about measures to encourage a more robust forward contracting environment.8 Such measures, together with improved
price formation, a focus on the demand side, and well-designed social policy, are fully capable of ensuring adequate investment
in low-carbon supply, whilst also avoiding an unnecessary extended investment hiatus.
8
See for example Schlecht, I., Hirth, L., and Maurer, C. (2022), Financial Wind CfDs, Kiel and Hamburg: ZBW–Leibniz Information Centre for
Economics.
The views expressed here are those of the authors. They do not represent the views of the Oxford Institute for Energy Studies or any
of its Members nor the position of the present or previous employer, or funding body, of any of the authors . 5
