Hybrid RL + Optimal Power Flow for Smart-Grid Voltage Control
A self-correcting controller with RL inference speed and OPF safety guarantees for DER-rich feeders
A hybrid control framework that fuses the millisecond inference speed of deep reinforcement learning with the deterministic safety guarantees of numerical Optimal Power Flow (OPF) for voltage regulation on distribution feeders saturated with distributed energy resources (DERs). A Proximal Policy Optimization (PPO) agent proposes DER curtailment actions; every action is verified in real time against a pandapower AC power-flow digital twin, and if it risks a voltage violation an AC OPF solver computes a safe fallback. Failed actions are logged for online behavior cloning so the policy keeps improving. On a simulated 10-house low-voltage feeder with five DERs, the hybrid controller held 100% of timesteps inside the 0.95–1.05 p.u. band with zero violations — at RL-grade latency.
Hybrid architecture: low-latency PPO controller backed by an AC OPF safety fallback
Simulation-in-the-loop verification physically tests every action — zero real-world violations
Self-correcting online behavior-cloning loop drives the verification failure rate down over time
100% constraint-compliant steps across a 72-step evaluation on a 10-house feeder
A closed verify-then-act loop
The heart of the system is a loop. A PPO agent reads a 26-dimensional snapshot of the grid and proposes curtailment fractions for each DER. Before anything touches the real feeder, that action runs through a pandapower AC power-flow digital twin. If every bus voltage lands inside 0.95–1.05 p.u., the action ships; if not, an AC OPF solver computes a safe, minimum-curtailment fallback — and that correction is logged to retrain the agent.
The test bed: a 10-house feeder
I modeled a 10-house low-voltage radial feeder with five heterogeneous DERs — three rooftop solar units, one wind turbine, and one gas micro-generator. It's intentionally small, but it reproduces the hard part: under low household demand, local solar injection drives voltage up, and the controller has to curtail just enough to keep every node safe without wasting clean energy.
Four controllers, head-to-head
Across 72 stochastic control steps I compared four controllers. No-control drifts out of bounds constantly; pure RL is fast but occasionally unsafe; pure OPF is safe but slow. My hybrid matched OPF-grade safety — 100% compliant steps, zero violations — while running at RL inference speed, and the behavior-cloning loop kept pushing the fallback rate down.
Want the full story?
Happy to talk through any of the engineering decisions, trade-offs, or what broke along the way.