Large-Scale Energy Forecasting & Decision Support

An energy company holds large volumes of hourly smart-meter data, but raw consumption data tells the customer very little. The question was how to make forecasts useful in practice: expected consumption, when the peaks occur, which grid-tariff capacity tier the customer ends up in, and what that means in money.

As part of a team, I developed a proof of concept with a modular forecasting architecture in which different models answer different questions, instead of one forecast curve trying to optimise everything: expected normal consumption, personalised daily and hourly consumption, the size and timing of the daily peak, capacity tier and cost. Cost is calculated in a separate rule-based engine for spot price, grid tariff and capacity charge, so tariff changes do not require retraining. Anomaly diagnostics and load-shifting scenario analysis sit on top of the forecasts.

A Databricks/Spark pipeline from raw metering data to model-ready features, with weather data, spot prices, tariffs and calendar information. Everything was evaluated chronologically, on a deterministic, stratified sample of metering points that made training scalable, and simple baselines were established before any machine learning. The architecture was driven by measurement: a simple per-meter calibration lowered hourly error but worsened peak, capacity and cost estimates, so it was rejected. Customer-specific shape profiles were also tested and rejected. The project ended with a forecast contract, lineage between datasets and an automated final audit.

On a chronologically separate test year, the global model’s hourly error (WAPE) was about 0.29, against about 0.35 for the strongest simple baseline (a 24-hour rolling mean), around 16% lower. Dedicated peak models had the largest effect where the value lies: the mean error on the daily peak fell by about 54% compared with the starting point without a dedicated peak model, and the share of cases where the capacity tier was estimated exactly right rose from about 74% to about 91%, without changing the hourly forecast. The solution was assessed as technically complete with documented limitations and a recommended, controlled next phase. It has not been put into production.

The cost figures are tariff-simulated estimates using a fixed tariff configuration, not historical bills, and the scenario analysis shows estimated potential under explicit assumptions, not realised savings. The evaluation uses rolling forecasts with available history and observed weather, not a genuine day-ahead forecast in operation. The test year was also used for confirmation during development, so the figures do not come from a completely untouched final set. No service has been deployed and no customer pilot has measured actual effect.