Organization: Open Climate Fix
Work Repository: TabAdjust
Open Climate Fix’s (OCF) Quartz Solar forecasting model produces short-term solar photovoltaic (PV) generation forecasts. These forecasts are post-processed using a simple rule-based adjuster, which computes the mean of recent forecast errors and applies it to correct the incoming forecast. While effective in stable weather conditions, this rule-based approach is rigid and can perform poorly during:
- Rapid weather transitions (e.g., sudden cloud cover changes)
- Atypical error patterns (e.g., persistent underestimation or overestimation)
The aim of this project was to investigate whether TabPFN, a transformer-based foundation model for tabular data, or its time-series variant (TabPFN-TS), can outperform the existing adjuster in terms of Mean Absolute Error (MAE) for the P50 quantile forecast. Another goal of the project was to check whether TabPFN’s accuracy and runtime make it feasible for operational deployment. If so, what necessary changes need to be done to make it feasible.
TabPFN is a pre-trained transformer model designed to work out-of-the-box for small to medium-sized tabular datasets without extensive hyperparameter tuning. It simulates training on 100M synthetic datasets from a Bayesian meta-learning perspective, enabling rapid inference and high accuracy.
Key benefits for this project:
- SoTA model that works well with small daily retraining datasets for tabular regression tasks
- Can incorporate time-series features when using TabPFN-TS
- Requires no explicit gradient-based training during inference
- Uses the mean of past 7 days' forecast errors for each
(time of day, forecast horizon)combination - Adds this mean error to the incoming forecast to generate the adjusted forecast
- Simple, statistical, and computationally efficient
- Limitation: Cannot adapt quickly to changing weather patterns or rare events
For this project, I used the adjuster data from https://github.com/openclimatefix/PVNet, the library for the PVNet model which is running live and the dataset used is from the live forecasts it had produced in the past and its performance against ground truth Solar PV outputs in a region of the UK. The dataset contained the following columns:
| Column Name | Description |
|---|---|
| forecast_period_start_datetime_utc | Start time of forecast period (UTC) |
| forecast_period_end_datetime_utc | End time of forecast period (UTC) |
| forecasted_pv_generation_MW | Forecasted PV generation in MW |
| forecast_version | Version of the forecast |
| actual_pv_generation_MW | Actual measured PV generation in MW |
| forecast_creation_datetime_utc | Time when the forecast was created (UTC) |
| forecast_horizon_minutes | Forecast horizon in minutes |
| forecast_error_MW | Forecasted error in MW (actual - forecast) |
This diagram below contrasts the existing rule-based adjuster (left) with the proposed TabAdjust pipeline (right). The rule-based path aligns timestamps, computes a mean error by horizon × start hour, fills gaps, and smooths.
TabAdjust on the other hand wraps this baseline with a tabular ML/DL adjuster that engineers features (including time-series and lagged error/actual features) and then runs two complementary tasks:
i) a time-series task where TabPFN-TS/XGB are retrained each day on the past week to predict the current day, and
ii) a regression task where XGB/AutoGluon train on months of history while TabPFN(Reg) uses rolling 7/15/30-day lookbacks during the test window.
Predictions from these models are merged across horizons/days, then the adjusted forecast is computed by adding the model-predicted correction to the baseline signal (as shown: combine the forecast error signal with model predictions). Finally, performance is reported via MAE, enabling a clear comparison between the rule-based adjuster and the learned TabAdjust variants.
Using a similar setup and helpers from TabAdjust, I also conducted 12 extensive experiments based on the 2 tasks just mentioned.
a) Experiments 1–2: Time Series Task. Features are TabPFN-TS derived, with and without classical lags.
b) Experiments 3–12: Regression Task. Fixed splits (tabular) and basic TS features. Some variants restrict to 04:30–21:00 local hours to focus on daylight.
In these experiments, I compared the following approaches/ML models:
- Baseline: Unadjusted forecast (Raw)
- OCF: Rule-based mean error adjustment (Statistical)
- XGB: Gradient boosting on engineered features (ML)
- XGB (Tuned): XGB tuned with more deep tree features (such as
n_estimators=1000) (ML) - AG (30 mins): AutoGluon trained with 30 min runtime constraint (AutoML)
- TabPFN (7d / 15d / 30d): TabPFN trained on rolling window of past 7, 15, or 30 days and tested for current day for the same period as other models. The window is chosen keeping in mind the constraints of TabPFN i.e. <10K Samples and 500 features and also benifits from smaller data windows. (ML)
Evaluation Metrics: Mean Absolute Error
| Name | Formula | Applies To | Description |
|---|---|---|---|
| Baseline MAE | MAE(actual_pv_generation_MW, forecasted_pv_generation_MW) |
Baseline | Error of the raw forecast. |
| Adjusted MAE | MAE(actual_pv_generation_MW, adjusted_forecast) |
OCF, XGB, AG, TabPFN | adjusted_forecast = forecasted_pv_generation_MW + adjuster_error |
| Model | adjuster_error definition |
|---|---|
| OCF | Mean of the past 7 days forecast_error_MW for the same time-of-day (forecast_period_start_datetime_utc) and forecast_horizon_minutes. |
| XGB / AG / TabPFN | Predicted value of forecast_error_MW from the model for the given forecast_period_start_datetime_utc and forecast_horizon_minutes (i.e., the model predicts the correction). |
More details about the experimental setup and analysis, can be found here: analysis.md
The figure below compares MAE across 12 experiments for four model. The black star marks the winner per experiment. As expected, OCF consistently improves over Baseline. From Exp. 3–10, TabPFN consistently delivers the lowest MAE, highlighting its strength in mid-season windows and across varying horizons.
In the late-spring windows (Exp. 11–12), a tuned XGB edges ahead, indicating regime sensitivity where tree ensembles can match or beat TabPFN. Overall, the plot shows a clear step down from Baseline → OCF, with learned adjusters (TabPFN/XGB) providing the largest gains.
Some more Key Takeaways:
- TabPFN often outperforms OCF when recent weather dynamics differ from longer-term averages and is better always when treated as a regression task and gives comparable performance with time-series variant of TabAdjust.
- Lag features improve performance in many scenarios as is also observed in Experiment 1, 2.
- XGB performance is sensitive to hyperparameters and feature selection.
- AG can potentially outperform OCF with longer training times, but requires heavy compute.
- TabPFN works well with daily retraining per horizon due to low computational overhead.
Further analysis can be found in analysis.md. To get more insights into how each individual experiment performed, please refer to final_results.
The experiments performed in the GSoC project clearly accomplishes the goals identified for this project. However, there is always a room for improvement and the following could be the tentative next steps:
- Experimentation on the Integration of TabPFN into current OCF Adjuster as a new method considering the current limitations of TabPFN: Add a simple end-to-end API using TabAdjust to switch/weight OCF vs. TabPFN (7/15/30d) vs. tuned XGB by season or recent bias; parallelize per-horizon TabPFN fits and cache features for speed;
- Extend AutoGluon’s training budget (and enable GPU) to unlock stronger ensembles; Currently due to hardware constraints, I could train AG only for 30mins.
- Hyperparameter tuning for XGB. Initially by using low
n_estimatorsXGBoost performed the worst overall. Later, I tried out different hyperparameters which seemed to do the trick. Thus, doing a Hyperparameter Optimization over the XGBoost params can also be another possible extension. - Modify TabAdjust to be more dynamic and easy-to-use as an API would do
TabPFN demonstrates potential as a replacement for OCF’s rule-based adjuster, especially when weather conditions shift rapidly. It provides a balance between accuracy gains and computational feasibility where if needed to get the forecast error for each day will take ~1-2 minutes on NVIDIA-RTX2080, making it promising for operational deployment in real-time solar forecasting. In my opinion, when compute is not an issue, combining TabPFN predictions with OCF via a simple router or ensemble is the most reliable path to consistently low MAE.
Initially, I struggled while dealing with a huge dataset and the limitation of TabPFN as is observed in Experiment 1, and 2 which were run in Phase 1 of this project that I was not able to beat the current OCF MAE which is the bare minimum. However, later in Phase 2 I identified that filtering data per horizon is more meaningful than the entire past data which further helped in reducing computational overhead for TabPFN inference.
As a final word, I would thank GSoC Community to help me connect with an amazing organization where I could enhance my skills and understand the forecasting domain better.
This project would not have been possible without the valuable guidance, support, and collaboration of the following people:
- Sukh Patel, Zak Watts, and Peter Dudfield from Open Climate Fix, for providing domain expertise in solar forecasting, sharing insights into the existing OCF infrastructure, and offering constructive feedback throughout the project and helping me understand the forecasting domain better.
- Prof. (Dr.) Frank Hutter from the Machine Learning Lab, University of Freiburg, for his continuous mentorship, encouragement, resources, and expertise in machine learning, which greatly influenced the experimental design and methodology.
-
PVNet (Open Climate Fix)
Primary repository and documentation for OCF’s multimodal PV forecasting model (NWP + satellite + metadata).
• PVNet main repo: openclimatefix/PVNet. -
TabPFN-TS (Time-Series wrapper around TabPFN)
Time-series package that frames forecasting as tabular regression with lightweight feature engineering.
• Official code: PriorLabs/tabpfn-ts -
TabPFN (Tabular Prior-Data Fitted Networks)
Foundation model for small/medium tabular tasks; in-context learning via a pretrained Transformer.
• Original paper: TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second (arXiv:2207.01848).
• Nature paper: Accurate predictions on small data with a tabular foundation model (Nature).
• Official code: Priorlabs/tabpfn.