Forecasting Routine Vaccine Administration Under Uncertainty
A Comparative Evaluation
Udeshi Salgado, Cardiff University, UK
Lead supervisor: Prof. Bahman Rostami-Tabar
Co-supervisors: Dr Thanos E Goltsos, Dr Geraint Palmer, Dr Paul Wang
Data Lab for Social Good, Cardiff University, UK
30 June 2026
Outline
- Background and motivation
- Current practice and literature
- Data and Methodology
- Results
Outline
- Background and motivation
- Current practice and literature
- Data and Methodology
- Results
Background
- 1 in 5 children worldwide still lack access to essential vaccines.
- A key operational driver is inefficiency in vaccine supply chains.
- In low and middle-income countries this shows up as:
- inaccurate demand forecasts
- inventory decisions made with no measure of uncertainty
- wastage and stockouts
The immunisation supply chain
![]()
Outline
- Background and motivation
- Current practice and literature
- Data and Methodology
- Results
Projecting consumption demand, three ways
FSP4All projects consumption three ways, then blends them into one annual figure.
Demographic
- Multiplies the eligible child population by a coverage target and the dose schedule
- Converted to a procurement quantity with an assumed wastage rate
- \(Q_{\mathrm{dem}} = P \cdot \frac{c}{100} \cdot d \cdot \frac{100}{100 - w}\)
Consumption
- Averages the last 12 months of issues into one flat mean
- Adjusts for stockout days, then projects forward with an assumed growth rate
- \(Q_{\mathrm{cons}} = \bar{C} \cdot \frac{D}{D - D_{\mathrm{so}}} \cdot 12 \cdot (1+g)^{t} \cdot \frac{100-w_{\mathrm{old}}}{100-w_{\mathrm{new}}} \cdot (1+r)\)
Session
- Adds up every planned session’s expected attendance
- Built bottom-up from the session plan and attendance
- \(Q_{\mathrm{sess}} = \sum_{s} n_{s} \cdot \bar{a}_{s} \cdot d\)
Then combine
- \(Q_{\mathrm{FSP}} = \sum_{m} \omega_{m}\, Q_{m}\)
- Weights are decided by the experts in the immunisation supply chain. Default weights are equal
Notation \(P\): target population
\(c\): coverage (%)
\(d\): doses per child
\(\bar{C}\): avg. monthly consumption
\(D,\ D_{\mathrm{so}}\): days in period, stockout days
\(g\): population growth
\(t\): years from data to plan year
\(r\): planner’s manual % adjustment, e.g. for a campaign
\(w\): wastage (%)
\(n_s\): sessions
\(\bar{a}_s\): attendance
\(\omega_m\): method weights
Vaccine demand forecasting in the literature
|
Reference
|
Approach / method
|
Target variable
|
Level
|
Horizon
|
Evaluation focus
|
Inventory integration
|
Benchmark
|
Probabilistic forecasting
|
Distributional post-processing
|
Reproducibility
|
|
Chiu et al. (2008)
|
ARIMA, BPNN
|
Annual demand
|
Regional
|
1 year
|
Average error
|
No
|
No
|
No
|
No
|
No
|
|
Kotagiri et al. (2011)
|
Birth-cohort model
|
Inventory requirements
|
Local inventory
|
1 year
|
Inventory impact
|
Yes
|
No
|
No
|
No
|
No
|
|
Mueller et al. (2016)
|
Simulation
|
Vaccine quantity
|
Multi-level
|
12 months
|
Availability, cost
|
Yes
|
No
|
No
|
No
|
No
|
|
Azadi et al. (2018)
|
Regression
|
Childhood demand
|
Regional
|
n.s.
|
Not reported
|
No
|
No
|
No
|
No
|
No
|
|
Cernuschi et al. (2018)
|
Demand-supply assessment
|
BCG demand, supply
|
National, global
|
14 years
|
Procurement
|
No
|
No
|
No
|
No
|
No
|
|
Alegado & Tumibay (2020)
|
ARIMA, MLP
|
Monthly demand
|
Local
|
12 months
|
RMSE, MAE
|
No
|
No
|
No
|
No
|
No
|
|
Hariharan et al. (2020)
|
Random forest
|
Utilisation
|
Facility
|
Biweekly
|
RMSE
|
No
|
Yes
|
No
|
No
|
No
|
|
Sahisnu et al. (2020)
|
ARIMA
|
Stock levels
|
Local
|
10 months
|
MAPE
|
Yes
|
No
|
No
|
No
|
No
|
|
Vinitha et al. (2024)
|
Regression, trees, LSTM, ANN
|
Infant demand
|
Local
|
n.s.
|
RMSE, R²
|
No
|
No
|
No
|
No
|
No
|
|
Current study
|
FSP-informed administered benchmark, statistical, ML, DL, foundation, ensemble
|
Administered doses, monthly
|
Sub-county
|
6 months
|
CRPS, rMAE, rRMSE
|
No
|
Yes
|
Yes
|
Yes
|
Yes
|
From practice and literature to our contribution
We combine the domain knowledge built into FSP4All with the open gaps in the forecasting literature, and use the data we do have, doses administered, to build a feasibility-aware probabilistic forecast.
From the tools · domain knowledge
- FSP4All’s demographic feasibility logic
- Population, coverage and capacity structure
- A principled ceiling for plausible demand
From the literature · the open gaps
- Forecasts are rarely probabilistic
- Reproducible benchmarks are scarce
- Distributions ignore physical feasibility
Our contribution: model the administered history probabilistically, then make the distribution feasibility-aware with FSP-informed bounds.
Research Question How can data-driven probabilistic forecasting, combined with FSP-informed demographic-consumption truncation, advance routine childhood vaccine administered-demand forecasting at the most granular operational level, where the supply decisions are actually made?
Outline
- Background and motivation
- Current practice and literature
- Data and Methodology
- Results
Data
5 vaccines BCG · Pentavalent · Measles-Rubella · OPV birth · OPV routine
306 sub-counties the most granular operational level
1,530 series one per vaccine and sub-county
Monthly, 2013 to 2021 doses administered, calendar-normalised
Demographic and coverage FSP target population · WHO coverage targets
External drivers floods · drought · epidemics · CPI · holidays
Demand across administrative levels
![]()
Trend, seasonality and noise
![]()
Methodology
Data sources
Administered doses African country, sub-county · 2013 to 2021 · monthly · BCG, DPT-HepB-Hib, Measles-Rubella, OPV birth, OPV routine
Demographic and external FSP target population (sub-county level), WHO coverage · floods, drought, epidemics · Consumer Price Index (CPI) · working days, holidays, lags
▼
Forecasting setup
FSP-informed administered benchmark capacity-adjusted demographic
Statistical ARIMA · ETS · Naive · sNaive
Machine learning Lasso · Elastic Net · RF · XGBoost · LightGBM
Deep learning and foundation ANN · LSTM · Chronos
Rolling-origin cross-validation · 6-month horizon (h = 1 to 6) · raw output \(\mathcal{N}(\mu, \sigma)\)
▼
Distributional post-processing
1 · Calibration 2018 to 2019 hold-out · per vaccine and sub-county · \(k = Q_{0.975}(y / E)\)
2 · Define bounds \(L = 0\) (non-negativity) · \(U = k\,E\) (demographic-consumption ceiling)
3 · Truncated distribution recast \(\mathcal{N}(\mu,\sigma)\) as \(\mathrm{TN}(\mu, \sigma, 0, U)\) · scored with exact TN CRPS
▼
Feasibility-aware probabilistic forecast: bounded support, calibrated tails, decision-relevant CRPS
Forecasting setup: data-driven forecasting models
Statistical, ML, DL and foundation models learn directly from the historical administered doses and external drivers. Whatever the family, each is turned into a probabilistic forecast the same way.
Model families
Statistical ARIMA · ETS · Naive · sNaive
Machine learning Lasso · Elastic Net · RF · XGBoost · LightGBM
Deep learning & foundation ANN · LSTM · Chronos
▼
Turning each model into a predictive distribution
1 · Point forecast fit on history and features, predict the mean \(\hat{\mu} = \hat{y}_{i,v,t+h}\)
2 · Error spread \(\hat{\sigma}_h\) from rolling-origin forecast errors, widening with horizon \(h\)
3 · Gaussian forecast combine into \(Y_{i,v,t+h} \sim \mathcal{N}(\hat{\mu}, \hat{\sigma}_h)\), for \(h = 1\) to \(6\)
The Gaussian fits the near-Normal body of the forecast errors; the heavier tails are corrected downstream by feasibility truncation
Distributional post-processing: Calibration
The benchmark \(E\) is a theoretical plan; what clinics actually administer departs from it. We learn one empirical factor \(k\) from history to tie the two together.
1 · Administered benchmark, at 100% coverage
\[E = \tfrac{1}{12}\, P \cdot \tfrac{c}{100} \cdot d \qquad\xrightarrow{\;c\,=\,100\;}\qquad E = \tfrac{1}{12}\, P \cdot d\]
2 · Scaling factor \(k\), the 97.5th percentile of realised demand against the plan, on a 2018–2019 hold-out
\[k_{i,v} = Q_{0.975}\!\left(\frac{y}{E} \;\Big|\; 2018\text{–}2019\right)\]
\(k>1\): administered demand routinely exceeds the raw demographic, so \(k\) grounds the ceiling in operational reality while trimming the impossible.
Distributional post-processing: defining bounds
Lower bound
\[L = 0\]
A structural constraint: monthly doses administered cannot be negative.
Upper bound
\[U_{i,v,t} = k_{i,v}\,E^{\uparrow}_{i,v,t}, \quad E^{\uparrow}_{i,v,t} = \tfrac{1}{12}\, P^{\uparrow} d\]
\(E^{\uparrow}\) uses the upper population \(P^{\uparrow}\) at 100% coverage, so the ceiling never cuts legitimate surges; \(k\) sets it at the 97.5th percentile of observed demand.
From bounds to truncated distribution
Once \([L, U]\) is set, the raw Gaussian forecast is re-cast as a truncated Normal on that interval.
\[Y_{i,v,t+h} \sim \mathcal{N}(\mu, \sigma)
\;\Longrightarrow\;
Y_{i,v,t+h} \sim \mathrm{TN}(\mu, \sigma, L, U)\]
Standardised bounds and normaliser (\(\Phi\), \(\phi\): standard Normal CDF and PDF):
\[\alpha = \frac{L-\mu}{\sigma},\qquad \beta = \frac{U-\mu}{\sigma},\qquad Z = \Phi(\beta) - \Phi(\alpha)\]
Truncated mean and variance:
\[\mu^{\mathrm{tr}} = \mu + \sigma\,\frac{\phi(\alpha) - \phi(\beta)}{Z}, \qquad
(\sigma^{\mathrm{tr}})^2 = \sigma^2\!\left[1 + \frac{\alpha\,\phi(\alpha) - \beta\,\phi(\beta)}{Z} - \left(\frac{\phi(\alpha)-\phi(\beta)}{Z}\right)^{2}\right]\]
The truncated distribution is scored with the exact closed-form truncated Normal CRPS.
Truncation in practice
![]()
The untruncated raw Gaussian wastes mass below 0 and above \(U\). Truncation concentrates it into the feasible range, improving the CRPS.
Evaluation metrics
Every error is scaled by the FSP-informed administered benchmark: below 1 beats it, above 1 is worse.
Point accuracy
\[\mathrm{MSE} = \tfrac{1}{n}\sum_t (y_t-\hat{y}_t)^2\]
\[\mathrm{MAE} = \tfrac{1}{n}\sum_t |y_t-\hat{y}_t|\]
\[\mathrm{rRMSE} = \sqrt{\tfrac{\mathrm{MSE}}{\mathrm{MSE}_{\text{FSP-informed admin.}}}}\]
\[\mathrm{rMAE} = \tfrac{\mathrm{MAE}}{\mathrm{MAE}_{\text{FSP-informed admin.}}}\]
Distributional accuracy
\[\mathrm{CRPS}(F,y) = \int_{-\infty}^{\infty}\big(F(x)-\mathbf{1}\{x\ge y\}\big)^2\,dx\]
\[\mathrm{rCRPS} = \tfrac{\mathrm{CRPS}}{\mathrm{CRPS}_{\text{FSP-informed admin.}}}\]
Truncated forecasts are scored with the exact closed-form CRPS of the truncated Normal, not the raw Gaussian.
Outline
- Background and motivation
- Current practice and literature
- Data and Methodology
- Results
Truncation Effect: rRMSE
![]()
Distribution of rRMSE
![]()
Truncation Effect: rCRPS
![]()
Truncation tightens the dispersed forecasters at longer horizons
![]()
Accuracy and computational efficiency
![]()
The road ahead
Where this work goes next.
Add the wastage component
- Extend administered forecasts to total consumption
- Supports end-to-end procurement quantification
Link to inventory simulation
- Translate forecast distributions into stockouts, wastage, and service levels
Hierarchical reconciliation
- Keep forecasts coherent across national → sub-county levels
Any questions or thoughts? 💬
![]()
Visit my website for slides
Weak trend and seasonality
![]()
Before truncation: point accuracy
![]()
Before truncation: distributional accuracy
![]()
Distribution of errors across horizons
![]()
Where truncation helps most
![]()
