Continuous Ranked Probability Score (CRPS)

A Note on Continuous Ranked Probability Score (CRPS)

What is Continuous Ranked Probability Score (CRPS) ?

How to evaluate probablistic forcasting is good or not? Continuous Ranked Probability Score (CRPS) is a evaluation metrics to quantify this. Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477), 359-378.

Below is an example of probability forcasting of airline passengers Data from: https://raw.githubusercontent.com/jbrownlee/Datasets/master/airline-passengers.csv where blue line is the ground truth and the red line is the forecase mean with the ribbon is various confidence interval.

Definition

The CRPS measures the difference between a predicted cumulative distribution function (CDF) \(F(x)\) and the observed value \(y\). It is defined as:

\[\text{CRPS}(F, y) = \int_{-\infty}^\infty \left( F(x) - \mathbb{1}(x \geq y) \right)^2 dx.\]

The perfect prediction is \(\mathbb{1}(x \geq y)\) where the CDF is \(1\) greater and equal to observation \(y\). The square of difference measure the closeness of prediction density to this perfect density.

Quantile Loss

The quantile loss \(\rho_{\alpha}(u)\) at a quantile level \(\alpha \in [0, 1]\) for a deviation \(u = z - q(\alpha)\) is defined as:

\[\rho_{\alpha}(u) = \begin{cases} \alpha \cdot u & \text{if } u \geq 0, \\ (1 - \alpha) \cdot |u| & \text{if } u < 0. \end{cases}\]

where \(z\) is the true value, \(q(\alpha)\) is the predicted quantile at level $\alpha$, and \(u = z - q(\alpha)\) is the difference between true and prediction. The intuition make sense if we want to predict 75th quantile, we will penelize \(3\) times for under-estimated than over-estimated, for more example you can refer to here https://www.kaggle.com/code/vyacheslavefimov/quantile-loss-quantile-regression.

Quantile Decomposition of CRPS

We can rewrite CRPS as integral over quantile loss Gneiting, T., & Ranjan, R. (2011). Comparing density forecasts using threshold-and quantile-weighted scoring rules. Journal of Business & Economic Statistics, 29(3), 411-422.

\[\begin{align} \text{CRPS}(F, y) &= \int_{-\infty}^\infty \left( F(x) - \mathbb{1}(x \geq y) \right)^2 dx \\ &= \int_{0}^{1} 2 \rho_{\alpha} \left( z - q(\alpha) \right) d \alpha, \end{align}\]

where (2) is the quantile decomposition of the CRPS. The key step in the derivation involves performing a change of variable from \(x\) to \(\alpha\) and utilizing the relationship between the quantile function and the cumulative distribution function (CDF). Specifically, the quantile function \(q(\alpha)\) is defined as the inverse of the CDF \(F(x)\): \(\begin{align} q(\alpha) = F^{-1}(\alpha) = \inf \{ x \in \mathbb{R} \mid F(x) \geq \alpha \}. \end{align}\)