1. What is probability calibration?
Suppose a soccer model predicts BTTS Yes at 70% for a large set of matches. If about 70% of those matches actually end with both teams scoring, that group is calibrated. It does not mean the model should be judged correct every time it predicts 70%, and it does not establish a betting advantage.
Calibration is conditional on the events, period and population being evaluated. A model may be calibrated overall but poorly calibrated for a particular league, season, high-confidence subset or change in team conditions.
2. What is a Brier score?
For a binary event such as BTTS Yes, encode the observed outcome as 1 if it happened and 0 if it did not. If the forecast probability is p and the outcome is y, the squared error is (p − y)². The Brier score is the mean of those squared errors across the evaluated events.
Smaller is better, but the score has to be interpreted relative to a relevant baseline and the event frequency. Comparing raw Brier scores across different event populations can be misleading.
3. A four-match arithmetic example
These four probabilities and outcomes are invented solely to demonstrate the calculation. They are not Cateo predictions or observed soccer matches.
| Forecast BTTS Yes | Hypothetical outcome | Squared error |
|---|---|---|
| 80% | 1 | (0.80 − 1)² = 0.04 |
| 60% | 0 | (0.60 − 0)² = 0.36 |
| 30% | 1 | (0.30 − 1)² = 0.49 |
| 10% | 0 | (0.10 − 0)² = 0.01 |
The total squared error is 0.90. Dividing by four gives a Brier score of 0.225. A constant 50% forecast would score 0.25 on these same four illustrative binary outcomes. Neither difference nor sample size supports a real-world model-performance claim.
4. How reliability diagrams help
Group predictions into probability ranges, such as 0–10%, 10–20% and so on. For each range, plot the average forecast against the observed event frequency. Points near the diagonal indicate closer calibration; a model consistently above or below it may be overconfident or underconfident.
Show the number of forecasts in each range, uncertainty intervals, and the exact binning method. A nearly empty 90–100% bin is not convincing evidence of calibration. If the model changes over time, evaluate the newer forecasts separately.
5. Calibration versus predictive usefulness
A model that always forecasts a league-wide BTTS frequency can be reasonably calibrated without ranking individual fixtures well. Evaluate discrimination and overall scoring quality, not just calibration. Log loss is another proper scoring rule, and a useful review compares both against simple baselines and past model versions.
Different markets or targets require separate evaluation: BTTS, over 2.5 goals and home/draw/away do not share a single binary outcome definition. For multiclass match results, use the corresponding multiclass scoring formulation.
6. Prevent look-ahead bias
Save each probability, target definition, model version, fixture identity, timestamp, input snapshot and publication eligibility before kickoff. Verify the result afterward from an authorized source. Train on earlier observations, test on later unseen observations, and report misses as well as successes.
Retrospective backtests should be labeled separately from a prospective track record. If a model is tuned repeatedly on the same test set, the test ceases to provide independent evidence. A short winning streak is not proof of accuracy or profit.
7. How Cateo intends to use these metrics
The Cateo research roadmap calls for comparing Model A and Model B on calibration, Brier score, log loss and other out-of-sample measures. The current public experience is a fictional research preview; no verified live forecast performance record is being claimed here. When eligible real forecasts are published, their time-stamped predictions, evaluation periods, sample sizes and failures should accompany any reported metrics.