Knowledge center · Validation & statistics
Confidence Calibration
Whether a signal shown with X% confidence was actually right about X% of the time.
Formula
Per confidence bucket: realized hit rate vs. the bucket's expected (midpoint) confidence
Interpretation
A reliability diagram. If the 70% confidence bucket only won 55% of the time historically, the engine's confidence score is overstated for that bucket, and should be read with a discount.
What good looks like
Realized hit rate tracks close to the expected marker across most buckets.
What bad looks like
Realized hit rate sitting well below the marker in every bucket (systematic overconfidence), especially in high-confidence buckets.
Typical range
A gap of a few percentage points between expected and realized is normal on modest sample sizes; a gap over ~15 points in a well-populated bucket is worth distrusting.
How OmniSignal uses it
Confidence is only useful if it's honest — calibration is the direct test of whether the number can be taken at face value.
References
Reliability diagrams, standard in forecast verification (meteorology and ML both use this exact tool)
Related concepts