Interpret Diagnostics

Objective

Read and act on the diagnostics report produced by a DSAMbayes runner execution, understanding which checks matter most and what remediation steps to take.

Prerequisites

  • A completed runner run execution with artefacts under 40_diagnostics/.
  • Familiarity with Diagnostics Gates definitions.

Steps

1. Open the diagnostics report

cat results/<run_dir>/40_diagnostics/diagnostics_report.csv

Each row is one diagnostic check. The key columns are:

Column What to look at
check_id Identifies the specific diagnostic
status pass, warn, fail, or skipped
value The observed metric value
threshold The threshold that was applied
message Human-readable explanation

2. Check the overall status

The overall status follows a simple rule:

  • Any fail → overall fail.
  • Any warn (no fails) → overall warn.
  • All pass → overall pass.

If the overall status is pass, no further action is required for the configured policy mode.

3. Triage failing checks

Focus on fail rows first, then warn rows. Use the check phase to prioritise:

Phase Priority Meaning
P0 Highest Data integrity issues — fix before interpreting model results
P1 High Sampler quality or residual issues — may affect inference reliability

4. Common diagnostics and actions

Design matrix issues (P0)

Check Symptom Action
pre_response_finite fails Non-finite values in response Clean data; remove or impute NA/Inf rows
pre_design_constants_duplicates fails Constant or duplicate columns Remove redundant terms from formula
pre_design_condition_number warns/fails High collinearity Reduce correlated predictors; simplify formula

Sampler quality (P1, MCMC only)

Check Symptom Action
sampler_rhat_max warns/fails Poor convergence Increase fit.mcmc.iter and fit.mcmc.warmup; simplify model
sampler_ess_bulk_min or sampler_ess_tail_min warns/fails Insufficient effective samples Increase iterations; check for multimodality
sampler_divergences fails Divergent transitions Increase fit.mcmc.adapt_delta (e.g. 0.95 → 0.99); consider reparameterisation
sampler_treedepth_frac warns/fails Max treedepth saturation Increase fit.mcmc.max_treedepth
sampler_ebfmi_min warns/fails Low energy diagnostic Indicates difficult posterior geometry; simplify model or increase warmup

Residual behaviour (P1)

Check Symptom Action
resid_ljung_box_p warns/fails Significant residual autocorrelation Add time controls (trend, seasonality, holidays)
resid_acf_max warns/fails High residual ACF at early lags Same as above; check for missing structural components

Boundary and variation checks (P1)

Check Symptom Action
boundary_hit_fraction warns/fails Posterior draws hitting parameter bounds Review boundary specification; widen constraints or remove unnecessary bounds
within_var_ratio warns/fails Low within-group variation (hierarchical) Check group structure; some groups may have insufficient temporal variation

Identifiability gate (P1)

Check Symptom Action
pre_identifiability_baseline_media_corr warns/fails High baseline-media correlation Add controls to separate baseline from media effects; review formula specification

5. Review diagnostic plots

Cross-reference the numeric report with visual diagnostics in 40_diagnostics/:

  • Residual diagnostics plot — check for patterns in residuals over time.
  • Boundary hits plot — identify which parameters are constrained.
  • Latent residual ACF plot — confirm autocorrelation structure.

See Diagnostics Plots for interpretation guidance.

6. Decide on next steps

Overall status Action
pass Proceed to post-run analysis and reporting
warn Review warnings; proceed if acceptable for the use case
fail Remediate failing checks before using model results for decisions

7. Change policy mode if appropriate

If you are in early model development, consider switching to explore mode to relax thresholds:

diagnostics:
  policy_mode: explore

For production or audit runs, use publish (default) or strict.