Model Fit Plots

Purpose

Model fit plots summarise the posterior and compare fitted values against observed data. They answer two questions: does the model track the response variable adequately, and are the estimated coefficients plausible? These plots are written to 20_model_fit/ within the run directory.

The runner generates them via write_model_fit_plots() in R/run_artifacts_enrichment.R. All four plots require ggplot2 and the fitted model object. Each is wrapped in tryCatch so that a failure in one does not prevent the others from being written.

Plot catalogue

Filename What it shows Conditions
fit_timeseries.png Observed vs fitted over time with 95% credible band Always generated after a successful fit
fit_scatter.png Observed vs fitted scatter Always generated after a successful fit
posterior_forest.png Coefficient point estimates with 90% CIs Posterior draws available via get_posterior()
prior_posterior.png Prior-to-posterior density shift for media terms Model has a .prior table with media (m_*) parameters

Fit time series

Filename: fit_timeseries.png

Fit time series Fit time series

What it shows

The observed KPI (orange) and posterior mean fitted values (blue) plotted over time, with a shaded 95% credible interval band. The subtitle reports in-sample fit metrics: R², RMSE, MAE, mean error (bias), sMAPE, 95% prediction interval coverage, lag-1 ACF of residuals, and sample size. For hierarchical models the plot facets by group.

When it is generated

Always, provided the model has been fitted successfully and the fit table (observed, mean, percentiles) can be computed.

How to interpret it

The fitted line should track the general level and seasonal pattern of the observed series. The 95% credible band should contain most observed points — the subtitle reports the actual coverage, which should be close to 95%. Systematic departures reveal model misspecification: if the fitted line consistently overshoots during holidays or undershoots during quiet periods, the formula may lack appropriate seasonal or event terms.

Warning signs

  • Coverage well below 95%: The model underestimates uncertainty. Common when the noise prior is too tight or the model is overfit to a subset of the data.
  • Coverage well above 95%: The credible interval is too wide. The model is underfit or the noise prior is too diffuse.
  • Persistent bias (ME far from zero): The model systematically over- or under-predicts. Check for missing structural terms (trend, level shifts, intercept misspecification).
  • High lag-1 ACF (> 0.3): Residuals are autocorrelated. The model is missing temporal structure — consider adding lagged terms or checking adstock specifications.

Action

If coverage or bias is unacceptable, revisit the formula (missing controls, wrong functional form) or the prior specification (overly tight noise SD). Cross-reference with the residuals diagnostics for a more detailed picture.

  • fit_metrics_by_group.csv in 20_model_fit/ provides the same metrics in tabular form, broken down by group for hierarchical models.

Fit scatter

Filename: fit_scatter.png

Fit scatter Fit scatter

What it shows

A scatter plot of observed values (y-axis) against posterior mean fitted values (x-axis), with a 45-degree reference line. Points on the line indicate perfect fit. For hierarchical models the plot facets by group.

When it is generated

Always, provided the fit table is available.

How to interpret it

Points should cluster tightly around the diagonal. Curvature away from the line suggests a systematic misfit — for instance, if the model underpredicts at high KPI values, the response may need a nonlinear term or a log transformation. Outliers far from the line warrant investigation: they may correspond to anomalous weeks (data errors, one-off events) that the model cannot capture.

Warning signs

  • Fan shape (wider scatter at higher values): Heteroscedasticity. A log-scale model or a variance-stabilising transform may be more appropriate.
  • Systematic curvature: The mean function is misspecified. Consider adding polynomial or interaction terms.
  • Isolated outliers: Check the dates of extreme residuals against the residuals time series and the input data for data quality issues.

Action

If the scatter reveals non-constant variance, consider fitting on the log scale (model.scale or a log-transformed formula). If curvature is evident, review the functional form of media transforms and control variables.

Posterior forest plot

Filename: posterior_forest.png

Posterior forest Posterior forest

What it shows

A horizontal forest plot of posterior coefficient estimates. Each row is a model term (excluding the intercept). The point marks the posterior median; the horizontal bar spans the 5th to 95th percentile (90% credible interval). Terms whose interval excludes zero are drawn in colour; those consistent with zero are grey.

For hierarchical models, the plot displays population-level (group-averaged) estimates.

When it is generated

The runner generates this plot when posterior draws are available via get_posterior(). It is skipped if the posterior extraction fails.

How to interpret it

Focus on the media coefficients. Positive values indicate that higher media exposure is associated with higher KPI, which is the expected direction for most channels. The width of the interval reflects estimation precision: a narrow interval means the data informed the estimate strongly; a wide interval means the prior dominates.

Terms ordered by absolute magnitude (bottom to top) give a quick ranking of effect sizes, but note that these are on the model’s internal scale. For models fitted on the log scale, coefficients represent approximate percentage effects; for levels models, they represent absolute KPI units per unit of the transformed media input.

Warning signs

  • Media coefficient crosses zero: The model cannot confidently distinguish the channel’s effect from noise. This is not necessarily wrong — some channels may genuinely have weak effects — but it warrants scrutiny, especially if the prior was informative.
  • Implausibly large coefficients: Check for scaling issues. If model.scale: true, coefficients are on the standardised scale and must be interpreted accordingly.
  • All intervals very wide: The data may not have enough variation to identify individual effects. Review the VIF bar chart for multicollinearity.

Action

If a media coefficient is unexpectedly negative, investigate whether the data supports it (e.g. counter-cyclical spend) or whether multicollinearity is pulling the estimate. Cross-reference with the prior vs posterior plot to see how far the data moved the estimate from its prior.

Prior vs posterior

Filename: prior_posterior.png

Prior vs posterior Prior vs posterior

What it shows

Faceted density plots for each media coefficient (m_* parameters). The grey distribution is the prior (Normal, as specified in the model’s .prior table); the blue distribution is the posterior (estimated from MCMC draws). Overlap indicates that the data did not strongly inform the estimate; separation indicates data-driven updating.

For hierarchical models, posterior draws are averaged across groups to show the population-level density.

When it is generated

The runner generates this plot when:

  • The model has a .prior table (i.e. it is a requires_prior model).
  • The prior table contains at least one m_* parameter.
  • Posterior draws are available.

If the model has no prior table (e.g. a pure OLS updater), the plot is skipped.

How to interpret it

A well-identified coefficient shifts noticeably from prior to posterior. If the two densities sit on top of each other, the data provided little information for that channel — the estimate is prior-driven. This is not inherently wrong (the prior may be well-calibrated from previous studies), but it does mean the current dataset alone cannot validate the estimate.

Warning signs

  • No shift at all: The channel has insufficient variation or is too collinear with other terms for the data to update the prior. The resulting coefficient is essentially assumed, not estimated.
  • Posterior much narrower than prior: Expected and healthy. The data concentrated the estimate.
  • Posterior shifted to the boundary: If a boundary constraint is active (e.g. non-negativity), the posterior may pile up at zero. Cross-reference with the boundary hits plot to confirm.

Action

If key media channels show no prior-to-posterior shift, consider whether the prior is appropriate, whether the data period is long enough, or whether multicollinearity prevents identification. For channels where the prior dominates, document this clearly when reporting ROAS or contribution estimates — the output reflects an assumption, not a data-driven finding.

Cross-references