Run from YAML

Objective

Execute a complete DSAMbayes model run from a YAML configuration file and verify the staged output artefacts.

Prerequisites

• DSAMbayes installed locally (see Install and Setup).
• A YAML config file (see Config Schema for structure).
• Data file(s) referenced by the config are accessible.

Steps

1. Set up the environment

mkdir -p .Rlib .cache
export R_LIBS_USER="$PWD/.Rlib"
export XDG_CACHE_HOME="$PWD/.cache"

2. Validate the configuration (dry run)

Rscript scripts/dsambayes.R validate --config config/my_config.yaml

Expected outcome:

• Exit code 0.
• A timestamped run directory under results/ containing 00_run_metadata/config.resolved.yaml and 00_run_metadata/session_info.txt.
• No Stan compilation or sampling occurs.

If validation fails:

• Check the error message for missing data paths, invalid YAML keys, or formula errors.
• Fix the config and re-run validate before proceeding.

3. Run the model

Rscript scripts/dsambayes.R run --config config/my_config.yaml

Expected outcome:

• Exit code 0.
• Full staged artefact tree under the run directory.

4. Locate the run directory

The runner prints the run directory path during execution. It follows the pattern:

results/YYYYMMDD_HHMMSS_<run_label>/

5. Verify artefacts

Check that the following stage folders are populated:

6. Quick verification commands

# Check diagnostics overall status
head -1 results/<run_dir>/40_diagnostics/diagnostics_report.csv

# View posterior summary
head results/<run_dir>/30_post_run/posterior_summary.csv

# Count artefact files
find results/<run_dir> -type f | wc -l

Failure handling

Related pages

• CLI Usage — command flags and exit codes
• Config Schema — YAML configuration reference
• Output Artefacts — stage folder contract
• Quickstart — minimal first run

Interpret Diagnostics

Objective

Read and act on the diagnostics report produced by a DSAMbayes runner execution, understanding which checks matter most and what remediation steps to take.

Prerequisites

• A completed runner run execution with artefacts under 40_diagnostics/.
• Familiarity with Diagnostics Gates definitions.

Steps

1. Open the diagnostics report

cat results/<run_dir>/40_diagnostics/diagnostics_report.csv

Each row is one diagnostic check. The key columns are:

2. Check the overall status

The overall status follows a simple rule:

• Any fail → overall fail.
• Any warn (no fails) → overall warn.
• All pass → overall pass.

If the overall status is pass, no further action is required for the configured policy mode.

3. Triage failing checks

Focus on fail rows first, then warn rows. Use the check phase to prioritise:

4. Common diagnostics and actions

Design matrix issues (P0)

Sampler quality (P1, MCMC only)

Residual behaviour (P1)

Boundary and variation checks (P1)

Identifiability gate (P1)

5. Review diagnostic plots

Cross-reference the numeric report with visual diagnostics in 40_diagnostics/:

• Residual diagnostics plot — check for patterns in residuals over time.
• Boundary hits plot — identify which parameters are constrained.
• Latent residual ACF plot — confirm autocorrelation structure.

See Diagnostics Plots for interpretation guidance.

6. Decide on next steps

7. Change policy mode if appropriate

If you are in early model development, consider switching to explore mode to relax thresholds:

diagnostics:
  policy_mode: explore

For production or audit runs, use publish (default) or strict.

Related pages

• Diagnostics Gates — full threshold tables and policy modes
• Diagnostics Plots — visual diagnostic outputs
• Run from YAML — end-to-end runner workflow

Compare Runs

Objective

Compare multiple DSAMbayes runner executions and select a candidate model for reporting or decision-making, using predictive scoring and diagnostic summaries.

Prerequisites

• Two or more completed runner run executions (MCMC fit method).
• Artefacts under 50_model_selection/ for each run (LOO summary, ELPD outputs).
• Familiarity with Diagnostics Gates and Model Selection Plots.

Steps

1. Collect run directories

Identify the run directories to compare:

results/20260228_083808_blm_synth_kpi_os_hfb01/
results/20260228_084410_blm_synth_kpi_os_hfb01/
results/20260228_084602_blm_synth_kpi_os_hfb01/

2. Compare ELPD scores

The compare_runs() helper ranks runs by expected log predictive density (ELPD):

library(DSAMbayes)
comparison <- compare_runs(
  run_dirs = c(
    "results/20260228_083808_blm_synth_kpi_os_hfb01",
    "results/20260228_084410_blm_synth_kpi_os_hfb01"
  )
)
print(comparison)

The output ranks runs by ELPD (higher is better) and reports Pareto-k diagnostics.

3. Check Pareto-k reliability

Examine the loo_summary.csv in each run’s 50_model_selection/ folder:

cat results/<run_dir>/50_model_selection/loo_summary.csv

Key metrics:

If many observations have high Pareto-k values, the LOO approximation is unreliable for that run. Consider time-series cross-validation as an alternative.

4. Review time-series CV (if available)

If diagnostics.time_series_selection.enabled: true was configured, check:

cat results/<run_dir>/50_model_selection/tscv_summary.csv

This provides expanding-window blocked CV scores (holdout ELPD, RMSE, SMAPE) that are more appropriate for time-series data than standard LOO.

5. Cross-reference diagnostics

For each candidate run, check the diagnostics overall status:

head -1 results/<run_dir>/40_diagnostics/diagnostics_report.csv

A model with better ELPD but failing diagnostics should not be preferred over a model with slightly lower ELPD and passing diagnostics.

6. Compare fit quality visually

Review the fit time series and scatter plots in 20_model_fit/ for each run:

• Fit time series — does the model track the observed KPI?
• Fit scatter — is the predicted-vs-observed relationship close to the diagonal?
• Posterior forest — are coefficient estimates reasonable and well-identified?

7. Selection decision matrix

8. Record the selection

Document the selected run directory and rationale. If using the runner for release evidence, the selected run’s artefacts form part of the evidence pack.

Caveats

• ELPD is not causal validation. Predictive scoring measures in-sample predictive quality, not whether the model identifies causal media effects correctly.
• Pooled models do not support time-series CV (rejected by config validation).
• MAP-fitted models do not produce LOO diagnostics. Use MCMC for model comparison.

Related pages

• Diagnostics Gates — diagnostic quality checks
• Model Selection Plots — Pareto-k, LOO-PIT, ELPD influence plots
• Run from YAML — executing runs for comparison
• Config Schema — model_selection.* YAML keys

Debug Run Failures

Objective

Diagnose and resolve the most common failure modes encountered when running DSAMbayes via the YAML/CLI runner.

Prerequisites

• A failed runner execution (non-zero exit code or missing artefacts).
• Access to the terminal output or log from the failed run.
• Familiarity with CLI Usage and Config Schema.

Triage by failure stage

Stage 0: Config resolution failures

Symptoms: runner exits immediately after validate or at the start of run; no run directory created or only 00_run_metadata/ is present.

Quick check:

Rscript scripts/dsambayes.R validate --config config/my_config.yaml

If validate passes, the config is structurally valid.

Stage 1: Stan compilation failures

Symptoms: runner reports compilation errors after “Compiling model”; may reference C++ or Stan syntax errors.

Quick check:

mkdir -p .cache
export XDG_CACHE_HOME="$PWD/.cache"

Stage 2: Data preparation failures

Symptoms: runner fails after compilation but before sampling; error messages reference prep_data_for_fit, model.frame, or scaling.

Stage 3: Sampling failures

Symptoms: runner fails during rstan::sampling() or rstan::optimizing(); may report Stan runtime errors.

Stage 4: Post-fit artefact failures

Symptoms: runner completes sampling but some artefact folders are empty or missing files.

Quick check:

find results/<run_dir> -type f | sort

Compare against the expected artefact list in Output Artefacts.

Stage 5: Plot generation failures

Symptoms: CSV artefacts are present but PNG plot files are missing.

General debugging steps

1. Read the full error message. DSAMbayes uses cli::cli_abort() with descriptive messages that identify the failing function and parameter.

2. Check the resolved config. If a run directory was created, inspect 00_run_metadata/config.resolved.yaml to see what defaults were applied.

3. Check session info. Inspect 00_run_metadata/session_info.txt for package version mismatches.

4. Clear the Stan cache. Stale compiled models can cause unexpected failures:

   rm -rf .cache/dsambayes/

5. Run validate before run. Always validate first to catch config errors before committing to a full MCMC run.

6. Reduce iterations for debugging. Use a minimal config with fit.mcmc.iter: 200 and fit.mcmc.warmup: 100 to iterate quickly on formula and data issues.

Related pages

• CLI Usage — exit codes and error modes
• Config Schema — YAML key reference
• Output Artefacts — expected artefact contract
• Install and Setup — environment prerequisites
• Diagnostics Gates — post-fit quality checks