# Validation protocol

System Canvas · Validation protocol

Source: https://systems-canvas.phibkro.org/research/system-canvas-validation-protocol/

---

**Prospective plan, no results reported**

## Priority study: accessibility without loss of validity

**Pre-register the principal comparison:** System Canvas's minimal canvas and selective drill-down versus *properly taught* established practice (for example IAD, VSM/ECM, 4EM or a credible enterprise-architecture workshop). Do not compare trained System Canvas participants with untrained control participants.

| Primary measure | Operationalization | Failure signal |
|---|---|---|
| **Mechanism quality** | Blind assessor scoring against case evidence and documented rival explanations | More unsupported arrows or omitted consequential mechanisms |
| **Time-to-competence** | Instruction time and attempts required to reach preset minimum quality | Similar or greater cost with no accuracy gain |
| **View selection** | Whether participants choose sufficient but not gratuitous lenses | Ritualistic six-panel completion or unjustified omissions |
| **Transfer** | Independently analyze a novel domain after a delay | Familiar examples look good, unfamiliar cases fail |
| **Intervention quality** | Blind review of causal fit, implementability, stakeholder effects and testability | Visually attractive but weaker proposals |
| **Comprehension & workload** | Explain a diagram to another participant; validated workload instrument when feasible | High subjective clarity paired with poor correctness |
| **Calibration** | Confidence compared to actual evidential support | More confident unsupported diagnoses |

### Comparative protocol adjustments

1. Prepare an **evidence-backed case packet**, explicit scoring rubric and independent expert reference analysis, including ambiguity and multiple defensible diagnoses.
2. Randomize or counterbalance similarly experienced participants; keep case materials, training support and total available time equivalent. Report deviations.
3. Compare **minimum canvas**, **progressively disclosed six-lens interface** and credible conventional methods separately where sample size allows.
4. Use blind assessors to score reasoning and proposal quality; determine agreement, adjudicate disputes and report uncertainties.
5. Capture interaction logs or observation notes: when people requested a deeper view, whether it changed a decision, and where visual simplicity concealed relevant complexity.
6. In later field pilots, evaluate actual effects under ethical and feasible causal designs. Retain negative results and unintended harms.

### Usability versus preference

Separately report (a) aesthetics / stated preference, (b) diagram comprehension, (c) valid independent diagnosis, and (d) real intervention outcome. **Neither (a) nor (b) establishes (c) or (d).** A test should explicitly include a misleadingly simple scenario to check whether learners recognize when escalation is necessary.

### Suggested first prototype

Use the journalism/platform case with a one-page problem overview, selectable Structure/Behavior/Knowledge/Authority/Purpose/Outcomes views, an evidence ledger and an intervention comparison. Give an unfamiliar workplace governance case as the transfer task. Compare against a baseline with real IAD or 4EM materials, not only a generic brainstorming exercise.



## Evaluation ladder

| Stage | Design | What it can show |
|---|---|---|
| A: usability | Small diverse learner pilot, think-aloud | Clarity, effort, confusion |
| B: reproducibility | Independent analysts analyze the same case | Agreement, coverage, evidence quality |
| C: comparison | Randomized method vs credible baseline; blinded assessors | Relative diagnostic and proposal quality |
| D: field intervention | Real implementation with suitable counterfactual | Actual outcomes and unintended effects |

A handful of workshop examples cannot establish causal efficacy. Early qualitative findings are still valuable for revising instructions.

## Comparators

Stakeholder mapping plus causal-loop diagrams; Ostrom IAD; Beer's VSM; 4EM; or an appropriately supported unstructured method. Match analyst time and case information.

## Measures

- **Diagnostic accuracy:** supported true positives, unsupported mechanism claims, and coverage of independently checked mechanisms.
- **Intervention quality:** feasibility, mechanism fit, consent, accountability, distributional impact, safety and testability.
- **Teachability:** independent transfer to an unfamiliar case, retention and time-to-competence.
- **Cost:** analyst time and mental workload.
- **Real impact:** prespecified outcomes, harms, and appropriate causal comparison.

Use explicit assessor rubrics; document disagreements and uncertainty, including cases without knowable ground truth.

## Case record

Case identifier, boundary, sources, consent, analyst method/version, diagrams, evidence labels, alternative hypotheses, interventions, predictions, assessors, outcomes, mistakes, and revised instructions.

## First pilot

Use one common documented newsroom scenario. Randomize comparable learners into a standard causal-loop/stakeholder-mapping task or System Canvas. Provide equal time. Blind reviewers to method. Score evidence support, relevant mechanisms, governance awareness and intervention testability. Report negative and ambiguous results.

## Ethics

Ensure privacy and consent. Prevent invented evidence, cherry-picking, instructor contamination, conflicts of interest and harmful live experiments. External review is needed for consequential public-sector experiments.