Validation protocol
System Canvas · Validation protocol
Prospective plan, no results reported
Priority study: accessibility without loss of validity
Pre-register the principal comparison: System Canvas’s minimal canvas and selective drill-down versus properly taught established practice (for example IAD, VSM/ECM, 4EM or a credible enterprise-architecture workshop). Do not compare trained System Canvas participants with untrained control participants.
| Primary measure | Operationalization | Failure signal |
|---|---|---|
| Mechanism quality | Blind assessor scoring against case evidence and documented rival explanations | More unsupported arrows or omitted consequential mechanisms |
| Time-to-competence | Instruction time and attempts required to reach preset minimum quality | Similar or greater cost with no accuracy gain |
| View selection | Whether participants choose sufficient but not gratuitous lenses | Ritualistic six-panel completion or unjustified omissions |
| Transfer | Independently analyze a novel domain after a delay | Familiar examples look good, unfamiliar cases fail |
| Intervention quality | Blind review of causal fit, implementability, stakeholder effects and testability | Visually attractive but weaker proposals |
| Comprehension & workload | Explain a diagram to another participant; validated workload instrument when feasible | High subjective clarity paired with poor correctness |
| Calibration | Confidence compared to actual evidential support | More confident unsupported diagnoses |
Comparative protocol adjustments
- Prepare an evidence-backed case packet, explicit scoring rubric and independent expert reference analysis, including ambiguity and multiple defensible diagnoses.
- Randomize or counterbalance similarly experienced participants; keep case materials, training support and total available time equivalent. Report deviations.
- Compare minimum canvas, progressively disclosed six-lens interface and credible conventional methods separately where sample size allows.
- Use blind assessors to score reasoning and proposal quality; determine agreement, adjudicate disputes and report uncertainties.
- Capture interaction logs or observation notes: when people requested a deeper view, whether it changed a decision, and where visual simplicity concealed relevant complexity.
- In later field pilots, evaluate actual effects under ethical and feasible causal designs. Retain negative results and unintended harms.
Usability versus preference
Separately report (a) aesthetics / stated preference, (b) diagram comprehension, (c) valid independent diagnosis, and (d) real intervention outcome. Neither (a) nor (b) establishes (c) or (d). A test should explicitly include a misleadingly simple scenario to check whether learners recognize when escalation is necessary.
Suggested first prototype
Use the journalism/platform case with a one-page problem overview, selectable Structure/Behavior/Knowledge/Authority/Purpose/Outcomes views, an evidence ledger and an intervention comparison. Give an unfamiliar workplace governance case as the transfer task. Compare against a baseline with real IAD or 4EM materials, not only a generic brainstorming exercise.
Evaluation ladder
| Stage | Design | What it can show |
|---|---|---|
| A: usability | Small diverse learner pilot, think-aloud | Clarity, effort, confusion |
| B: reproducibility | Independent analysts analyze the same case | Agreement, coverage, evidence quality |
| C: comparison | Randomized method vs credible baseline; blinded assessors | Relative diagnostic and proposal quality |
| D: field intervention | Real implementation with suitable counterfactual | Actual outcomes and unintended effects |
A handful of workshop examples cannot establish causal efficacy. Early qualitative findings are still valuable for revising instructions.
Comparators
Stakeholder mapping plus causal-loop diagrams; Ostrom IAD; Beer’s VSM; 4EM; or an appropriately supported unstructured method. Match analyst time and case information.
Measures
- Diagnostic accuracy: supported true positives, unsupported mechanism claims, and coverage of independently checked mechanisms.
- Intervention quality: feasibility, mechanism fit, consent, accountability, distributional impact, safety and testability.
- Teachability: independent transfer to an unfamiliar case, retention and time-to-competence.
- Cost: analyst time and mental workload.
- Real impact: prespecified outcomes, harms, and appropriate causal comparison.
Use explicit assessor rubrics; document disagreements and uncertainty, including cases without knowable ground truth.
Case record
Case identifier, boundary, sources, consent, analyst method/version, diagrams, evidence labels, alternative hypotheses, interventions, predictions, assessors, outcomes, mistakes, and revised instructions.
First pilot
Use one common documented newsroom scenario. Randomize comparable learners into a standard causal-loop/stakeholder-mapping task or System Canvas. Provide equal time. Blind reviewers to method. Score evidence support, relevant mechanisms, governance awareness and intervention testability. Report negative and ambiguous results.
Ethics
Ensure privacy and consent. Prevent invented evidence, cherry-picking, instructor contamination, conflicts of interest and harmful live experiments. External review is needed for consequential public-sector experiments.