Before We Start
Why "did the policy work?" is genuinely underspecified
Asking whether a policy "worked" sounds like a single, simple question — but genuine policy evaluation requires answering four distinct, separate questions: did it achieve its goals, at what cost, for whom, and was the change large enough to actually solve the problem? A policy can score well on some of these criteria while genuinely failing on others.
💡 The Real Binding Constraint Isn't Technical
Even a policy that scores excellently across all four evaluation criteria can still fail to be adopted or sustained — political feasibility is the genuine binding constraint on real-world policy, not simply a secondary consideration to be addressed after the technical analysis is complete.
Mnemonic
REIA — the four evaluation criteria
Effectiveness
Did the program achieve its stated goals?
The most direct question — comparing the program's actual outcomes against what it explicitly set out to accomplish.
Efficiency
At what cost per unit of outcome?
Even an effective program might not be an efficient use of resources — this criterion specifically asks whether the resources spent could have produced more benefit if used differently.
Equity
Who benefits and who bears costs — distributed fairly?
A specific, distinct question from either effectiveness or efficiency — a program could be both effective and efficient overall while still distributing its benefits and costs in a genuinely unfair way across different groups.
Adequacy
Is the magnitude of change sufficient to solve the problem?
A genuinely distinct question from effectiveness — a program might be technically "effective" (achieving statistically significant improvement) while still being inadequate in magnitude to actually solve the underlying problem it was meant to address.
💊 Two additional evaluation concepts worth knowing directly: formative evaluation happens DURING implementation (improving the program as you go), while summative evaluation happens AFTER the program concludes (assessing whether to continue or cut it) — these represent genuinely different timing and purpose, not simply two names for the same evaluation activity. Randomized control trials (RCTs) are considered the gold standard for establishing genuine causal evidence of a program's effects.
⚖️ Applying the Framework — Evaluating a Program Across All Four Criteria
A job training program successfully increases participants' employment rates by a statistically significant amount (meeting its stated goal), but at a very high cost per person trained, and the benefits accrue disproportionately to participants who were already relatively advantaged before entering the program, with only a modest overall improvement given the scale of the underlying unemployment problem.
Score the Program Across Each Criterion Separately
Effectiveness: high (met its stated employment goal). Efficiency: low (very high cost per outcome). Equity: concerning (benefits accrued disproportionately to already-advantaged participants). Adequacy: low (modest improvement relative to the underlying problem's actual scale). This program scores genuinely differently across all four criteria — demonstrating exactly why a single "did it work?" verdict is insufficient for complete evaluation.
Draw the Complete, Nuanced Conclusion
Rather than a simple "yes it worked" or "no it failed" verdict, a complete evaluation would report this program as effective but inefficient, inequitable, and inadequate relative to the scale of the problem — four genuinely distinct findings that together provide a much more complete and useful picture than any single overall verdict could. This is precisely why the REIA framework's four separate criteria matter for genuine, complete policy evaluation.
📌 Exam Application
Policy evaluation questions test both the four distinct criteria and the formative/summative distinction:
Criteria recall: "What are the four criteria for evaluating whether a public policy is working?" → Effectiveness, efficiency, equity, adequacy.
Distinction: "What is the difference between formative and summative evaluation?" → Formative occurs during implementation (improving as you go); summative occurs after the program concludes (deciding whether to continue or cut it).
Real constraint: "What is the real binding constraint on policy, beyond the technical evaluation criteria?" → Political feasibility.
⚠️ The Trap — Treating "Effectiveness" as the Only Relevant Evaluation Criterion
Because "did it achieve its goal?" (effectiveness) is the most intuitive, direct question, it's tempting to treat it as the complete evaluation. But a program can be effective while still being inefficient, inequitable, or inadequate — genuinely separate, equally important dimensions this framework specifically requires evaluating.
The safeguard: Always evaluate all four REIA criteria separately, rather than treating effectiveness alone as sufficient for a complete policy evaluation.
✓ Quick Self-Test
Answer before checking:
1. What does REIA stand for?
2. What is the difference between effectiveness and adequacy?
3. What is the difference between formative and summative evaluation?
4. What is considered the gold standard for establishing causal evidence?
Answers:
1. Effectiveness, Efficiency, Equity, Adequacy (Results is sometimes used for Effectiveness).
2. Effectiveness asks if the program achieved its stated goals; adequacy asks if the magnitude of change is sufficient to actually solve the underlying problem.
3. Formative occurs during implementation to improve the program; summative occurs after the program to decide whether to continue or cut it.
4. Randomized control trials (RCTs).
Next Lesson
CASE — Regulatory Approaches
→