Evaluation Framework
Phillips adds a fifth level to Kirkpatrick: return on investment. The formula at the end is simple arithmetic. The hard part is the data that feeds it, and most of that data comes from asking participants the right questions while they still remember the program.
Jack Phillips developed the ROI Methodology in the 1970s and built the ROI Institute around it. The first four levels track Kirkpatrick with slightly different names: reaction and planned action, learning, application and implementation, business impact. Then comes the addition the model is named for.
ROI (%) = (net program benefits ÷ program costs) × 100
A program that costs £50,000 and produces £90,000 in benefits has a net benefit of £40,000 and an ROI of 80%. The related benefit-cost ratio would be 1.8. Any spreadsheet can do this part.
So why do so few training evaluations ever produce an ROI figure? Because the formula needs two numbers, and one of them, the monetary benefit attributable to the program, is genuinely hard to establish. That's where the model earns its keep.
Suppose sales rose 12% in the quarter after a negotiation program. How much of that rise was the program, and how much was the new pricing, the market, or the three people who joined the team? Phillips calls this isolating the effects, and it's the methodological core of the model. An ROI number that skips isolation is just a hopeful guess with a percent sign.
Control groups and trend-line analysis are the strongest isolation techniques. They're also unavailable to most consultancies most of the time. You rarely get to withhold training from half a client's sales team to keep them as a comparison group.
So the method most Phillips evaluations rely on in practice is participant estimation, adjusted for confidence. You ask the people who did the work three things: how much did the measure improve, what share of that improvement do you attribute to the program, and how confident are you in that attribution. Then you multiply through.
A participant reports their team's output improved about 20%. They attribute half of that to the program. They're 70% confident in that estimate.
20% × 50% × 70% = 7%
The claim entering the ROI calculation is a 7% improvement, not 20%. Phillips' guiding principles push every choice in this direction: when in doubt, take the lower number, and a claim with no data behind it counts as zero. The discounting is what makes the final figure defensible in front of a skeptical finance director.
Notice what that estimation chain is made of. Three survey questions. The barrier to Phillips-style evaluation was never the math.
Levels 1 to 3 look the same as a Kirkpatrick evaluation: reaction questions after the program, learning and application questions measured with retrospective before-and-after scales. Level 4 is where the Phillips-specific questions come in, usually six to twelve weeks after the program when effects have had time to show up.
Impact
"Since the program, how has [the business measure you targeted] changed? Estimate the improvement as a percentage."
Attribution
"What percentage of that improvement do you attribute to the program, as opposed to other factors like market conditions, staffing, or process changes?"
Confidence
"How confident are you in that estimate, from 0% (a pure guess) to 100% (certain)?"
Monetary value (optional)
"If you can, estimate what this improvement is worth to the organization over a year, and briefly explain how you arrived at the figure."
The explanation question matters more than it looks. Phillips' standard is that estimates need a credible basis, and an answer like "we close two extra deals a month at roughly £8k each" survives scrutiny in a way a bare number doesn't.
Phillips himself is clear on this: full ROI studies belong on a small fraction of programs, the expensive and strategically visible ones. The ROI Institute's own guidance puts it at roughly one program in ten or twenty. Running the complete machinery, isolation, data conversion, cost capture, on a half-day workshop costs more than the answer is worth.
For everything else, application data is usually what clients actually want. A leadership program where participants can show specific behavior change at eight weeks has made its case for renewal. The ROI level exists for the moments when the budget conversation moves to the CFO's office.
ImpactCheck collects the inputs: reaction, learning, and application data, plus the estimate chain at Level 4. The ROI calculation itself happens in your spreadsheet, with your cost data, under your assumptions. We'd rather be honest about that boundary than pretend a survey tool can read your program budget.
Impact estimates, attribution, confidence adjustment. Build the question chain once as a template and send it after every program.
Start your free trial30-day free trial. No credit card required.