A pilot that can't be evaluated is not a pilot.
By the end of this lesson, you'll be able to design a pilot that produces a defensible answer at Day 30 with the right cohort, a scoped use case, and success criteria locked before the work starts.
The cohort you choose determines the data you get.
Cohort selection is a design decision, not an availability decision. The profile of your pilot developers directly shapes what the Day 30 readout can claim.
Early adopters and enthusiasts generate high usage data and positive qualitative feedback. But they don’t represent the typical developer. Engagement metrics from this group won’t extrapolate to the broader team, and won’t hold up to scrutiny in a CIO readout.
Developers who already own the work the use case targets. They have active tickets, familiar patterns, and a realistic starting baseline. Their before-and-after data is attributable and defensible. The metric story writes itself.
Scope and duration are design decisions, not defaults.
The default “run it for a month and see what happens” produces inconclusive data. Explicit scope and duration produce a closing argument.
Not “code modernization” but “Java-to-Spring Boot migration of the auth service module.” The more specific the scope, the cleaner the ROI attribution.
The developers actively working on the scoped use case, not those with the most calendar space.
Long enough for at least two complete task cycles (ticket-to-merge) per developer. For most migration work, 25–30 days is enough.
Scope creep during a pilot dilutes the ROI story. If a new use case emerges, log it for Phase 2 and protect the original scope.
Success criteria must exist before the pilot starts.
If you don’t know what success looks like at Day 30, you can’t design the pilot to prove it.
The metric you’re trying to move, measured before Day 1. PR cycle time, task hours, migration velocity. Without a baseline, there’s no before/after story.
The number that constitutes a win. “25% reduction in PR cycle time” is a claim. “PR cycle time improved” is not. Thresholds are what get quoted in renewal conversations.
How you will capture the data. Existing tooling (git, JIRA, PR analytics) is preferable to self-reporting. Objective data survives scrutiny.
The specific date you’ll assess. Day 30 is a milestone, not a deadline. Name it before the pilot starts so everyone knows what’s being measured when.
A success criterion you can’t measure before Day 1 is a hypothesis, not a commitment. Set the baseline before the kickoff meeting.
The Kellner Industries Pilot Design
Kellner Industries runs a 45-developer engineering team building enterprise infrastructure software. The CTO has approved a Claude Code pilot. Three decisions, up to 6 points.
Kellner has approved a pilot for their C++-to-Rust migration project. You’re selecting the cohort. The migration team has 8 developers. The broader engineering org has expressed interest.
Which cohort design produces the most defensible Day 30 result?
The 8-developer migration cohort is confirmed. Now define the primary success criterion.
Which success criterion gives you the most defensible story at Day 30?
By Week 2, two of the developers are excited about using Claude for a separate documentation project and want to expand the scope.
What do you do?
By the end of this course you’ll have a complete Delivery Plan Pack. This lesson contributes the pilot design section.
You can design the pilot. Now measure it.
A pilot without defensible success criteria is a trial run, not an engagement deliverable. Cohort ownership, baseline capture, and scope protection are the three decisions that determine whether you have a Day 30 story worth telling.