Agile helps AI teams stay productive when you treat the model as a hypothesis, wrap every work item in a measurable acceptance test, and keep experiments as short as your smallest useful training run. The method fails on a fixed two-week sprint of design up front, because data quality and model behavior don't cooperate with a calendar. Make the sprint the outer container and your experiment cadence the inner one, and the usual stall β a team three months in with no deployable model β stops happening.
Here's the working definition: an AI sprint is a two-week container holding three to five timeboxed experiments, each with a go/no-go metric agreed before training starts. The container is Agile; the experiments are quasi-scientific method. Mixing those two rhythms is where most teams either drown in ceremony or lose all control.
The change sounds small and it isn't. Standard Scrum assumes you can estimate a story and finish it. Training a model is closer to a lab session: you don't know how long the answer takes because you don't know if there is an answer. Productivity comes from shortening the loop between "we think this works" and "we can prove it doesn't."
Map the AI Workflow Into Five Agile Work Types Before You Plan Anything
Start by splitting every possible task into labels that behave differently when estimated. The five that cover most model development: data work, feature and pipeline engineering, experiment, evaluation harness, and deployment. Each has a different failure mode and a different definition of done, so lumping them into one backlog column is why velocity readings on AI teams feel meaningless.
Data work almost never finishes in a story-size chunk. A stratification fix on a training set can take two days or ten depending on what the audits turn up, so plan it in capacity ranges, not point estimates. Pipeline engineering behaves closer to ordinary software and can be estimated normally. The experiment type is the one to timebox ruthlessly.
The evaluation harness deserves its own work type because teams under-invest in it and then can't trust their own results. Build the test set, the metric computation, and the slice reporting before the model is worth evaluating. If you can't score a model reproducibly, you can't run a productive sprint β you can only run a hopeful one.
Deployment work in AI includes model packaging, serving, monitoring, and drift detection. Treat those as first-class backlog items, not a phase at the end. Teams that leave deployment out of the definition of done ship a model into a notebook and call it a release. A table makes the difference visible when you're planning.
| Work type | Typical planning shape | Definition of done | Most common failure |
|---|---|---|---|
| Data work | Capacity range, not points | Documented audit and versioned dataset | Silent label drift between versions |
| Pipeline engineering | Story points, sprint-sized | Reproducible run with config checked in | Environment drift between machines |
| Experiment | Fixed timebox, one metric | Metric recorded against a pre-agreed threshold | Metric chosen after results |
| Evaluation harness | Story points, one sprint | Slice-level report reproducible by another engineer | Single aggregate number hides regressions |
| Deployment | Story points plus rollout plan | Model serving, monitored, with a rollback path | Monitoring added as an afterthought |
The table also sets your planning conversation. When a work type has a capacity range rather than a point estimate, the sprint plan becomes a set of commitments plus a set of bets, and everyone can see which is which.
Write Acceptance Tests as Model Contracts, Not as Accuracy Goals
An acceptance test for a model is not "accuracy above 90 percent." It's a named test set, a named metric, a named slice breakdown, and a threshold agreed before training starts. Write it on the ticket. Real number, real dataset, real slices. If the team can't write that on the ticket, the ticket is a research question, and research questions belong in a separate, explicitly funded stream.
The reason is simple: metric choice made after results is a decision made with the answer visible. Teams that define the threshold in advance catch their own overfitting to the test set, and reviewers can see the number before the score. It's the cheapest discipline in the whole process and the one most often skipped.
Slice-level thresholds matter more than the headline figure for most production models. A model that hits its overall target and fails on the smallest user segment can pass review and cause a real problem in production. Name the slices on the ticket: language, region, device class, long-tail category, or whatever segments your product actually serves. Five or six named slices is enough to start.
Failure handling belongs in the contract too. If the experiment misses threshold, the ticket closes as a no-go with the result recorded β the next ticket starts from that information. If it passes, the question becomes how to productionize. The no-go path is where the productivity gain actually lives, since most experiments should fail and the team shouldn't treat that as a wasted sprint.
Timebox Experiments to Your Smallest Useful Training Run
Your experiment cadence should be set by how long the smallest genuinely informative training run takes, not by the sprint calendar. On small tabular and vision models, that run is often under two hours, and you can fit four to six experiments into a two-week sprint. On large language-model fine-tunes, a single meaningful run can take a day or more of compute, and one or two experiments per sprint is honest.
Work out your number before planning: run a deliberately small configuration once, time it, and use that as the experiment budget unit. If a training run takes six hours on your own hardware and you have four GPUs of capacity, you can plan six to eight runs across the sprint with reasonable confidence. That arithmetic belongs in the sprint plan, not discovered mid-sprint.
Two failure cases come up repeatedly. The first is the open-ended experiment ticket that gets carried over three sprints because every result suggests another variation. Cap it: one sprint, one hypothesis, then decide. The second is the team that schedules experiments back-to-back with no evaluation time, then discovers the results can't be compared because the harness changed. Budget evaluation hours as a separate line.
Reduced-fidelity experimentation deserves a fixed slot in every sprint. Train on a subsample, a smaller model, or fewer epochs to screen ideas quickly, and only promote promising configurations to a full run. That screening step cuts wasted full-run compute and keeps the sprint loop honest about what's actually been tested.
Cut Ceremonies to Four Standup Minutes and One Decisions-Only Review
AI teams carry a specific ceremony debt. Daily standup becomes a data war story. Sprint review becomes a demo of a notebook, not a decision. Retrospective becomes a discussion of the data pipeline. Trim each to its decision function and the rest of the sprint opens up.
The daily standup answer that matters in a model sprint is one of three: what result did I see, what am I running next, and what did I block on. Anything else β data gossip, model theory, infrastructure stories β gets a follow-up with one other person after standup. Four minutes total for a team of five is achievable when the format is enforced.
The sprint review should be a decisions-only meeting. Each ticket gets one of three outcomes: keep, kill, or scale. No demos of intermediate notebooks. That's the whole agenda. If a stakeholder needs a demo, schedule a separate demo session β those are cheap and don't pollute the decision.
Retrospective in an AI sprint works better as a short review of the failed experiments than as a team-mood session. Ask which hypotheses were wrong and what was learned, and whether any test was agreed too late. Five minutes of that beats thirty minutes of process complaints, and it produces a concrete change to the next sprint's contract writing.
Name the Metrics That Don't Belong in a Sprint Goal
Velocity, story points, and sprint burndown are weak signals for AI teams and can be gamed by relabeling tickets. They don't measure the useful thing, which is how many hypotheses were resolved. Track resolved experiments per sprint, share of experiments that produced a keep/kill/scale decision, and production-ready models shipped per quarter. Those are the numbers that tell you whether Agile is doing its job.
Resolved experiments per sprint is the simplest and most honest metric. An experiment is resolved when the pre-agreed threshold has been checked against a recorded result, whether it passed or failed. That prevents the carryover spiral and prevents relabeled tickets from inflating output. Teams typically move from one or two resolved experiments per sprint early on to four or five once the harness stabilizes.
The share of experiments that produced a decision is a quality check on the process itself. If a large fraction of experiments end without a keep/kill/scale outcome, the contract writing is failing β the threshold was never clear. Fix the tickets before adding headcount.
Finally, track production-ready models per quarter, not per sprint. Model deployment cycles don't fit two-week increments, and trying to force them creates fake releases. Teams that accept this get more real deployments, not fewer, because the release decision is made on evidence instead of calendar pressure.
Run a Two-Week Sprint in This Order, With Five Named Checks
Here's the working sequence. It assumes a team of four to six engineers, one product owner, and a data and evaluation budget already in place. The order matters because it protects the experiment window from the planning overhead that would otherwise eat it.
1: Day 1, sprint planning (90 minutes). Pull three to five tickets, each with a written contract: named test set, named metric, named slices, threshold, timebox. Any ticket without a contract goes back to the backlog. 2: Day 1β2, environment and data check. Confirm the training environment is reproducible and the test set version is pinned in the ticket. Run one short smoke test per planned ticket to prove the pipeline is alive. 3: Day 2β8, experiment window. Run the timeboxed experiments in parallel where compute allows. Log results as they finish. Reserve the last two hours of each day for evaluation, not training. 4: Day 9, evaluation and slice review. Score each model against the pre-agreed metric and slices. Any result outside the contract is a no-go, no discussion. 5: Day 10, sprint review and decision. Each ticket gets keep, kill, or scale. Update the backlog, plan the next sprint's three to five contracts, and close the sprint.
If the pipeline is not reproducible on day two, stop the experiment window and fix the environment first. Running experiments on a broken pipeline produces results you'll re-derive later. If evaluation is behind, borrow time from the experiment window rather than shortening the review β the review is what makes the sprint a sprint.
Two failure scenarios to plan for. If a training run blows its timebox by more than 50 percent, kill it and log why; the cost is information, not failure. If two experiments collide on the same test set and can't be compared, mark both as provisional and add a comparison experiment to the next sprint.
Fix the Data Versioning Gap That Silently Breaks Sprint Velocity
Every AI team eventually ships a model that can't be reproduced because the training set changed underneath it. Fix that before chasing velocity. A model is only as reproducible as the data snapshot, config, and code commit that produced it, and if any of the three is unversioned, the sprint results can't be trusted.
Concretely, pin a data snapshot hash per experiment ticket and record it alongside the model artifact. Record the code commit and the training config in the same record. A single row per experiment β snapshot hash, commit, config, metric, slices, decision β makes the whole sprint auditable in minutes.
The failure case is ugly and common. A team fixes a label bug mid-sprint, retrains, and beats the previous best by a wide margin. Nobody remembers that the previous run used the old labels because the ticket said "same data." The result gets reported as a model improvement when it's a data improvement, and the next sprint's plan is built on a false baseline.
Pin the snapshot before the experiment starts and don't change it mid-run. New labels become a new ticket with a new snapshot, and the comparison between the two is itself a valid experiment. This sounds slow and it saves weeks.
Budget Compute the Way You Budget Engineer Time
Compute is the sprint's second capacity, and most teams plan it implicitly. Make it explicit. Reserve a compute budget per sprint β a named GPU-Hours figure or a dollar figure β and split it across the planned experiments. If the budget runs out, the sprint closes with what's been resolved, not with more training.
The arithmetic is defensible when you work it from your own figures. A six-hour training run at four GPUs uses 24 GPU-Hours. Six runs across a two-week sprint uses 144 GPU-Hours plus evaluation time. Add a 20 to 30 percent buffer for retries and screening experiments, and the sprint compute plan writes itself.
Price daily compute against deliverable value, using your own contract rate. If a GPU-hour costs you X and an experiment needs 24 of them, the cost per experiment is 24X. That number should sit in the sprint plan. When it's invisible, teams spend compute on speculative experiments and skip the evaluation work that makes the results useful.
Reserve a fixed share of the sprint's compute for the evaluation harness and the screening pass. Both deliver more per hour than a full training run on an unvalidated idea, and both get skipped when the budget is a shared pool.
Build Inference and Monitoring Into Every Definition of Done
A model isn't done when it scores well on the test set. It's done when it serves, when its inference path is reproducible, and when someone can see it drift. Build those requirements into the definition of done for every ticket that could become a release, and the production gap stops being a surprise.
Inference reproducibility is the piece teams miss. The same input must produce the same output under the same model version and serving config, and the serving config must be versioned. If the inference pipeline changes between training and serving β different tokenizer, different preprocessing, different library version β the model's reported test score doesn't describe its production behavior.
Monitoring at launch means named drift checks and a named owner. Define which input distribution checks run, who gets paged when a check fires, and what the rollback path is. Reviewers should be able to find those in the release ticket before approving it.
Cost the inference side too. Serving cost per thousand requests at the planned throughput belongs on the release ticket. A model that passes accuracy checks and costs more than the business case allows isn't done either.
FAQ
How many experiments should an AI team run per sprint?
Three to five resolved experiments per two-week sprint is a healthy target for a team of four to six engineers with a stable evaluation harness. The right number is set by your smallest useful training run: if one run is two hours, four to six fits; if it's a day, one or two is honest. Track resolved experiments, not started ones.
Does AI model development fit standard Scrum?
Partially. AI work fits Scrum's cadence and roles but not its estimating model, because data and model work resist story-point estimation. Keep the sprint container and the review, replace point-based velocity with resolved experiments, and add a written model contract to every ticket. Teams that run standard Scrum unchanged tend to carry experiment tickets across sprints indefinitely.
What are the biggest mistakes in applying Agile to machine learning?
The most expensive mistake is defining the metric after seeing results, which makes the number meaningless. The second is skipping the evaluation harness until late, which makes every sprint's result provisional. The third is leaving deployment and monitoring off the definition of done, which turns a model demo into a false release.
How do you handle a model that keeps failing evaluation?
Give it a hard stop: one sprint, one hypothesis, a documented no-go. A model that misses its pre-agreed threshold on a pinned test set with a working harness has told you something useful β the hypothesis was wrong, the data doesn't support it, or the slice definition needs work. Carry the information forward as a new ticket, not the old one.
How long is an AI sprint compared to a software sprint?
Two weeks remains the most common container, and it works when experiments fit inside it. If a single training and evaluation run takes more than a working week, either shrink the model or split the work into a screening sprint and a full-run sprint. Trying to force a ten-day training cycle into a two-week sprint will fail every time.
Is an Agile board even useful for model work?
Yes, with a different column structure. Use columns that match the work types β contract, in experiment, in evaluation, decided β instead of the standard to-do, in-progress, done. The board's real value in AI work is showing how many tickets are stuck in evaluation, which is where most sprint time quietly disappears. Aim to keep that column moving.
How should you review a model that passed its test but hurts a production slice?
Treat the slice regression as a blocking issue even if the headline metric passes. Check whether the slice was named in the contract and whether the threshold was slice-specific; if not, add it to the contract and rerun the evaluation. Do not release on the headline number alone β the slice regression is the evidence that the metric was too aggregate.