An unexpected DOE confirmation result does not mean the experiment was wasted. It shows that the observed response and the prediction did not agree as expected, but the cause may be execution, changed conditions, measurement error, or model inadequacy. Before abandoning the predicted optimum or changing production settings, investigate systematically.
This article walks you through a structured troubleshooting process for unexpected confirmation results. You will learn how to verify factor settings, choose an appropriate comparison interval, identify model inadequacy through residual diagnostics, and design a targeted follow-up experiment. For a broader foundation, review Air Academy Associates' Design of Experiments overview.
Key Takeaways
- Treat an unexpected confirmation result as a prompt to investigate before changing production settings.
- Verify factor settings, process conditions, and the measurement system first.
- Use an interval that matches whether you are assessing one future result or a replicated mean.
- Check residuals and lack of fit before relying on the fitted model near the optimum.
- Use targeted sequential experiments when the original design cannot explain the result.
Understanding why a confirmation run fails requires more than reviewing the data. It requires a disciplined look at every assumption made during the original DOE.
What a Failed DOE Confirmation Run Actually Tells You
A confirmation experiment is a model-validation check performed after the DOE analysis. You repeat the selected settings and compare the observed response with the model prediction using a suitable uncertainty interval. NIST guidance on confirmation runs recommends repeating the best setting even when it appeared in the original design, so you can check that conditions have not changed and results remain close to the prediction.
When a result falls outside the appropriate interval, it is inconsistent with the fitted model under the conditions used. Treat that outcome as an unexpected confirmation result—not automatic proof that the entire DOE failed—because execution error, changed conditions, measurement error, or model inadequacy may be responsible.
Compare confirmation data with the fitted response surface and the assumptions behind it. NIST's DOE results interpretation checklist specifically includes confirmation settings and prediction intervals among the outputs to review.
One unexpected run is enough to pause before implementing a process change, but it is usually not enough to diagnose the cause. Replication matters: NIST recommends at least three runs at the selected settings so the team can estimate variability there.
1. Verify That Factor Settings Were Executed Correctly
The first question to ask is whether the confirmation run was actually conducted at the intended settings. Even a small deviation in temperature, pressure, or feed rate can shift the response meaningfully when you are operating near an optimum.
Check run logs, equipment calibration records, and operator notes before drawing any conclusions about the model itself.
2. Confirm That Process Conditions Have Not Changed
Process conditions can shift between the original DOE and the confirmation run. Raw material lots, ambient conditions, or equipment wear can all shift the baseline response without any change to the controlled factors.
If conditions changed, the model is not wrong—it is simply no longer describing the current process state.
3. Check Whether the Predicted Optimum Is Near a Model Boundary
Prediction uncertainty generally increases as settings move away from the center of the experimental region, and extrapolation beyond the studied ranges is especially risky. A wider interval reflects that added uncertainty; it does not by itself make an out-of-interval result more likely.
Review the location of the predicted optimum relative to your original factor ranges before concluding the model is flawed.
4. Assess Measurement System Adequacy
A measurement system that was adequate for detecting factor effects during the original DOE may not be precise enough for confirmation. Revisit measurement system analysis basics and, when appropriate, use a Gage R&R study guide to assess repeatability and reproducibility.
Measurement error inflates observed variation and can cause a valid prediction to appear as a failure.
5. Review Residual Diagnostics for Model Inadequacy
If settings, conditions, and measurement are verified, next review the model residual plots. Curvature, non-constant variance, unusual run-order patterns, or influential observations can signal violated assumptions or an inadequate fitted model.
This is where residual analysis becomes a direct investigative tool rather than a routine check.
With these five checks complete, you have enough information to decide whether to adjust settings, rebuild the model, or run a targeted follow-up experiment.
Comparing Confirmation Results Against the Prediction Interval for Model Validation
Compare the confirmation result with an interval that matches the quantity being assessed. A prediction interval is appropriate for a single future observation, while a confirmation mean from replicated runs requires an interval calculated for that mean; NIST's DOE interpretation guidance calls for reporting both the confirmation settings and prediction intervals.
The table below summarizes the three possible confirmation outcomes and the recommended response for each.
| Confirmation Outcome | What It Suggests | Recommended Next Step |
|---|---|---|
| Observed mean within prediction interval | Results are consistent with the model at the tested settings | Review practical risk and process requirements before implementation |
| Observed mean outside interval, settings verified | Possible model inadequacy, changed conditions, or measurement problems | Investigate residuals; consider augmenting the design |
| Observed mean outside interval, settings unverified | Execution error during confirmation run | Repeat the confirmation under controlled conditions |
A prediction interval is not the same as a confidence interval for the mean response. The interval must match the comparison: one future observation, a future mean based on replicated runs, or the fitted mean response.
Do not automatically compare a replicated mean with an interval intended for one future observation. Verify the interval definition, confidence level, number of confirmation replicates, and whether the software includes both model uncertainty and residual variation.
Planning a Targeted Follow-Up Experiment After a Failed Confirmation
An unexpected confirmation result does not automatically require discarding the entire DOE. A targeted follow-up experiment can often resolve the specific uncertainty revealed by the result, although the scope should depend on the suspected cause and the adequacy of the original design.
NIST recommends sequential experimentation when confirmation checks cannot explain the unexpected result. The original design provides a foundation, and the follow-up adds information where the fitted model may be inadequate.
Identify the Specific Region of Failure
Narrow your follow-up to the factor ranges where the confirmation run failed. Running additional points across the full design space wastes resources and dilutes the diagnostic signal you need.
Add Center Points or Axial Points to Detect Curvature
A two-level factorial without center points cannot assess curvature. Adding center points can detect overall pure quadratic curvature, but it cannot estimate separate quadratic terms; central composite design augmentation adds axial points that support estimation of a quadratic response surface.
Block the Follow-Up Against the Original Runs
Blocking separates the time-related variation between the original DOE and the follow-up experiment. This protects the combined analysis from being distorted by process drift or material lot changes that occurred between the two sets of runs.
Run at Least Three Confirmation Points in the Follow-Up
A single confirmation observation provides limited statistical power. Running at least three replicates at the selected settings, as NIST recommends, provides an estimate of variability there and a more reliable confirmation mean.
Air Academy Associates offers the Operational Design of Experiments Course, which covers experiment planning, response surface modeling, and confirmation strategy in an applied, hands-on format. Practitioners working through a failed confirmation will find the course directly relevant to the decisions described in this article.
When Changed Production Settings Are and Are Not Justified
One of the most consequential decisions after a failed confirmation is whether to implement the predicted optimum in production anyway. The answer depends on what the investigation revealed, not on whether the confirmation run was close to the prediction.
Implementing unvalidated production settings carries real risk, particularly in regulated industries like healthcare, aerospace, and government testing programs. A failed confirmation that has not been resolved is not a validated process change.
The conditions under which changing production settings is justified are specific.
- The confirmation failure was traced to an execution error, and a repeat run under controlled conditions passed the prediction interval check.
- The follow-up experiment confirmed the predicted optimum with a rebuilt or augmented model.
- The observed result, while outside the prediction interval, still meets the process goal and the deviation is understood and documented.
- A measurement system study confirmed that the original confirmation failure was attributable to gage error rather than process behavior.
If none of these conditions apply, the responsible path is to complete the follow-up experiment before changing any production settings.
For practitioners who need structured support reviewing failed confirmation evidence and planning a next experiment, Air Academy Associates provides one-to-one DOE coaching support with experienced Master Black Belts. This is direct, case-specific guidance—not a general course—and it is available for teams at any stage of a DOE investigation.
Air Academy Associates Resources for DOE Confirmation Runs
The resources below address specific skill gaps that practitioners encounter when a confirmation run fails. Each one connects to a distinct phase of the investigation described in this article.
Operational Design of Experiments Course
This course covers the full arc of a DOE study—from planning and factor selection through modeling, optimization, and confirmation. Practitioners who understand how confirmation fits into the broader experimental sequence are better positioned to diagnose failures when they occur.
- Covers response surface methods and prediction interval interpretation
- Addresses sequential experimentation and follow-up design strategies
- Applies directly to the confirmation planning decisions described in this article
View the Operational DOE Course
Validation Testing Short Course
This short course develops broader test-condition coverage skills and helps practitioners distinguish combinatorial testing strategies from statistical confirmation. Understanding that distinction is important when a confirmation run fails and the team needs to decide how much additional testing is warranted.
- Clarifies the boundary between validation testing and DOE confirmation
- Supports decisions about follow-up test scope after a failed confirmation
View the Validation Testing Course
Residuals Analysis Short Course
Residual diagnostics are the primary tool for detecting model inadequacy after a confirmation failure. This short course teaches practitioners to read residual plots, identify patterns that signal curvature or non-constant variance, and connect those patterns to specific corrective actions in the model.
- Directly applicable to Step 5 of the investigation process described above
- Builds the diagnostic skill needed to distinguish model error from process change
View the Residual Analysis Course
Coaching and One-to-One Training
When a confirmation run fails on an active project, a general course may not be the right immediate resource. One-to-one coaching provides direct review of the specific failed confirmation evidence and structured guidance on planning the next experiment. Sessions are conducted by Master Black Belts with hands-on DOE consulting experience.
- Case-specific review of your confirmation data and residual diagnostics
- Guidance on follow-up experiment design and blocking strategy
- Available for individual practitioners and project teams
Request One-to-One Coaching Support
Final Thoughts on DOE Confirmation Runs and Model Validation
A failed confirmation run is a diagnostic event, not a project failure. Working through settings verification, prediction interval assessment, residual analysis, and a targeted follow-up experiment gives you a structured path back to a validated model and defensible production settings. The investigation itself often produces more process knowledge than the original DOE—and that knowledge is what makes the confirmation step worth taking seriously in the first place.
Air Academy Associates offers Design of Experiments training led by experienced Master Black Belts. The training helps teams plan, analyze, and confirm experiments, including the follow-up work needed when confirmation results are unexpected. Get started today.
FAQs
Why Did the Best DOE Setting Fail Its Confirmation Run?
Most "failed" confirmations trace to one of four issues: (1) the optimum was outside the true safe operating window (extrapolation, unmodeled curvature, or constraints), (2) key noise factors shifted between the DOE and confirmation (materials, environment, operator, tool wear), (3) the measurement system wasn't stable or capable (gage error, recalibration, different test conditions), or (4) the model was incomplete (missing interaction, inadequate replication, or an outlier/influential point). In our DOE work, we treat confirmation as a model validation step—if it misses, we diagnose these fundamentals before changing settings.
What Should Be Checked Before Accepting New Production Settings?
Verify the optimum is practical and repeatable by checking: measurement system adequacy (MSA, calibration, resolution), process stability (control charts, warm-up/tooling effects), factor settings can be held tightly (tolerances, setup methods), raw material and environmental ranges match the DOE, the predicted gain is meaningful versus normal variation (confidence/prediction intervals), and the change passes risk and compliance requirements (FMEA, safety, customer specs). Air Academy Associates typically recommends a short, structured pre-implementation checklist plus a controlled pilot run to confirm performance under real operating conditions.
When Should a Failed Confirmation Trigger a Follow-Up Experiment?
Run a follow-up experiment when the confirmation result falls outside the model's prediction interval, the direction of the effect reverses or is inconsistent across repeats, you discover a likely missing factor or interaction, the optimum lies near a boundary/constraint, or the process/noise conditions differ from the original DOE (new supplier, seasonality, equipment changes). A targeted augmentation—adding center points, replicates, or a small response-surface or split-plot follow-up—usually resolves the gap quickly and is a standard best practice in high-impact DOE programs.
Related Articles
