10.2  Stepwise Regression
In this section, we learn about the stepwise regression procedure. While we will soon learn the finer details, the general idea behind the stepwise regression procedure is that we build our regression model from a set of candidate predictor variables by entering and removing predictors — in a stepwise manner — into our model until there is no justifiable reason to enter or remove any more.
Our hope is, of course, that we end up with a reasonable and useful regression model. There is one sure way of ending up with a model that is certain to be underspecified — and that's if the set of candidate predictor variables doesn't include all of the variables that actually predict the response. This leads us to a fundamental rule of the stepwise regression procedure — the list of candidate predictor variables must include all of the variables that actually predict the response. Otherwise, we are sure to end up with a regression model that is underspecified and therefore misleading.
An example
Let's learn how the stepwise regression procedure works by considering a data set that concerns the hardening of cement. Sounds interesting, eh? In particular, the researchers were interested in learning how the composition of the cement affected the heat evolved during the hardening of the cement. Therefore, they measured and recorded the following data (cement.txt) on 13 batches of cement:
 Response y: heat evolved in calories during hardening of cement on a per gram basis
 Predictor x_{1}: % of tricalcium aluminate
 Predictor x_{2}: % of tricalcium silicate
 Predictor x_{3}: % of tetracalcium alumino ferrite
 Predictor x_{4}: % of dicalcium silicate
Now, if you study the scatter plot matrix of the data:
you can get a hunch of which predictors are good candidates for being the first to enter the stepwise model. It looks as if the strongest relationship exists between either y and x_{2} or between y and x_{4} — and therefore, perhaps either x_{2} or x_{4} should enter the stepwise model first. Did you notice what else is going on in this data set though? A strong correlation also exists between the predictors x_{2} and x_{4}! How does this correlation among the predictor variables play out in the stepwise procedure? Let's see what happens when we use the stepwise regression method to find a model that is appropriate for these data.
Note. The number of predictors in this data set is not large. The stepwise procedure is typically used on much larger data sets, for which it is not feasible to attempt to fit all of the possible regression models. For the sake of illustration, the data set here is necessarily small, so that the largeness of the data set does not obscure the pedagogical point being made.
The procedure
Again, before we learn the finer details, let me again provide a broad overview of the steps involved. First, we start with no predictors in our "stepwise model." Then, at each step along the way we either enter or remove a predictor based on the partial Ftests — that is, the ttests for the slope parameters — that are obtained. We stop when no more predictors can be justifiably entered or removed from our stepwise model, thereby leading us to a "final model."
Now, let's make this process a bit more concrete. Here goes:
Starting the procedure. The first thing we need to do is set a significance level for deciding when to enter a predictor into the stepwise model. We'll call this the AlphatoEnter significance level and will denote it as α_{E}. Of course, we also need to set a significance level for deciding when to remove a predictor from the stepwise model. We'll call this the AlphatoRemove significance level and will denote it as α_{R}. That is, first:
 Specify an AlphatoEnter significance level. This will typically be greater than the usual 0.05 level so that it is not too difficult to enter predictors into the model. Many software packages — Minitab included — set this significance level by default to α_{E} = 0.15.
 Specify an AlphatoRemove significance level. This will typically be greater than the usual 0.05 level so that it is not too easy to remove predictors from the model. Again, many software packages — Minitab included — set this significance level by default to α_{R} = 0.15.
Step #1. Once we've specified the starting significance levels, then we:
 Fit each of the onepredictor models — that is, regress y on x_{1}, regress y on x_{2}, ..., and regress y on x_{p1}.
 Of those predictors whose ttest Pvalue is less than α_{E} = 0.15, the first predictor put in the stepwise model is the predictor that has the smallest ttest Pvalue.
 If no predictor has a ttest Pvalue less than α_{E} = 0.15, stop.
Step #2. Then:
 Suppose x_{1} had the smallest ttest Pvalue below α_{E} = 0.15 and therefore was deemed the "best" single predictor arising from the the first step.
 Now, fit each of the twopredictor models that include x_{1} as a predictor — that is, regress y on x_{1}and x_{2}, regress y on x_{1}and x_{3}, ..., and regress y on x_{1} and x_{p1}.
 Of those predictors whose ttest Pvalue is less than α_{E} = 0.15, the second predictor put in the stepwise model is the predictor that has the smallest ttest Pvalue.
 If no predictor has a ttest Pvalue less than α_{E} = 0.15, stop. The model with the one predictor obtained from the first step is your final model.
 But, suppose instead that x_{2} was deemed the "best" second predictor and it is therefore entered into the stepwise model.
 Now, since x_{1} was the first predictor in the model, step back and see if entering x_{2} into the stepwise model somehow affected the significance of the x_{1 }predictor. That is, check the ttest Pvalue for testing β_{1} = 0. If the ttest Pvalue for β_{1} = 0 has become not significant — that is, the Pvalue is greater than α_{R} = 0.15 — remove x_{1} from the stepwise model.
Step #3. Then:
 Suppose both x_{1} and x_{2} made it into the twopredictor stepwise model and remained there.
 Now, fit each of the threepredictor models that include x_{1} and x_{2} as predictors — that is, regress y on x_{1}, x_{2}, and x_{3}, regress y on x_{1}, x_{2}, and x_{4}, ..., and regress y on x_{1}, x_{2}, and x_{p1}.
 Of those predictors whose ttest Pvalue is less than α_{E} = 0.15, the third predictor put in the stepwise model is the predictor that has the smallest ttest Pvalue.
 If no predictor has a ttest Pvalue less than α_{E} = 0.15, stop. The model containing the two predictors obtained from the second step is your final model.
 But, suppose instead that x_{3} was deemed the "best" third predictor and it is therefore entered into the stepwise model.
 Now, since x_{1} and x_{2} were the first predictors in the model, step back and see if entering x_{3} into the stepwise model somehow affected the significance of the x_{1 }and x_{2} predictors. That is, check the ttest Pvalues for testing β_{1} = 0 and β_{2} = 0. If the ttest Pvalue for either β_{1} = 0 or β_{2} = 0 has become not significant — that is, the Pvalue is greater than α_{R} = 0.15 — remove the predictor from the stepwise model.
Stopping the procedure. Continue the steps as described above until adding an additional predictor does not yield a ttest Pvalue below α_{E} = 0.15.
Whew! Let's return to our cement data example so we can try out the stepwise procedure as described above.
The example again
To start our stepwise regression procedure, let's set our AlphatoEnter significance level at α_{E} = 0.15, and let's set our AlphatoRemove significance level at α_{R} = 0.15. Now, regressing y on x_{1}, regressing y on x_{2}, regressing y on x_{3}, and regressing y on x_{4}, we obtain:
Each of the predictors is a candidate to be entered into the stepwise model because each ttest Pvalue is less than α_{E} = 0.15. The predictors x_{2} and x_{4} tie for having the smallest ttest Pvalue — it is 0.001 in each case. But note the tie is an artifact of Minitab rounding to three decimal places. The tstatistic for x_{4} is larger in absolute value than the tstatistic for x_{2}—4.77 versus 4.69—and therefore the Pvalue for x_{4} must be smaller. As a result of the first step, we enter x_{4} into our stepwise model.
Now, following step #2, we fit each of the twopredictor models that include x_{4} as a predictor — that is, we regress y on x_{4} and x_{1}, regress y on x_{4} and x_{2}, and regress y on x_{4} and x_{3}, obtaining:
The predictor x_{2} is not eligible for entry into the stepwise model because its ttest Pvalue (0.687) is greater than α_{E} = 0.15. The predictors x_{1} and x_{3} are candidates because each ttest Pvalue is less than α_{E} = 0.15. The predictors x_{1} and x_{3} tie for having the smallest ttest Pvalue—it is < 0.001 in each case. But, again the tie is an artifact of Minitab rounding to three decimal places. The tstatistic for x_{1} is larger in absolute value than the tstatistic for x_{3}—10.40 versus 6.35—and therefore the Pvalue for x_{1} must be smaller. As a result of the second step, we enter x_{1} into our stepwise model.
Now, since x_{4} was the first predictor in the model, we must step back and see if entering x_{1} into the stepwise model affected the significance of the x_{4} predictor. It did not—the ttest Pvalue for testing β_{1} = 0 is less than 0.001, and thus smaller than α_{R} = 0.15. Therefore, we proceed to the third step with both x_{1} and x_{4} as predictors in our stepwise model.
Now, following step #3, we fit each of the threepredictor models that include x1 and x_{4} as predictors — that is, we regress y on x_{4}, x_{1}, and x_{2}; and we regress y on x_{4}, x_{1}, and x_{3}, obtaining:
Both of the remaining predictors—x_{2} and x_{3}—are candidates to be entered into the stepwise model because each ttest Pvalue is less than α_{E} = 0.15. The predictor x_{2} has the smallest ttest Pvalue (0.052). Therefore, as a result of the third step, we enter x_{2} into our stepwise model.
Now, since x_{1} and x_{4} were the first predictors in the model, we must step back and see if entering x_{2} into the stepwise model affected the significance of the x_{1} and x_{4} predictors. Indeed, it did—the ttest Pvalue for testing β_{4} = 0 is 0.205, which is greater than α_{R} = 0.15. Therefore, we remove the predictor x_{4} from the stepwise model, leaving us with the predictors x_{1} and x_{2} in our stepwise model:
Now, we proceed fitting each of the threepredictor models that include x_{1} and x_{2} as predictors — that is, we regress y on x_{1}, x_{2}, and x_{3}; and we regress y on x_{1}, x_{2}, and x_{4}, obtaining:
Neither of the remaining predictors—x_{3} and x_{4}—are eligible for entry into our stepwise model, because each ttest Pvalue—0.209 and 0.205, respectively—is greater than α_{E} = 0.15. That is, we stop our stepwise regression procedure. Our final regression model, based on the stepwise procedure contains only the predictors x_{1} and x_{2}:
Whew! That took a lot of work! The good news is that most statistical software—including Minitab—provides a stepwise regression procedure that does all of the dirty work for us. For example in Minitab v17, select Stat > Regression > Regression > Fit Regression Model, click the Stepwise button in the resulting Regression Dialog, select Stepwise for Method and select Include details for each step under Display the table of model selection details. Here's what the Minitab stepwise regression output looks like for our cement data example:
Minitab tells us that :
 a stepwise regression procedure was conducted on the response y and four predictors x_{1}, x_{2}, x_{3}, and x_{4}
 the AlphatoEnter significance level was set at α_{E} = 0.15 and the AlphatoRemove significance level was set at α_{R} = 0.15
The remaining portion of the output contains the results of the various steps of Minitab's stepwise regression procedure. One thing to keep in mind is that Minitab numbers the steps a little differently than described above. Minitab considers a step any addition or removal of a predictor from the stepwise model, whereas our steps—step #3, for example—considers the addition of one predictor and the removal of another as one step.
The results of each of Mintab's steps are reported in a column labeled by the step number. It took Minitab 4 steps before the procedure was stopped. Here's what the output tells us:
 Just as our work above showed, as a result of Minitab's first step, the predictor x_{4} is entered into the stepwise model. Minitab tells us that the estimated intercept ("Constant") b_{0} = 117.57 and the estimated slope b_{4} = 0.738. The Pvalue for testing β_{4} = 0 is 0.001. The estimate S, which equals the square root of MSE, is 8.96. The R^{2}value is 67.45% and the adjusted R^{2}value is 64.50%. Mallows' Cpstatistic, which we learn about in the next section, is 138.73. The output also includes a predicted R^{2}value, which we'll come back to in Section 10.5.
 As a result of Minitab's second step, the predictor x_{1} is entered into the stepwise model already containing the predictor x_{4}. Minitab tells us that the estimated intercept b_{0} = 103.10, the estimated slope b_{4} = 0.614, and the estimated slope b_{1} = 1.44. The Pvalue for testing β_{4} = 0 is < 0.001. The Pvalue for testing β_{1} = 0 is < 0.001. The estimate S is 2.73. The R^{2}value is 97.25% and the adjusted R^{2}value is 96.70%. Mallows' Cpstatistic is 5.5.
 As a result of Minitab's third step, the predictor x_{2} is entered into the stepwise model already containing the predictors x_{1} and x_{4}. Minitab tells us that the estimated intercept b_{0} = 71.6, the estimated slope b_{4} = 0.237, the estimated slope b_{1} = 1.452, and the estimated slope b_{2} = 0.416. The Pvalue for testing β_{4} = 0 is 0.205. The Pvalue for testing β_{1} = 0 is < 0.001. The Pvalue for testing β_{2} = 0 is 0.052. The estimate S is 2.31. The R^{2}value is 98.23% and the adjusted R^{2}value is 97.64%. Mallows' Cpstatistic is 3.02.
 As a result of Minitab's fourth and final step, the predictor x_{4} is removed from the stepwise model containing the predictors x_{1}, x_{2}, and x_{4}, leaving us with the final model containing only the predictors x_{1} and x_{2}. Minitab tells us that the estimated intercept b_{0} = 52.58, the estimated slope b_{1} = 1.468, and the estimated slope b_{2} = 0.6623. The Pvalue for testing β_{1} = 0 is < 0.001. The Pvalue for testing β_{2} = 0 is < 0.001. The estimate S is 2.41. The R^{2}value is 97.87% and the adjusted R^{2}value is 97.44%. Mallows' Cpstatistic is 2.68.
Does the stepwise regression procedure lead us to the "best" model? No, not at all! Nothing occurs in the stepwise regression procedure to guarantee that we have found the optimal model. Case in point! Suppose we defined the best model to be the model with the largest adjusted R^{2}value. Then, here, we would prefer the model containing the three predictors x_{1}, x_{2}, and x_{4}, because its adjusted R^{2}value is 97.64%, which is higher than the adjusted R^{2}value of 97.44% for the final stepwise model containing just the two predictors x_{1} and x_{2}.
Again, nothing occurs in the stepwise regression procedure to guarantee that we have found the optimal model. This, and other cautions of the stepwise regression procedure, are delineated in the next section.
Cautions!
Here are some things to keep in mind concerning the stepwise regression procedure:
 The final model is not guaranteed to be optimal in any specified sense.
 The procedure yields a single final model, although there are often several equally good models.
 Stepwise regression does not take into account a researcher's knowledge about the predictors. It may be necessary to force the procedure to include important predictors.
 One should not overinterpret the order in which predictors are entered into the model.
 One should not jump to the conclusion that all the important predictor variables for predicting y have been identified, or that all the unimportant predictor variables have been eliminated. It is, of course, possible that we may have committed a Type I or Type II error along the way.
 Many ttests for testing β_{k} = 0 are conducted in a stepwise regression procedure. The probability is therefore high that we included some unimportant predictors or excluded some important predictors.
It's for all of these reasons that one should be careful not to overuse or overstate the results of any stepwise regression procedure.
More examples
Let's close up our discussion of stepwise regression by taking a quick look at two more examples.
Example #1. Are a person's brain size and body size predictive of his or her intelligence? Interested in this question, some researchers (Willerman, et al, 1991) collected the following data (iqsize.txt) on a sample of n = 38 college students:
 Response (y): Performance IQ scores (PIQ) from the revised Wechsler Adult Intelligence Scale. This variable served as the investigator's measure of the individual's intelligence.
 Potential predictor (x_{1}): Brain size based on the count obtained from MRI scans (given as count/10,000).
 Potential predictor (x_{2}): Height in inches.
 Potential predictor (x_{3}): Weight in pounds.
A matrix plot of the resulting data looks like:
Using Minitab to perform the stepwise regression procedure, we obtain:
The output tells us:
 The first predictor entered into the stepwise model is Brain. Minitab tells us that the estimated intercept is 4.7 and the estimated slope for Brain is 1.177. The Pvalue for testing β_{Brain} = 0 is 0.019. The estimate S is 21.2, the R^{2}value is 14.27%, the adjusted R^{2}value is 11.89%, and Mallows' Cpstatistic is 7.34.
 The second and final predictor entered into the stepwise model is Height. Minitab tells us that the estimated intercept is 111.3, the estimated slope for Brain is 2.061, and the estimated slope for Height is 2.730. The Pvalue for testing β_{Brain} = 0 is 0.001. The Pvalue for testing β_{Height} = 0 is 0.009. The estimate S is 19.5, the R^{2}value is 29.49%, the adjusted R^{2}value is 25.46%, and Mallows' Cpstatistic is 2.00.
 At no step is a predictor removed from the stepwise model.
 When α_{E} = α_{R} = 0.15, the final stepwise regression model contains the predictors Brain and Height.
Example #2. Some researchers observed the following data (bloodpress.txt) on 20 individuals with high blood pressure:
 blood pressure (y = BP, in mm Hg)
 age (x_{1} = Age, in years)
 weight (x_{2} = Weight, in kg)
 body surface area (x_{3} = BSA, in sq m)
 duration of hypertension (x_{4} = Dur, in years)
 basal pulse (x_{5} = Pulse, in beats per minute)
 stress index (x_{6} = Stress)
The researchers were interested in determining if a relationship exists between blood pressure and age, weight, body surface area, duration, pulse rate and/or stress level.
The matrix plot of BP, Age, Weight, and BSA looks like:
and the matrix plot of BP, Dur, Pulse, and Stress looks like:
Using Minitab to perform the stepwise regression procedure, we obtain:
When α_{E} = α_{R} = 0.15, the final stepwise regression model contains the predictors Weight, Age, and BSA.
_{ }
PRACTICE PROBLEMS: Stepwise regression
Brain size and body size. Imagine that you do not have automated stepwise regression software at your disposal, and conduct the stepwise regression procedure on the iqsize.txt data set. Setting AlphatoRemove and AlphatoEnter at 0.15, verify the final model obtained above by Minitab. That is:
