Criticisms of Best Subset and Stepwise Regression Tools

Criticisms of these approaches include:

1. The entry and removal of variables from stepwise models is achieved through evaluation of p-values from F-tests or chi-squared tests, but there is no correction for the fact that many tests are completed during the variable selection process.  Many techniques exist for dealing with the "multiple tests" problem (see the Simes correction for one example), but similar approaches are not applied to these tools for comparing regression models.  

2. These approaches are often strongly affected by the presence of collinearity among datasets, and R-squared values tend to be biased to be high.

3. Similarly, p-values that are presented as output from these tests are conditional upon previous steps where variables have been excluded or included.  The output from SpaceStat (and other statistical tools) does not include any modification based on the number of variables evaluated or tests performed.

4. These techniques are often viewed as using computational power rather than brain power (thinking about relationships among relevant variables) to drive model building and evaluation.

This overview was based on a helpful summary of critiques by many statisticians (mostly negative, and including references) created by Dr. Rich Ulrich, found here (at http://www.childrens-mercy.org/stats/faq/faq12.asp).