Stepwise Regression with BIC in R

Stepwise regression is a popular method used for selecting a subset of predictor variables by either adding or removing them from the model based on certain criteria. In this blog post, we will learn how to perform stepwise regression in R using the Bayesian Information Criterion (BIC) as the selection criterion.

Why Use BIC in Stepwise Regression?

There are several criteria for selecting variables in stepwise regression, such as AIC (Akaike Information Criterion) and BIC. While AIC focuses on fitting the model to the data well, BIC introduces a larger penalty for models with more parameters, thus favoring simpler models. Using BIC can help avoid overfitting.

Advantages of BIC:

• Introduces stricter penalties for model complexity.
• Favors simpler, more interpretable models.
• Helps prevent overfitting, especially when the number of observations is large.

Stepwise Regression in R

In R, you can use the step() function to perform stepwise regression. By default, the step() function uses AIC as the selection criterion, but we can easily switch to BIC by adjusting the k parameter (where k = log(n), and n is the number of observations).

Data Setup

Let’s begin by setting up some sample data using the built-in mtcars dataset, which contains data on fuel consumption, vehicle design, and engine performance.

data(mtcars)
head(mtcars)

This dataset contains various predictors such as mpg (miles per gallon), hp (horsepower), wt (weight), and others. We will perform stepwise regression to identify the best subset of predictors for modeling mpg.

Performing Stepwise Regression with BIC

We will start by fitting a full model, then perform forward and backward stepwise selection using BIC.

# Full model
full_model <- lm(mpg ~ ., data = mtcars)

# Stepwise regression using BIC
stepwise_model <- step(full_model, direction = "both", k = log(nrow(mtcars)))

# Summary of the final model
summary(stepwise_model)

In the code above, the argument k = log(nrow(mtcars)) tells R to use BIC instead of AIC (since k = 2 corresponds to AIC).

Interpreting the Results

The step() function will print the sequence of models evaluated and the corresponding BIC values (although the output will still be labeled AIC). The model with the lowest BIC is selected as the final model. You can inspect the summary of the final model using summary(stepwise_model).

Conclusion

In this tutorial, we demonstrated how to perform stepwise regression using BIC in R. BIC is a useful criterion when you want to emphasize model simplicity and avoid overfitting, especially in large datasets. The step() function in R makes it easy to implement this approach.