Grid search in the tidyverse

深藏阁楼爱情的钟 2022-07-14 13:55 199阅读 0赞

## Grid Search ##

For anyone who’s unfamiliar with the term, grid search involves running a model many times with combinations of various hyperparameters. The point is to identify which hyperparameters are likely to work best. A more technical definition from [Wikipedia][], grid search is:

> an exhaustive searching through a manually specified subset of the hyperparameter space of a learning algorithm

## [ ][Link 1]What this post isn’t about ##

To keep the focus on grid search, this post does **NOT** cover…

*  k-fold cross-validation. Although a practically essential addition to grid search, I’ll save the combination of these techniques for a future post. If you can’t wait, check out my [last post][] for some inspiration.
 *  Complex learning models. We’ll stick to a simple decision tree.
 *  Getting a *great* model fit. I’ve deliberately chosen input variables and hyperparameters that highlight the approach.

## [ ][Link 2]Decision tree example ##

Say we want to run a simple decision tree to predict cars’ transmission type (`am`) based on their miles per gallon (`mpg`) and horsepower (`hp`) using the `mtcars`data set. Let’s prep the data:

library(tidyverse)
    
    d <- mtcars %>% 
           # Convert `am` to factor and select relevant variables
           mutate(am = factor(am, labels = c("Automatic", "Manual"))) %>% 
           select(am, mpg, hp)
    
    ggplot(d, aes(mpg, hp, color = am)) +
      geom_point()

[![unnamed-chunk-2-1.png][]][unnamed-chunk-2-1.png 1]

For a decision tree, it looks like a step-wise function until `mpg > 25`, at which point it’s all Manual cars. Let’s grow a full decision tree on this data:

library(rpart)
    library(rpart.plot)
    
    # Set minsplit = 2 to fit every data point
    full_fit <- rpart(am ~ mpg + hp, data = d, minsplit = 2)
    prp(full_fit)

[![unnamed-chunk-3-1.png][]][unnamed-chunk-3-1.png 1]

We don’t want a model like this, as it almost certainly has overfitting problems. So the question becomes, which hyperparameter specifications would work best for our model to generalize?

## [ ][Link 3]Training-Test Split ##

To help validate our hyperparameter combinations, we’ll split our data into training and test sets (in an 80/20 split):

set.seed(245)
    n <- nrow(d)
    train_rows <- sample(seq(n), size = .8 * n)
    train <- d[ train_rows, ]
    test  <- d[-train_rows, ]

## [ ][Link 4]Create the Grid ##

Step one for grid search is to define our hyperparameter combinations. Say we want to test a few values for `minsplit` and `maxdepth`. I like to setup the grid of their combinations in a tidy data frame with a `list` and `cross_d` as follows:

# Define a named list of parameter values
    gs <- list(minsplit = c(2, 5, 10),
               maxdepth = c(1, 3, 8)) %>% 
      cross_d() # Convert to data frame grid
    
    gs
    #> # A tibble: 9 × 2
    #>   minsplit maxdepth
    #>      <dbl>    <dbl>
    #> 1        2        1
    #> 2        5        1
    #> 3       10        1
    #> 4        2        3
    #> 5        5        3
    #> 6       10        3
    #> 7        2        8
    #> 8        5        8
    #> 9       10        8

Note that the **list names are the names of the hyperparameters** that we want to adjust in our model function.

## [ ][Link 5]Create a model function ##

We’ll be iterating down the `gs` data frame to use the hyperparameter values in a`rpart` model. The easiest way to handle this is to define a function that accepts a row of our data frame values and passes them correctly to our model. Here’s what I’ll use:

mod <- function(...) {
      rpart(am ~ hp + mpg, data = train, control = rpart.control(...))
    }

Notice the argument `...` is being passed to `control` in `rpart`, which is where these hyperparameters can be used.

## [ ][Link 6]Fit the models ##

Now, to fit our models, use `pmap` to iterate down the values. The following is iterating through each row of our `gs` data frame, plugging the hyperparameter values for that row into our model.

gs <- gs %>% mutate(fit = pmap(gs, mod))
    gs
    #> # A tibble: 9 × 3
    #>   minsplit maxdepth         fit
    #>      <dbl>    <dbl>      <list>
    #> 1        2        1 <S3: rpart>
    #> 2        5        1 <S3: rpart>
    #> 3       10        1 <S3: rpart>
    #> 4        2        3 <S3: rpart>
    #> 5        5        3 <S3: rpart>
    #> 6       10        3 <S3: rpart>
    #> 7        2        8 <S3: rpart>
    #> 8        5        8 <S3: rpart>
    #> 9       10        8 <S3: rpart>

## [ ][Link 7]Obtain accuracy ##

Next, let’s assess the performance of each fit on our test data. To handle this efficiently, let’s write another small function:

compute_accuracy <- function(fit, test_features, test_labels) {
      predicted <- predict(fit, test_features, type = "class")
      mean(predicted == test_labels)
    }

Now apply this to each fit:

test_features <- test %>% select(-am)
    test_labels   <- test$am
    
    gs <- gs %>%
      mutate(test_accuracy = map_dbl(fit, compute_accuracy,
                                     test_features, test_labels))
    gs
    #> # A tibble: 9 × 4
    #>   minsplit maxdepth         fit test_accuracy
    #>      <dbl>    <dbl>      <list>         <dbl>
    #> 1        2        1 <S3: rpart>     0.7142857
    #> 2        5        1 <S3: rpart>     0.7142857
    #> 3       10        1 <S3: rpart>     0.7142857
    #> 4        2        3 <S3: rpart>     0.8571429
    #> 5        5        3 <S3: rpart>     0.8571429
    #> 6       10        3 <S3: rpart>     0.7142857
    #> 7        2        8 <S3: rpart>     0.8571429
    #> 8        5        8 <S3: rpart>     0.8571429
    #> 9       10        8 <S3: rpart>     0.7142857

## [ ][Link 8]Arrange results ##

To find the best model, we `arrange` the data based on`desc(test_accuracy)`. The best fitting model will then be in the first row. You might see above that we have many models with the same fit. This is unusual, and likley due to the example I’ve chosen. Still, to handle this, I’ll break ties in accuracy with `desc(minsplit)` and `maxdepth` to find the model that is most accurate and also simplest.

gs <- gs %>% arrange(desc(test_accuracy), desc(minsplit), maxdepth)
    gs
    #> # A tibble: 9 × 4
    #>   minsplit maxdepth         fit test_accuracy
    #>      <dbl>    <dbl>      <list>         <dbl>
    #> 1        5        3 <S3: rpart>     0.8571429
    #> 2        5        8 <S3: rpart>     0.8571429
    #> 3        2        3 <S3: rpart>     0.8571429
    #> 4        2        8 <S3: rpart>     0.8571429
    #> 5       10        1 <S3: rpart>     0.7142857
    #> 6       10        3 <S3: rpart>     0.7142857
    #> 7       10        8 <S3: rpart>     0.7142857
    #> 8        5        1 <S3: rpart>     0.7142857
    #> 9        2        1 <S3: rpart>     0.7142857

It looks like a `minsplit` of 5 and `maxdepth` of 3 is the way to go!

To compare to our fully fit tree, here’s a plot of this top-performing model. Remember, it’s in the first row so we can reference `[[1]]`.

prp(gs$fit[[1]])

[![unnamed-chunk-11-1.png][]][unnamed-chunk-11-1.png 1]

## [ ][Link 9]Food for thought ##

Having the results in a tidy data frame lets us do a lot more than just pick the optimal hyperparameters. It lets us quickly wrangle with and visualize the results of the various combinations. Here are some ideas:

*  Search among the top performers for the simplest model.
 *  Plot performance across the hyperparameter combinations.
 *  Save time by restricting the hypotheses before model fitting. For example, in a large data set, it’s practically pointless to try a small `minsplit` and small `maxdepth`. In this case, before fitting the models, we can `filter`the `gs` data frame to exclude certain combinations.

## [ ][Link 10]Sign off ##

Thanks for reading and I hope this was useful for you.

For updates of recent blog posts, follow [@drsimonj][drsimonj] on Twitter, or email me at[drsimonjackson@gmail.com][drsimonjackson_gmail.com] to get in touch.

If you’d like the code that produced this blog, check out the [blogR GitHub repository][].

[Wikipedia]: https://en.wikipedia.org/wiki/Hyperparameter_optimization
[Link 1]: http://drsimonj.svbtle.com/grid-search-in-the-tidyverse#what-this-post-isn39t-about_2
[last post]: https://drsimonj.svbtle.com/k-fold-cross-validation-with-modelr-and-broom
[Link 2]: http://drsimonj.svbtle.com/grid-search-in-the-tidyverse#decision-tree-example_2
[unnamed-chunk-2-1.png]: /images/20220714/aab4b3a301004241a130807b974b9817.png
[unnamed-chunk-2-1.png 1]: https://i1.wp.com/svbtleusercontent.com/afba3qginknbq.png?ssl=1
[unnamed-chunk-3-1.png]: /images/20220714/21d64059f5a144aebd8f2dbd8a43d26f.png
[unnamed-chunk-3-1.png 1]: https://i2.wp.com/svbtleusercontent.com/x0cgd5zrhwxlyq.png?ssl=1
[Link 3]: http://drsimonj.svbtle.com/grid-search-in-the-tidyverse#trainingtest-split_2
[Link 4]: http://drsimonj.svbtle.com/grid-search-in-the-tidyverse#create-the-grid_2
[Link 5]: http://drsimonj.svbtle.com/grid-search-in-the-tidyverse#create-a-model-function_2
[Link 6]: http://drsimonj.svbtle.com/grid-search-in-the-tidyverse#fit-the-models_2
[Link 7]: http://drsimonj.svbtle.com/grid-search-in-the-tidyverse#obtain-accuracy_2
[Link 8]: http://drsimonj.svbtle.com/grid-search-in-the-tidyverse#arrange-results_2
[unnamed-chunk-11-1.png]: /images/20220714/e08b91cb4f234df2b42d5c513055a96c.png
[unnamed-chunk-11-1.png 1]: https://i0.wp.com/svbtleusercontent.com/8v4tygcg2vfbtg.png?ssl=1
[Link 9]: http://drsimonj.svbtle.com/grid-search-in-the-tidyverse#food-for-thought_2
[Link 10]: http://drsimonj.svbtle.com/grid-search-in-the-tidyverse#sign-off_2
[drsimonj]: https://twitter.com/drsimonj
[drsimonjackson_gmail.com]: mailto:drsimonjackson@gmail.com
[blogR GitHub repository]: https://github.com/drsimonj/blogR