When you're starting to build a new machine learning model and you're deciding on the model architecture, there are a number of issues that arise. You have to monitor code changes you make, note any differences in the data you've used for training, and keep up with hyperparameter value updates. Being able to track all of these changes is important so that you can reproduce your experiments without wondering which changes gave you the best model. You can go back to any point in your experimenting process to see which changes gave you the best results. In this post, we're going to go through an example of hyperparameter tuning with reproducibility using DVC. You can add this to any existing project you're working on or start from a fresh project. Background on Hyperparameters Before jumping straight into training and experiments, let's briefly go over some background on hyperparameters. are the values that define your model. This includes things like the number of layers in a neural network or the learning rate for gradient descent. Hyperparameters These parameters are different from model parameters because we can't get them from training our model. They are used to the model we train with. create Optimizing these values means running training steps for different kinds of models to see how accurate the results are. We can get the best model by iterating through different hyperparameter values and seeing how they affect our accuracy. That's why we do hyperparameter tuning. There are a couple of common methods that we'll do some code examples with: grid search and random search. Tuning with DVC Let's start by talking about DVC a bit because we'll be using it to add reproducibility to our tuning process. This is the tool we'll be using to track changes in our data, code, and hyperparameters. With DVC, we can add some automation to the tuning process and be able to find and restore any really good models that emerge. A few things DVC makes easier to do: Letting you make changes without worrying about finding them later Onboarding other engineers to a project Sharing experiments with other engineers on different machines For hyperparameter tuning, this means you can play with their values without losing track of which changes made the best model and also have other engineers take a look. We'll do an example of this with grid search in DVC first. Working with a DVC project We're going to be working with an existing NLP project. You can . It already has DVC set up, but you can check out if you want to know how the DVC pipeline was created. get the code we're working with in this repo the Get Started docs First, make sure you're in a virtual environment with a command similar to this. $ python -m venv .venv After you've cloned the repo, install all of the dependencies with this command. $ pip install -r requirements.txt You should be able to open your terminal and run an experiment with the following command. $ dvc exp run This will trigger the training process to run and it will record the of your model. You can check out the results of your experiment with the following command. ROC-AUC $ dvc exp show --no-timestamp --include-params train.n_est,train.min_split We're adding a few options here to make the table view clearer. We aren't showing timestamps and we're only looking at two hyperparameter values. You can run dvc exp show without the options to see the entire table. This will produce a table similar to this. ┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Experiment              ┃ avg_prec ┃ roc_auc ┃ train.min_split ┃ train.n_est ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ workspace               │ 0.51682  │ 0.93819 │ 175         │ 64              │
│ master                  │ 0.56447  │ 0.94713 │ 100         │ 64              │
│ └── a1e8716 [exp-09074] │ 0.57333  │ 0.94801 │ 100         │ 32              │
└─────────────────────────┴──────────┴─────────┴─────────────┴─────────────────┘ Start tuning with grid search Now that you've seen how to run an experiment, we're going to write a small script to automate grid search for us using DVC. Using grid search in hyperparameter tuning means you have an exhaustive list of hyperparameter values you want to cycle through. Grid search will cover every combination of those hyperparameter values. We'll do this by creating queues. A queue is how DVC allows us to create experiments that won't be run until later. That way we can cycle through multiple hyperparameters quickly instead of manually updating a config file with new hyperparameter values for each experiment run. The command syntax for creating queues looks like this: $ dvc exp run --queue --set-param train.min_split=8 In the example queue above, we're updating the value that's inside of the file. This file holds all of the hyperparameter values and is where DVC looks to determine if any values have changed. With the command above, we're automatically updating that value in the using a queued experiment. train.min_split params.yaml params.yaml Now we can make the script. You can add a new file to the directory called . Inside of the file, add the following code. src grid_search.py import itertools
import subprocess

# Automated grid search experiments
n_est_values = [250, 300, 350, 400, 450, 500]
min_split_values = [8, 16, 32, 64, 128, 256]

# Iterate over all combinations of hyperparameter values.
for n_est, min_split in itertools.product(n_est_values, min_split_values):
    # Execute "dvc exp run --queue --set-param train.n_est=<n_est> --set-param train.min_split=<min_split>".
    subprocess.run(["dvc", "exp", "run", "--queue",
                    "--set-param", f"train.n_est={n_est}",
                    "--set-param", f"train.min_split={min_split}"]) This is a simple grid search. We have two hyperparameters we want to tune: and . So we have arrays with a few values in them to mimic the exhaustive search a grid search can handle. Then we loop through the values and create queued experiments for them using . n_est min_split subprocess You can run this script now and generate your queue with this command. $ python src/grid_search.py You'll see some outputs in the terminal telling you that your experiments have been queued. Then you can run them all with the following command. $ dvc exp run --run-all This will run every experiment that has been queued. Once all of those have run, take a look at your metrics for each experiment. $ dvc exp show --include-params=train.min_split,train.n_est --no-timestamp Your table should look similar to this when you run the command above. We've included the and options to give us a table that's easier to read. --include-params --no-timestamp ┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Experiment              ┃ avg_prec ┃ roc_auc ┃ train.min_split ┃ train.n_est ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ workspace               │  0.67038 │ 0.96693 │ 64              │ 100         │
│ try-large-dataset       │  0.67038 │ 0.96693 │ 64              │ 100         │
│ ├── 4899d41 [exp-ae5ed] │   0.6888 │ 0.97028 │ 8               │ 250         │
│ ├── bcdd8ed [exp-56613] │  0.68733 │ 0.96773 │ 16              │ 250         │
│ ├── 703f20b [exp-caa84] │  0.68942 │  0.9698 │ 32              │ 250         │
│ ├── 1a882e6 [exp-c208f] │    0.681 │ 0.96772 │ 64              │ 250         │
│ ├── 3ac33fb [exp-4c53e] │  0.67775 │ 0.96664 │ 128             │ 250         │
│ ├── ea90ee0 [exp-fdb47] │  0.65382 │ 0.96719 │ 256             │ 250         │
│ ├── b8277b1 [exp-3fb5c] │  0.68547 │ 0.97011 │ 8               │ 300         │
│ ├── 7be641e [exp-3bbbc] │   0.6883 │ 0.96724 │ 16              │ 300         │
│ ├── 4202757 [exp-38ca4] │  0.68808 │ 0.96968 │ 32              │ 300         │
│ ├── b71ee2f [exp-5384b] │  0.68111 │ 0.96848 │ 64              │ 300         │
│ ├── 1bbb0f4 [exp-f5d54] │  0.67707 │ 0.96753 │ 128             │ 300         │
│ ├── 71ba159 [exp-31749] │  0.65282 │ 0.96752 │ 256             │ 300         │
│ ├── 836c1c5 [exp-2ce0a] │  0.68758 │ 0.96998 │ 8               │ 350         │
│ ├── dac9e22 [exp-5c799] │  0.68778 │ 0.96779 │ 16              │ 350         │ Now you can see how your precision changed with each hyperparameter value update. This is a quick implementation of grid search in DVC. You could read the hyperparameter values from a different file or data source or make this tuning script as fancy as you like. The main thing you need is the command to execute when you add new values. dvc exp run --queue --set-param <param> Random search Another commonly used method for tuning hyperparameters is random search. This takes random values for hyperparameters and builds the model with them. It usually takes less time than an exhaustive grid search and it can perform better if run for a similar amount of time as a grid search. We're going to add an example of a random search in a new file called similar to the file we created for a grid search. This will add queued experiments with the randomly selected hyperparameter values. Add the following code to . random_search.py random_search.py import subprocess
import random

# Automated random search experiments
num_exps = 10
random.seed(0)

for _ in range(num_exps):
    params = {
        "rand_n_est_value": random.randint(250, 500),
        "rand_min_split_value": random.choice([8, 16, 32, 64, 128, 256])
    }
    subprocess.run(["dvc", "exp", "run", "--queue",
                    "--set-param", f"train.n_est={params['rand_n_est_value']}",
                    "--set-param", f"train.min_split={params['rand_min_split_value']}"]) This search could be far more complex with Bayesian optimization to handle the hyperparameter value selections, but we're keeping it super simple by choosing random numbers to focus on reproducibility. This will generate ten experiments with random values for each hyperparameter. You can run these new experiments with and then take a look at the results with . Your table should look something like this. dvc exp run --run-all dvc exp show --include-params=train.min_split,train.n_est --no-timestamp ┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Experiment              ┃ avg_prec ┃ roc_auc ┃ train.min_split ┃ train.n_est ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ workspace               │  0.67038 │ 0.96693 │ 64             │ 100         │
│ try-large-dataset       │  0.67038 │ 0.96693 │ 64              │ 100         │
│ ├── fc28c0c [exp-45902] │  0.68358 │ 0.96956 │ 64              │ 466         │
│ ├── f13ac72 [exp-b9dfa] │  0.68275 │ 0.96914 │ 64              │ 444         │
│ ├── a8cbc8f [exp-b0aeb] │  0.68989 │ 0.97003 │ 32              │ 260         │
│ ├── 4791c52 [exp-5f2b5] │  0.67711 │ 0.96809 │ 128             │ 497         │
│ ├── c5398e0 [exp-86c74] │   0.6811 │ 0.96829 │ 64              │ 374         │
│ ├── db16c91 [exp-db50f] │  0.68986 │ 0.97073 │ 32              │ 485         │
│ ├── 2dd08fa [exp-fee4f] │  0.68262 │ 0.96941 │ 64              │ 497         │
│ ├── 18d2ec5 [exp-d73c7] │  0.67696 │ 0.96726 │ 128             │ 341         │
│ ├── 1710032 [exp-dd198] │  0.68756 │  0.9687 │ 16              │ 478         │
│ ├── 4f0b80a [exp-746c1] │  0.68724 │ 0.96811 │ 16              │ 379         │ This shows the difference in the randomly selected values and the values from the grid search. You might find a better value with random search because it jumps around a range of values which might hit the optimum faster than it would with a grid search. Conclusion With the comparison between grid search and random search, you can see how reproducibility can help you find the best model for your project. You'll be able to see all of the hyperparameter changes and code changes that created each model. This gives you the ability to fine-tune your model because you can go to any experiment and resume training with different values, code, or data.

Super

Fine-Tuning Machine Learning Models with DVC Experiments for Transfer Learning 

How to Build A Passwordless Authentication with Email and JWT

Check out my site!

Nominated for 2022 - HackerNoon Contributor of the Year - Deep Learning

Too Long; Didn't Read

Get Free Unlimited IP Geo Data With IPinfo Lite

How to Use DVC for Tuning Hyperparameters in Machine Learning

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

7 Tips For Becoming A Better JavaScript Developer

A Step By Step Guide To Data Visualization With Power BI

An A-Z Guide to Decision Trees

Analysis And Prediction on HR Data Set for Beginners

Comparing Different Time-Series Databases

7 Tips For Becoming A Better JavaScript Developer

A Step By Step Guide To Data Visualization With Power BI

An A-Z Guide to Decision Trees

Analysis And Prediction on HR Data Set for Beginners

Comparing Different Time-Series Databases

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps