We’ve reimagined steel framing from the ground up. Hyperframe is up to 15x faster than traditional steel framing. No tools or training needed.

Hyperframe

The emergence of (LLMs) has been nothing short of a technological marvel. These mammoth models, such as GPT, BERT, RoBERTa, and others, have reshaped our understanding of natural language processing and transformed industries ranging from healthcare to entertainment. Large Language Models Yet, for all their genius, LLMs have a fundamental limitation: they are general-purpose models, which makes them akin to versatile Swiss army knives. While they can handle a myriad of tasks with admirable competence, the true mastery lies in task-specific performance. This is where the concept of fine-tuning steps in, promising to unlock their full potential. In this exploration, we delve into the fascinating journey of fine-tuning LLMs for specialized tasks. We will explore the landscape of chatbots, language translation, and sentiment analysis, illuminating the nuanced process behind each adaptation. This journey aims to reveal not just the 'how' but also the 'why' of fine-tuning. The Art of Fine-Tuning resembles sculpting a raw block of marble into a masterpiece. It starts with selecting a base model — often a pre-trained LLM like GPT-3.5 from openAI, Falcon, LLAMA2, and Mistral from Hugging face transformers library — as the foundation. This base model has already been trained on vast swaths of text, saturating it with a deep understanding of language. Fine-tuning an LLM In the process of finetuning, it adapts to specific tasks through a process of focused training. Imagine, for instance, crafting a chatbot capable of realistic, context-aware conversations. Fine-tuning involves feeding the model with task-specific data and objectives, allowing it to refine its responses gradually. Let us then embark on a journey through three distinct applications of fine-tuning: Chatbots, where we'll unveil the strategies behind crafting chatbots that feel human; Language Translation, which would allow us to explore how LLMs can seamlessly switch between languages and dialects; and finally, with Sentiment Analysis, we'll delve into the mechanics of sentiment analysis and how fine-tuning refines it. As we unravel the art and science of fine-tuning LLMs for specialized tasks, it becomes clear that this process is the key to unlocking their full potential. It's not just about having a Swiss army knife, but more about sharpening each blade to perfection for a specific purpose. The Fine-Tuning Process Fine-tuning is a process where we start with pre-trained models as our initial foundation and then train these models with custom data to tweak some of the model parameters. This helps refine the model’s representations and weights, aligning them more closely with the requirements of the downstream task. In this chapter, we will explore the art of fine-tuning with transformers trainer which helps us in training these pre-trained models, and Optuna, a versatile hyperparameter search framework, for finding the parameters that will give the best performance on your downstream task. Let’s assume a text classification problem, the data consists of two features namely “text” and “labels”. Given the text, our task is to Since we are using the transformers library for training, we have to convert the data into format. classify it into one of the labels. ds.arrow_dataset.Dataset Once data is set up, load the tokenizer and foundation model for the downstream task. The above code imports a tokenizer for our language model. It takes either a Namespace from Huggingface repo like “bert-base-uncased” above or a path to the vocab files required by the tokenizer if you have saved any previously using the save_pretrained method, as input. Similarly, we import our initial model and it’s config using the above code. Now, our tokenizer, data, and the initial model have all been set up. It’s time to tokenize our data using a tokenizer since tokenized data is the input for model training and not the raw data. The above code tokenizes the training data. After tokenizing, we have to remove the text column since it is of no use from now on — we only use tokens. If you have the length of the sequence that is more than the accepted model’s sequence length, you must truncate; and if the max sequence length of custom data is less than the accepted model’s sequence length, then you can pad all the sequences with the padding parameter as shown above. This results in getting all the sequences with a constant length. Similar code can be used for tokenizing validation data and test data which we will use for evaluation and inference. Once training, validation, and test data have been set up, we need to create a function that helps in reporting metrics for every epoch. Let’s call it compute_metrics. The above function takes the predictions from the trainer and it calculates accuracy, recall, and precision for every epoch and it is shown in stdout. How Optuna works We create an Optuna study that corresponds to an optimization task, i.e. a set of trails. Optuna requires an objective function to quantify the performance of a model with specific hyperparameters. The main purpose of this function is to guide the hyperparameter optimization process. It optimizes by conducting trials, each representing a unique combination of hyperparameters. Each trial will have fixed hyperparameters, and these are passed to the objective function to calculate the objective. It aims to find the set of hyperparameters that minimizes or maximizes this objective function, depending on whether it represents a loss or a score/metric. Each trial information is logged in the study, and this information can be saved to a database or in local storage. You can convert it into a data frame as well. Best trial and best parameter values are also stored in the study. One can view them using study.best_trial. Now, let’s define our objective function. The first step is defining our hyperparameter space in a dictionary as is shown below: Since we are using a transformer trainer, we will use this dictionary to define training arguments. One can find the training arguments . Each hyperparameter has been defined in a specific range using Optuna’s trial object. here So, the next step is to set up the training and evaluation code: The above code takes the train data, validation data and the training arguments that we defined previously and initiates the training process when the execution reaches . Then evaluation is done with method. trainer.train() trainer.evaluate() We predict the results for test data and store them in the variable. Since we are concerned about “recall” in this particular case, let’s return to test recall from the objective. Since this is a metric, we have to maximize the objective function. test_results The next is creating the Optuna study and starting the hyperparameter optimization job: This will create a new Optuna study with the name . Since we are returning a metric from the objective function, our direction is to “maximize” the objective function. will trigger the hyper-parameter optimization process and once it’s completed, one can view the best trial and see the parameters that are performing well. hyperprameter_optimization_bert study.optimize() Applications of Fine-Tuned LLMs The expert-level Large Language Models (LLMs), trained on vast datasets and meticulously refined, exhibit a remarkable versatility that extends across multiple facets of Natural Language Understanding, Content Generation, Specialized Domains, and, crucially, Ethical Considerations. In the arena of , fine-tuned models have proven their mettle in diverse applications. Firstly, they stand out in sentiment analysis, decoding the nuanced emotional tones embedded in textual content with remarkable accuracy. Fine-tuned LLMs are indispensable tools when analyzing public sentiment toward a political event or product feedback for market research. Moreover, they are good in language translation and multilingual tasks, exceeding language barriers to facilitate global communication and collaboration. Natural Language Understanding becomes an art form when fine-tuned LLMs are employed. They possess the uncanny ability to generate creative content, whether it be generating compelling narratives, poetic verses, or even witty advertisements. Industries like advertising and marketing have embraced these models, utilizing their capacity to create persuasive and engaging content that resonates with audiences deeply emotionally. Content Generation In , fine-tuned LLMs have cut out a niche of their own. They undergo fine-tuning to serve as diagnostic tools in the medical field, offering assistance in early disease detection and patient care. Simultaneously, their abilities extend to legal document analysis, parsing through extensive legal texts with precision, speeding up the work of legal professionals while ensuring accuracy and compliance. Specialized Domains Real-world examples of enterprises using LLM-based AI search engines are , which has refined LLMs by using targeted fine-tuning with a specialized set of legal documents, which has boosted the pace of data collection and analysis and unveiled the information concealed within documents. LawGeex and , IN-D AI However, as we tread into the future, cannot be overlooked. It is imperative that we address concerns related to misuse and biases in these fine-tuned models, since the list of concerns is quite long: it might be misinformation, plagiarism, privacy concerns, security threats, and even dehumanization of communication. Ensuring transparency, fairness, and accountability in their deployment is thus paramount. The responsible use of LLMs must be guided by ethical principles to multiply their potential while minimizing unintended consequences. Ethical Considerations Challenges and Limitations Continuing the topic of ethical concerns, fine-tuning does not absolve models of biases inherent in the data; in fact, it can amplify them. A model trained on biased data can lead to contradictory results, raising ethical concerns. However, they are not the only thing that should be kept a close eye on. Although fine-tuning in itself involves feeding the model with task-specific data and objectives,  one of the primary challenges in this process is still obtaining this very data. High-quality, labeled datasets are crucial for training models to achieve peak performance in specialized tasks. However, many domains lack the necessary volume of data or represent niche topics where data collection is exhausting. This scarcity restricts the model's ability to generalize well to real-world scenarios. Another concern is overfitting. When a model is fine-tuned on limited data, it might perform exceptionally well on that specific data but fail to generalize to unseen examples. Overfitting is particularly problematic with LLMs due to their capacity to memorize vast amounts of information. Other than that, fine-tuning LLMs demands considerable computational power. Training large models on specific tasks not only requires specialized hardware but also results in significant energy consumption and costs. For researchers and organizations with limited resources, this poses a barrier to achieving the full potential of LLMs. As one ventures deeper into the age of AI, the precision offered by fine-tuned LLMs could bring a true revolution to the industries, from healthcare diagnostics to bespoke content creation. Yet, with great promise comes the imperative to navigate the nuanced complexities involved, ensuring both efficiency and ethical integrity. By acknowledging these challenges, the community can work collaboratively to develop solutions that tame the power of LLMs while minimizing potential drawbacks.

A Detailed Guide to Fine-Tuning for Specific Tasks

Authors: (1) Muzhaffar Hazman, University of Galway, Ireland; (2) Susan McKeever, Technological University Dublin, Ireland; (3) Josephine Griffith, University of Galway, Ireland. Table of Links Abstract and Introduction Related Works Methodology Results Limitations and Future Works Conclusion, Acknowledgments, and References A Hyperparameters and Settings B Metric: Weighted F1-Score C Architectural Details D Performance Benchmarking E Contingency Table: Baseline vs. Text-STILT A Hyperparameters and Settings This paper is under CC 4.0 license. available on arxiv

Unimodal Training for Multimodal Meme Sentiment Classification: Hyperparameters and Settings

Authors: (1) Dominic Petrak, UKP Lab, Department of Computer Science, Technical University of Darmstadt, Germany; (2) Nafise Sadat Moosavi, Department of Computer Science, The University of Sheffield, United Kingdom; (3) Ye Tian, Wluper, London, United Kingdom; (4) Nikolai Rozanov, Wluper, London, United Kingdom; (5) Iryna Gurevych, UKP Lab, Department of Computer Science, Technical University of Darmstadt, Germany. Table of Links Abstract & Introduction Related Work Datasets Examined Manual Error Type Analysis and Taxonomies Automatic Filtering for Potentially Relevant Dialogs Statistical Analysis Evaluation and Experiments Discussion Conclusion, Limitation, Acknowledgments, and References A Integrated Error Taxonomy – Details B Error-Indicating Sentences And Phrases C Automatic Filtering – Implementation D Automatic Filtering – Sentence-Level Analysis E Task-Oriented Dialogs – Examples F Effectiveness Of Automatic Filtering – A Detailed Analysis G Inter-Annotator Agreement – Detailed Analysis H Annotation Guidelines I Hyperparameters and Baseline Experiments J Human-Human Dialogs – Examples I Hyperparameters and Baseline Experiments All baseline models were trained for five epochs. For the experiment using erroneous dialogs, we trained the models for ten epochs. We used a batch size of 32 and a learning rate of 5e − 5 with no warmup steps. As optimizer, we used the implementation of AdamW 11 (Loshchilov and Hutter, 2019) in Pytorch. Except for LLAMA (Touvron et al., 2023), we fully-finetuned all models. For LLAMA, we only finetuned the LoRA (Hu et al., 2022) weights, using a rank of 8, an alpha of 16, and a dropout rate of 0.05. Hyperparameters Table 17 shows the results of our baseline experiments using word-overlapping F1-Score and BLEU (Papineni et al., 2002). Results Compared to the results of our experiment using erroneous dialogs (Table 11), both F1-Score and BLEU are higher in case of GPT-2 and LLAMA. We attribute this to the comparatively bad quality of the erroneous dialogs. The majority of these dialogs comes from the SFC (Hancock et al., 2019) dataset, where most errors are due to out-of-context system utterances (see Section 6.1) and were ignored by users (see Section 6.2). This paper is under CC BY-NC-SA 4.0 DEED license. available on arxiv

Hyperparameters and Baseline Experiments in Dialog Systems

What is Hyperparameter Optimization? In machine learning, hyperparameter optimization or tuning is the problem of choosing a set of optimal hyperparameters for a learning algorithm. A hyperparameter is a parameter whose value is used to control the learning process. By contrast, the values of other parameters (typically node weights) are learned. Strategies Several strategies can be used for performing optimization. The most simple one is manual tuning. One such example is using the [Elbow Method](https://en.wikipedia.org/wiki/Elbow_method_(clustering) for determining the number of clusters in . On the other hand complex models, have dozens of hyperparameters, and combined with the fact that some of them are continuous, the size of the search space explodes, so the manual effort. To tackle this issue, several other "smarter" approaches exist. Some of them are: k -nearest neighbors algorithm Grid search Random search Bayesian optimization Gradient-based optimization Evolutionary optimization Population-based To get more familiar with how these approaches work and what are the differences between them, I encourage you to read these articles , , or . 7 Hyperparameter Optimization Techniques Every Data Scientist Should Know How To Make Deep Learning Models That Don’t Suck Hyperparameter Optimization Approaches Tooling Several frameworks provide implementations of the approaches mentioned above. In this tutorial, we are going to explore , . Weights & Biases - Sweeps (WANDB for short) Setup For this tutorial, we are going to build a classifier for the dataset. We will use from sklearn to predict the presence of heart disease. Heart Disease UCI RandomForestClassifier This tutorial does not focus on data pre-processing, so we'll dive straight into splitting the data into train and test data, and train the model once with default values for hyperparameters. import pandas as pd
from sklearn.model_selection import train_test_split

df = pd.read_csv('data\heart.csv')

X = df.drop(['target'], axis=1)
y = df['target']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) Train the model rfc = RandomForestClassifier()
rfc.fit(X_train, y_train) and that's it, now we have successfully trained a random forest classifier with default values for its hyperparameters. This classifier has the following hyperparameters: bootstrap max_depth max_features min_samples_leaf min_samples_split n_estimators Now let's get into setting up the optimization: Optimization Step 1: Define the Training Script For more details, see the . docs This script should serve as the main entry point for optimization. It performs one training and evaluation of the model with values for the hyperparameters injected from outside (through ). wandb.config It gets the configuration from the outside and performs training and evaluation of the model with fixed values for all of the hyperparameters. The name could be arbitrary, and for this example is . train.py import wandb
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

WANDB_PROJECT_NAME = "hyperparameter-optimization"

with wandb.init(project=WANDB_PROJECT_NAME):
    df = pd.read_csv('data\heart.csv')
    X = df.drop(['target'], axis=1)
    y = df['target']

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

    config = wandb.config
    rfc = RandomForestClassifier(
        bootstrap=config.bootstrap,
        max_depth = config.max_depth,
        max_features = config.max_features,
        min_samples_leaf = config.min_samples_leaf,
        min_samples_split = config.min_samples_split,
        n_estimators = config.n_estimators,
    )

    rfc.fit(X_train, y_train)
    y_pred = rfc.predict(X_test)

    wandb.log({'accuracy': accuracy_score(y_test, y_pred)}) Step 2: Define the Optimization Strategy and Configuration To determine the optimization strategy, i.e. which approach to be used for optimization, what values or ranges should be tried for every hyper-parameter, and what objective to be optimized, a configuration file needs to be defined. sweep.yml This file contains some additional configuration regarding the python path, training script path (from Step 1), to get more familiar see the . docs For this example the file is. sweep.yml program: train.py
method: bayes
project: hyperparameter-optimization
command:
- ${env} 
- ~/envs/hyperopt/bin/python
- ${program}
- ${args}

metric:
  name: accuracy
  goal: maximize
parameters:
  bootstrap:
    values: [True, False]
  max_depth:
    values: [2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, None]
  max_features:
    values: ['auto', 'sqrt']
  min_samples_leaf:
    values: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
  min_samples_split:
    values: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
  n_estimators:
    values: [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 200, 300, 500] Interpretation is that the training script is located at , Bayesian optimization is going to be used and for bootstrap, the values and `False is going to be tried, for max_depth [2, 3, 4, .... ] and similarly for all of the hyperparameters. The objective is maximizing accuracy. So in other words, . train.py True we want to assign values to the hyperparameters such that the accuracy is maximized Step 3: Initializing and Running the optimization To start optimization, open a shell (with your favorite terminal emulator). Activate the python virtual environment (for UNIX , for other see the official Python .) source ~/envs/hyperopt/bin/python guide Initialize the sweep wandb sweep .\sweep.yml Expected Output: Run the sweep wandb agent aleksandar1932/hyperparameter-optimization/r3s5xf4d Expected Output: And that's it, now the running sweep can be observed at WANDB. Step 4: Monitoring Go to the sweep URL, from your shell output. For this example, the output is available , and below. here As the model is trained for different combinations for the hyperparameters, the results are updated in real-time. We can wait for the given approach to find the combination of values that when used for training, provide the best model, or stop the optimization either by terminating the running shell or through WANDB. Conclusion This sweep performed 79 runs, and the best model scored accuracy on a randomly sampled test set for that particular run. train_test_split is done for every run) 0.9016 (In train.py So, it can be concluded that the best RandomForestClassifier should be instantiated with the following hyperparameters. model = RandomForestClassifier(
        bootstrap=False,
        max_depth = 4,
        max_features = 'sqrt',
        min_samples_leaf = 3,
        min_samples_split = 4,
        n_estimators = 200,
    ) The code from this tutorial is available on . GitHub First published here

Using Weights and Biases to Perform Hyperparameter Optimization

When you're starting to build a new machine learning model and you're deciding on the model architecture, there are a number of issues that arise. You have to monitor code changes you make, note any differences in the data you've used for training, and keep up with hyperparameter value updates. Being able to track all of these changes is important so that you can reproduce your experiments without wondering which changes gave you the best model. You can go back to any point in your experimenting process to see which changes gave you the best results. In this post, we're going to go through an example of hyperparameter tuning with reproducibility using DVC. You can add this to any existing project you're working on or start from a fresh project. Background on Hyperparameters Before jumping straight into training and experiments, let's briefly go over some background on hyperparameters. are the values that define your model. This includes things like the number of layers in a neural network or the learning rate for gradient descent. Hyperparameters These parameters are different from model parameters because we can't get them from training our model. They are used to the model we train with. create Optimizing these values means running training steps for different kinds of models to see how accurate the results are. We can get the best model by iterating through different hyperparameter values and seeing how they affect our accuracy. That's why we do hyperparameter tuning. There are a couple of common methods that we'll do some code examples with: grid search and random search. Tuning with DVC Let's start by talking about DVC a bit because we'll be using it to add reproducibility to our tuning process. This is the tool we'll be using to track changes in our data, code, and hyperparameters. With DVC, we can add some automation to the tuning process and be able to find and restore any really good models that emerge. A few things DVC makes easier to do: Letting you make changes without worrying about finding them later Onboarding other engineers to a project Sharing experiments with other engineers on different machines For hyperparameter tuning, this means you can play with their values without losing track of which changes made the best model and also have other engineers take a look. We'll do an example of this with grid search in DVC first. Working with a DVC project We're going to be working with an existing NLP project. You can . It already has DVC set up, but you can check out if you want to know how the DVC pipeline was created. get the code we're working with in this repo the Get Started docs First, make sure you're in a virtual environment with a command similar to this. $ python -m venv .venv After you've cloned the repo, install all of the dependencies with this command. $ pip install -r requirements.txt You should be able to open your terminal and run an experiment with the following command. $ dvc exp run This will trigger the training process to run and it will record the of your model. You can check out the results of your experiment with the following command. ROC-AUC $ dvc exp show --no-timestamp --include-params train.n_est,train.min_split We're adding a few options here to make the table view clearer. We aren't showing timestamps and we're only looking at two hyperparameter values. You can run dvc exp show without the options to see the entire table. This will produce a table similar to this. ┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Experiment              ┃ avg_prec ┃ roc_auc ┃ train.min_split ┃ train.n_est ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ workspace               │ 0.51682  │ 0.93819 │ 175         │ 64              │
│ master                  │ 0.56447  │ 0.94713 │ 100         │ 64              │
│ └── a1e8716 [exp-09074] │ 0.57333  │ 0.94801 │ 100         │ 32              │
└─────────────────────────┴──────────┴─────────┴─────────────┴─────────────────┘ Start tuning with grid search Now that you've seen how to run an experiment, we're going to write a small script to automate grid search for us using DVC. Using grid search in hyperparameter tuning means you have an exhaustive list of hyperparameter values you want to cycle through. Grid search will cover every combination of those hyperparameter values. We'll do this by creating queues. A queue is how DVC allows us to create experiments that won't be run until later. That way we can cycle through multiple hyperparameters quickly instead of manually updating a config file with new hyperparameter values for each experiment run. The command syntax for creating queues looks like this: $ dvc exp run --queue --set-param train.min_split=8 In the example queue above, we're updating the value that's inside of the file. This file holds all of the hyperparameter values and is where DVC looks to determine if any values have changed. With the command above, we're automatically updating that value in the using a queued experiment. train.min_split params.yaml params.yaml Now we can make the script. You can add a new file to the directory called . Inside of the file, add the following code. src grid_search.py import itertools
import subprocess

# Automated grid search experiments
n_est_values = [250, 300, 350, 400, 450, 500]
min_split_values = [8, 16, 32, 64, 128, 256]

# Iterate over all combinations of hyperparameter values.
for n_est, min_split in itertools.product(n_est_values, min_split_values):
    # Execute "dvc exp run --queue --set-param train.n_est=<n_est> --set-param train.min_split=<min_split>".
    subprocess.run(["dvc", "exp", "run", "--queue",
                    "--set-param", f"train.n_est={n_est}",
                    "--set-param", f"train.min_split={min_split}"]) This is a simple grid search. We have two hyperparameters we want to tune: and . So we have arrays with a few values in them to mimic the exhaustive search a grid search can handle. Then we loop through the values and create queued experiments for them using . n_est min_split subprocess You can run this script now and generate your queue with this command. $ python src/grid_search.py You'll see some outputs in the terminal telling you that your experiments have been queued. Then you can run them all with the following command. $ dvc exp run --run-all This will run every experiment that has been queued. Once all of those have run, take a look at your metrics for each experiment. $ dvc exp show --include-params=train.min_split,train.n_est --no-timestamp Your table should look similar to this when you run the command above. We've included the and options to give us a table that's easier to read. --include-params --no-timestamp ┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Experiment              ┃ avg_prec ┃ roc_auc ┃ train.min_split ┃ train.n_est ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ workspace               │  0.67038 │ 0.96693 │ 64              │ 100         │
│ try-large-dataset       │  0.67038 │ 0.96693 │ 64              │ 100         │
│ ├── 4899d41 [exp-ae5ed] │   0.6888 │ 0.97028 │ 8               │ 250         │
│ ├── bcdd8ed [exp-56613] │  0.68733 │ 0.96773 │ 16              │ 250         │
│ ├── 703f20b [exp-caa84] │  0.68942 │  0.9698 │ 32              │ 250         │
│ ├── 1a882e6 [exp-c208f] │    0.681 │ 0.96772 │ 64              │ 250         │
│ ├── 3ac33fb [exp-4c53e] │  0.67775 │ 0.96664 │ 128             │ 250         │
│ ├── ea90ee0 [exp-fdb47] │  0.65382 │ 0.96719 │ 256             │ 250         │
│ ├── b8277b1 [exp-3fb5c] │  0.68547 │ 0.97011 │ 8               │ 300         │
│ ├── 7be641e [exp-3bbbc] │   0.6883 │ 0.96724 │ 16              │ 300         │
│ ├── 4202757 [exp-38ca4] │  0.68808 │ 0.96968 │ 32              │ 300         │
│ ├── b71ee2f [exp-5384b] │  0.68111 │ 0.96848 │ 64              │ 300         │
│ ├── 1bbb0f4 [exp-f5d54] │  0.67707 │ 0.96753 │ 128             │ 300         │
│ ├── 71ba159 [exp-31749] │  0.65282 │ 0.96752 │ 256             │ 300         │
│ ├── 836c1c5 [exp-2ce0a] │  0.68758 │ 0.96998 │ 8               │ 350         │
│ ├── dac9e22 [exp-5c799] │  0.68778 │ 0.96779 │ 16              │ 350         │ Now you can see how your precision changed with each hyperparameter value update. This is a quick implementation of grid search in DVC. You could read the hyperparameter values from a different file or data source or make this tuning script as fancy as you like. The main thing you need is the command to execute when you add new values. dvc exp run --queue --set-param <param> Random search Another commonly used method for tuning hyperparameters is random search. This takes random values for hyperparameters and builds the model with them. It usually takes less time than an exhaustive grid search and it can perform better if run for a similar amount of time as a grid search. We're going to add an example of a random search in a new file called similar to the file we created for a grid search. This will add queued experiments with the randomly selected hyperparameter values. Add the following code to . random_search.py random_search.py import subprocess
import random

# Automated random search experiments
num_exps = 10
random.seed(0)

for _ in range(num_exps):
    params = {
        "rand_n_est_value": random.randint(250, 500),
        "rand_min_split_value": random.choice([8, 16, 32, 64, 128, 256])
    }
    subprocess.run(["dvc", "exp", "run", "--queue",
                    "--set-param", f"train.n_est={params['rand_n_est_value']}",
                    "--set-param", f"train.min_split={params['rand_min_split_value']}"]) This search could be far more complex with Bayesian optimization to handle the hyperparameter value selections, but we're keeping it super simple by choosing random numbers to focus on reproducibility. This will generate ten experiments with random values for each hyperparameter. You can run these new experiments with and then take a look at the results with . Your table should look something like this. dvc exp run --run-all dvc exp show --include-params=train.min_split,train.n_est --no-timestamp ┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Experiment              ┃ avg_prec ┃ roc_auc ┃ train.min_split ┃ train.n_est ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ workspace               │  0.67038 │ 0.96693 │ 64             │ 100         │
│ try-large-dataset       │  0.67038 │ 0.96693 │ 64              │ 100         │
│ ├── fc28c0c [exp-45902] │  0.68358 │ 0.96956 │ 64              │ 466         │
│ ├── f13ac72 [exp-b9dfa] │  0.68275 │ 0.96914 │ 64              │ 444         │
│ ├── a8cbc8f [exp-b0aeb] │  0.68989 │ 0.97003 │ 32              │ 260         │
│ ├── 4791c52 [exp-5f2b5] │  0.67711 │ 0.96809 │ 128             │ 497         │
│ ├── c5398e0 [exp-86c74] │   0.6811 │ 0.96829 │ 64              │ 374         │
│ ├── db16c91 [exp-db50f] │  0.68986 │ 0.97073 │ 32              │ 485         │
│ ├── 2dd08fa [exp-fee4f] │  0.68262 │ 0.96941 │ 64              │ 497         │
│ ├── 18d2ec5 [exp-d73c7] │  0.67696 │ 0.96726 │ 128             │ 341         │
│ ├── 1710032 [exp-dd198] │  0.68756 │  0.9687 │ 16              │ 478         │
│ ├── 4f0b80a [exp-746c1] │  0.68724 │ 0.96811 │ 16              │ 379         │ This shows the difference in the randomly selected values and the values from the grid search. You might find a better value with random search because it jumps around a range of values which might hit the optimum faster than it would with a grid search. Conclusion With the comparison between grid search and random search, you can see how reproducibility can help you find the best model for your project. You'll be able to see all of the hyperparameter changes and code changes that created each model. This gives you the ability to fine-tune your model because you can go to any experiment and resume training with different values, code, or data.

How to Use DVC for Tuning Hyperparameters in Machine Learning

You wrote a Python script that trains and evaluates your machine learning model. Now, you would like to automatically tune hyperparameters to improve its performance? I got you! In this article, I will show you how to convert your script into an objective function that can be optimized with any hyperparameter optimization library. It will take just 3 steps and you will be tuning model parameters like there is no tomorrow. Ready? Let’s go! I suppose your script looks something like this one: main.py pandas pd lightgbm lgb sklearn.model_selection train_test_split

data = pd.read_csv( , nrows= )
X = data.drop([ , ], axis= )
y = data[ ]
(X_train, X_valid, 
y_train, y_valid )= train_test_split(X, y, test_size= , random_state= )

train_data = lgb.Dataset(X_train, label=y_train)
valid_data = lgb.Dataset(X_valid, label=y_valid, reference=train_data)

params = { : , : , : , : , : , : , : }

model = lgb.train(params, train_data,
                  num_boost_round= ,
                  early_stopping_rounds= ,
                  valid_sets=[valid_data],
                  valid_names=[ ])

score = model.best_score[ ][ ]
print( , score) import as import as from import 'data/train.csv' 10000 'ID_code' 'target' 1 'target' 0.2 1234 'objective' 'binary' 'metric' 'auc' 'learning_rate' 0.4 'max_depth' 15 'num_leaves' 20 'feature_fraction' 0.8 'subsample' 0.2 300 30 'valid' 'valid' 'auc' 'validation AUC:' Step 1: Decouple search parameters from code Take the parameters that you want to tune and put them in a dictionary at the top of your script. By doing that you effectively decouple search parameters from the rest of the code. pandas pd lightgbm lgb sklearn.model_selection train_test_split

SEARCH_PARAMS = { : , : , : , : , : }

data = pd.read_csv( , nrows= )
X = data.drop([ , ], axis= )
y = data[ ]
X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size= , random_state= )

train_data = lgb.Dataset(X_train, label=y_train)
valid_data = lgb.Dataset(X_valid, label=y_valid, reference=train_data)

params = { : , : ,
          **SEARCH_PARAMS}

model = lgb.train(params, train_data,
                  num_boost_round= ,
                  early_stopping_rounds= ,
                  valid_sets=[valid_data],
                  valid_names=[ ])

score = model.best_score[ ][ ]
print( , score) import as import as from import 'learning_rate' 0.4 'max_depth' 15 'num_leaves' 20 'feature_fraction' 0.8 'subsample' 0.2 '../data/train.csv' 10000 'ID_code' 'target' 1 'target' 0.2 1234 'objective' 'binary' 'metric' 'auc' 300 30 'valid' 'valid' 'auc' 'validation AUC:' Step 2: Wrap training and evaluation into a function Now, you can put the entire training and evaluation logic inside of a function. This function takes parameters as input and outputs the validation score. train_evaluate pandas pd lightgbm lgb sklearn.model_selection train_test_split

SEARCH_PARAMS = { : , : , : , : , : } data = pd.read_csv( , nrows= )
    X = data.drop([ , ], axis= )
    y = data[ ]
    X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size= , random_state= )

    train_data = lgb.Dataset(X_train, label=y_train)
    valid_data = lgb.Dataset(X_valid, label=y_valid, reference=train_data)

    params = { : , : ,
              **search_params}

    model = lgb.train(params, train_data,
                      num_boost_round= ,
                      early_stopping_rounds= ,
                      valid_sets=[valid_data],
                      valid_names=[ ])

    score = model.best_score[ ][ ] score __name__ == :
    score = train_evaluate(SEARCH_PARAMS)
    print( , score) import as import as from import 'learning_rate' 0.4 'max_depth' 15 'num_leaves' 20 'feature_fraction' 0.8 'subsample' 0.2 : def train_evaluate (search_params) '../data/train.csv' 10000 'ID_code' 'target' 1 'target' 0.2 1234 'objective' 'binary' 'metric' 'auc' 300 30 'valid' 'valid' 'auc' return if '__main__' 'validation AUC:' Step 3: Run Hypeparameter Tuning script We are almost there. All you need to do now is to use this function as an objective for the black-box optimization library of your choice. train_evaluate I will use which I have described in great detail in another article but you can use any hyperparameter optimization library out there. Scikit Optimize In a nutshell I: define the search , SPACE create the function that will be minimized, objective run the optimization via function. skopt.forest_minimize In this example, I will try different configurations starting with randomly chosen parameter sets. 100 10 skopt script_step2 train_evaluate

SPACE = [
    skopt.space.Real( , , name= , prior= ),
    skopt.space.Integer( , , name= ),
    skopt.space.Integer( , , name= ),
    skopt.space.Real( , , name= , prior= ),
    skopt.space.Real( , , name= , prior= )] * train_evaluate(params)


results = skopt.forest_minimize(objective, SPACE, n_calls= , n_random_starts= )
best_auc = * results.fun
best_params = results.x

print( , best_auc)
print( , best_params) import from import 0.01 0.5 'learning_rate' 'log-uniform' 1 30 'max_depth' 2 100 'num_leaves' 0.1 1.0 'feature_fraction' 'uniform' 0.1 1.0 'subsample' 'uniform' @skopt.utils.use_named_args(SPACE) : def objective (**params) return -1.0 30 10 -1.0 'best result: ' 'best parameters: ' This is it. The object contains information about the that produced it. results best score and parameters Note: If you want to after it finishes you can add one callback and one function call to to Neptune. visualize your training and save diagnostic charts log every hyperparameter search Just use this . optuna monitoring helper function neptune neptunecontrib.monitoring.skopt sk_utils skopt script_step2 train_evaluate

neptune.init( )
neptune.create_experiment( , upload_source_files=[ ])

SPACE = [
    skopt.space.Real( , , name= , prior= ),
    skopt.space.Integer( , , name= ),
    skopt.space.Integer( , , name= ),
    skopt.space.Real( , , name= , prior= ),
    skopt.space.Real( , , name= , prior= )] * train_evaluate(params)


monitor = sk_utils.NeptuneMonitor()
results = skopt.forest_minimize(objective, SPACE, n_calls= , n_random_starts= , callback=[monitor])
sk_utils.log_results(results)

neptune.stop() import import as import from import 'jakub-czakon/blog-hpo' 'hpo-on-any-script' '*.py' 0.01 0.5 'learning_rate' 'log-uniform' 1 30 'max_depth' 2 100 'num_leaves' 0.1 1.0 'feature_fraction' 'uniform' 0.1 1.0 'subsample' 'uniform' @skopt.utils.use_named_args(SPACE) : def objective (**params) return -1.0 100 10 Now, when you run your parameter sweep you will see the following: Check out the with all the code, charts and results. skopt hyperparameter sweep experiment Final thoughts In this article, you’ve learned how to optimize hyperparameters of pretty much any Python script in just 3 steps. Hopefully, with this knowledge, you will build better machine learning models with less effort. Happy training! This article was originally posted on the Neptune blog . If you liked it, you may like it there :) You can also find me tweeting @Neptune_a i or posting on LinkedIn about ML and Data Science stuff.

Hyperparameter Tuning on Any Python Script in 3 Easy Steps [A How-To Guide]

Optuna Vs. Hyperopt: Which Hyperparameter Optimization Library You Should Choose

A few days ago, . Specifically, the new releases focuses on tuning and optimizing hyperparameters associated with SageMaker models. The release constitutes a powerful addition in order to streamline the adoption of SageMaker within the data science community. With the new hyperparameter tuning and model optimization capabilities, SageMaker joins a new group of platforms that are entering the market trying to solve this notorious challenge in deep learning applications. Amazon announced the availability of a new set of automatic model tuning capabilities in the AWS SageMaker platform Building deep learning solutions in the real world is a process of constant experimentation and optimization. Differently from any other type of software application, deep learning applications don’t have a linear lifecycle based on the fact that models need to constantly refined, optimized and tested. What are Hyperparameters? Among the possible optimizations in a deep learning model, none plays a more prominent role that the tuning of hyperparameters. Conceptually, hyperparameters are variables that, while external to the model, influence its behavior and knowledge. A mid-size deep learning model, can be influenced by many hyperparameters including some of the following: · The mother of all hyperparameters, the learning rate quantifies the learning progress of a model in a way that can be used to optimize its capacity. Learning Rate: · A classic hyperparameter in deep learning algorithms, the number of hidden units is key to regulate the representational capacity of a model. Number of Hidden Units: · In convolutional Neural Networks(CNNs), the Kernel Width influences the number of parameters in a model which, in turns, influences its capacity. Convolution Kernel Width: · This hyperparameter helps to know the direction of the next step with the knowledge of the previous steps. It helps to prevent oscillations. Momentum: · Mini batch size is the number of sub samples given to the network after which parameter update happens. Batch Size: The Players Tuning hyperparameters in a deep learning application can be an incredibly challenging process given that it requires not only the execution of an experiment but the evaluation of the results against other versions of the model. Recently, there are a new group of platforms that have entered the market with the surgical focus on solving this challenge across different deep learning frameworks. Let’s look at a few of the most prominent solutions in this new space: SageMaker With its latest release, SageMaker incorporates the creation of hyperparameter tuning jobs directly in the SageMaker console. The platform performs the evaluation and comparison of the different hyperparameter models and directly integrates with the training jobs and other aspects of the model lifecycle. Comet.ml Comet.ml provides a super simple interface for the tuning and optimization of hyperparameters across different deep learning frameworks such as TensorFlow, Keras, PyTorch, Scikit-Learn and many others. Developers can seamlessly integrate Comet.ml into their models using the several SDKs available with the platform as well as the REST API. The platform provides a very visual way to tune and evaluate hyperparameters. Weights&Biases One of my favorite stacks when comes to hyperparameter tuning is the newly released Weights&Biases(W&B). Used by deep learning power houses such as OpenAI, W&B provides an advanced toolset and programming model for recording and tuning hyperparameters across different experiments. The solution records and visualizes the different steps in the execution of the model and correlates it with the configuration of its hyperparameters DeepCognition DeepCognition is one of the new platforms making inroads in the world of self-service deep learning. Functionally, DeepCognition enables the implementation, training and optimization of deep learning models with minimum coding. As part of the optimization process, DeepCognition includes a very powerful and visual hyperparameter tuning engine that records and compares the execution of experiments based on specific hyperparameter configurations. Azure ML Azure ML provides native hyperparameter tuning capabilities as part of its ML Studio. While the existing functionality seems limited compared to some of the other options covered in this article, the platform is able to record and provide basic visualizations relevant to the model hyperparameters. The hyperparameter tuning capabilities of Azure ML can be combined with other services such as to streamline the creation and testing of new experiments. Azure ML Experimentation Cloud ML Just like AWS SageMaker and Azure ML, Google Cloud ML provides some basic hyperparameter tuning capabilities as part of its platform. The current Cloud ML feature set is a bit limited compared to some of its competitors but Google has proven its ability to iterate faster in this space.

HYPERFRAME

hyperframe.com

1-10 employees

Since 2018

COMPANY RANKING

HYPERFRAME

EVERGREEN INDEX #4796

Representing this company? Click here to claim and customize this page!

Hyperframe WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

HYPERFRAME

hyperframe.com

1-10 employees

Since 2018

COMPANY RANKING

HYPERFRAME

EVERGREEN INDEX #4796

RELATED COMPANIESRANK

STORIES ABOUT HYPERFRAME

Representing this company? Click here to claim and customize this page!

Hyperframe WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!