Wouldn’t it be cool if we could train a machine learning model to predict machine performance? In this post, we’ll look at a linear regression model I built using to predict the performance of a machine given hardware and software specifications. BQML All of the work I do is software related so I rarely think about the hardware running my code — this gave me an opportunity to learn about the hardware side of things. The dataset: SPEC To train this model I used data from *, an organization that builds tools for evaluating computer performance and energy efficiency. They published a series of , where they used 43 different tests to evaluate the performance of specific hardware. Their benchmarks are divided into 4 categories: integer and floating point tests measuring performance through both (called SPEC speed) and (called SPEC rate): SPEC benchmarking results from 2017 time throughput The four categories of SPEC 2017 benchmarks. For this post, I’ll focus mostly on the Integer Speed tests. Each test includes specs on the hardware and software of the system where the test was run, along with results for several benchmarks. The benchmarks are intended to simulate different types of applications or workloads. Here’s an example of the hardware and software data for : one test And here’s a subset of the SPEC benchmarks for the above system: Why so many benchmarks, and what do they all mean? SPEC’s goal is to evaluate machine performance across a variety of common workloads, all . For example, the evaluates the speed of a machine running the for playing a game of Chess. SPEC also has benchmarks for other ML workloads, ocean modeling, video compression, and more. outlined here 631.deepsjeng_s benchmark alpha-beta search algorithm Is there a relationship between hardware, software, and benchmark results? The short answer is — yes! But I wanted to confirm this before naively dropping inputs into a model and hoping for the best. To take a closer look at the data, I wrote a Python script to parse the CSV files for each test using the results. The script extracts data on the hardware and software used in each test, along with the score for each of the benchmarks run on that machine. In the script, I first initialize a with the data I want to capture from each test: CPU2017 Integer Speed dict() Then I write the header row to the CSV using the keys from the dict above: Here’s an that I’ll be iterating over in the script. The next step is to iterate over the local directory where I’ve saved all the CSV files for the integer speed tests, and use the Python module to read them: example CSV csv The files aren’t typical CSV format, but they all follow the same pattern so I can grab the benchmark data with the following: Using the benchmark start and end indices, I can iterate over the rows of the CSV that contain the benchmarks. Then I look for the specific data points I’m collecting within a CSV. Here’s how I grab the vendor and machine model name: When I’m done collecting all of the machine specs, I create a CSV string of the data and write it to my file: With all of the machine data and benchmark scores in a CSV file, I used a Python notebook and matplotlib to explore relationships between machine specs and benchmark results to see if there was a linear relationship. Here I’ve plotted the nominal speed of a machine and the (alpha-beta search) benchmark: 631.deepsjeng_s Looks like I could draw a line or curve that roughly estimates the relationship above. Let’s take a look at another input. Here’s a plot of a machine’s number of cores and the benchmark: 631.deepsjeng_s While there’s some relationship between the inputs and benchmark measured above, it’s clear that we couldn’t use a single feature (like number of cores) to accurately predict a machine’s speed. There are many features of a machine that affect its performance, and to make use of all of them we’ll build a linear regression model to evaluate the speed of a machine. Here’s what we’ll use as inputs: - The hardware vendor- The machine model name- Nominal and max speed of the microprocessor in MHz- Number of cores- Number of memory channels- Size of L1, L2, and L3 caches- Memory in GB- Memory speed- The OS running on the system- The compiler being used- The company running the test (Lenovo, Huawei, Dell, etc.) A linear regression model typically outputs a single numeric value, so we’ll need to create separate models to predict each benchmark score. It’s important to note that not all of our inputs are numerical — vendor name, model name, and OS name are categorical, string inputs. Since a linear regression model expects numerical inputs, we’ll need a way to encode these as integers before feeding them into our model. Building a linear regression model with BigQuery Machine Learning We could write the code for our linear regression model by hand, but I’d like to focus on the predictions generated by my model rather than the nitty gritty of the model code. (BQML) is a great tool for this job. If I create a BigQuery table with my input data and benchmarks, I can write a SQL query to train a linear regression model. BQML will handle the underlying model code, hyperparameter tuning, splitting my data into training and test sets, and converting categorical data to integers. BigQuery Machine Learning Since I’ve already got the data for my model as a CSV, I can use the BigQuery web UI to upload this data into a table. Here’s the schema for the table: And here’s a preview of the data: To get a sense of one of the categorical features we’re working with, let’s take a look at the field in our table, which refers to the company that ran an individual test. Here’s the query we’ll run to get a breakdown of our data by test sponsor: sponsor The results: And here are the top operating systems used in the tests: I can train my model with a single SQL query in BQML: This took 1 minute and 14 seconds to train. When it completes, I can look at the stats for each iteration of training: Loss metrics for each epoch of training in BQML The number I want to focus on here is the — this is the loss metric calculated after each iteration of training. We can see that it has steadily decreased from the first to the last iteration. To get additional evaluation metrics, I can run an query. Here are the results: Evaluation Data Loss ML.EVALUATE Evaluation metrics on my BQML model Mean squared error (MSE) measures the difference between the values our model predicted using the test set and the actual values. You can also think of it as the distance between your regression (best fit) line and the predicted values. A smaller value is better, and our model’s MSE is very good. .0121 Since this is a regression model (predicting a continuous numerical value), the best way to see how it performed is to evaluate the difference between the value predicted by the model and the ground truth benchmark score. We can do this with an query. ML.PREDICT When this query runs it’ll create a new field prefixed with and the name of our label column (in this case ). Let’s run across the original dataset and output a few features of the system being tested, the actual speed test result, and predicted benchmark: predicted_ _631_deepsjeng_s ML.PREDICT Our predictions are very close to the actual score values! To measure this another way, we can subtract the predicted score from the actual score, and then get the average of the absolute values of these differences: On average, our model’s prediction is only .04 off from the original value — pretty impressive. Generating predictions on new data Now that I’ve got a trained model, I can use it to predict the speed benchmarks for a machine that wasn’t part of my training set with the following query: The SPEC speed score for this machine on the benchmark should be . When I run this query my model predicts which is pretty close to the actual value. I can now use this model to generate predictions on new combinations of hardware and software specs that weren’t included in the test dataset. In addition to the alpha-beta search benchmark, I created models for other Integer Speed benchmarks that had similar accuracy to this one — let me know if you’re interested in these results and I can share details. 631.deepsjeng_s 5.19 5.20 Because I didn’t have to write any of the underlying model code, I was able to focus on feature engineering and finding the right combination of inputs to build a high accuracy model — thanks BQML! More with SPEC data If you’re interested in experimenting with this data and creating your own models, check out the . It’s also worth noting that SPEC has lots of data on previous CPU benchmarks. The have many more years worth of data (though SPEC has retired this set), so you could train a model using a similar approach to predict the performance of older machines. Integer Speed benchmark results here 2006 SPEC CPU benchmarks This post focused on integer speed benchmarks. I haven’t included the other SPEC benchmarks here (floating point speed and throughput) because they didn’t perform as well with linear regression models (MSE in the 100s compared to MSE of less than 0.1 in the integer speed models). I’m planning to experiment with these tests using other types of models, stay tuned :) In addition to SPEC, there are also many other performance benchmarks you could use like , which is used by . LINPACK Top 500 Have feedback? This is my first foray into hardware performance data, so I’d love any feedback you have on the features I used as inputs, other types of models to try, or anything else. Leave a comment below or find me on Twitter at . @SRobTweets If you want to learn more about BQML, my teammates have some great posts: covers predicting Stack Overflow response times, and walks you through building a model to predict flight delays. this one this one 😀 Thank you to these awesome people for their contributions and feedback: Brian Kang , Shane Glass , Abhishek Kashyap , Eric Van Hensbergen , Jon Masters , and a few other SPEC experts at Red Hat ********************* *Data source: Data retrieved on November 11 2018. SPEC Fair Use Rules: https://www.spec.org/cpu2017/results/cint2017.html https://www.spec.org/fairuse.html
