575 reads

A/B Experiments with Statsig Layers

by Daniel LoombApril 12th, 2022

Too Long; Didn't Read

The starter project can be downloaded [here [here] or you can follow the steps below. As the player, you can select a move and throw a move to play against you. Once a move has been picked, the page will update to display the result. The following command creates the files we will need: touch index.html styles.css and scripts.js. You still need to update the.scripts.js to include your client SDK key. You can also download the.starter project and get the completed.

Companies Mentioned

Coin Mentioned

featured image - A/B Experiments with Statsig Layers

If you just want to skip the tutorial and get the completed project, you can download it from https://tinyurl.com/rps-complete (You still need to update the scripts.js to include your client SDK key).

Starter

First, we need to set up the website we will be using Statsig on.

The starter project can be downloaded from https://tinyurl.com/rps-starter, or you can follow the steps below.

The following command creates the files we will need:

touch index.html styles.css scripts.js

In index.html, paste the following:

<html lang="en">

<head>
  <link rel="stylesheet" href="styles.css">
  <script src="scripts.js"></script>
</head>

<body>
  <div class="container">
    <p id="computer-move-text">Pick a Move</p>
    <p id="result-text">...</p>
    <div>
      <button onclick="onPick(0)" class="button">✊</button>
      <button onclick="onPick(1)" class="button">🖐</button>
      <button onclick="onPick(2)" class="button">✌</button>
    </div>
  </div>
</body>

</html>

In scripts.js, paste the following:

let moves = ["✊", "🖐", "✌"];


function onPick(playerIndex) {
  const randomMove = moves[Math.floor(Math.random() * moves.length)];

  const cpuIndex = moves.indexOf(randomMove);
  const loseIndex = (playerIndex + 1) % moves.length;

  let result = "Won";
  if (cpuIndex === playerIndex) {
    result = "Tied";
  } else if (cpuIndex === loseIndex) {
    result = "Lost";
  }

  document.getElementById("computer-move-text").innerHTML =
    "Computer picked " + randomMove;
  document.getElementById("result-text").innerHTML = "You " + result;
}

And in styles.css, paste the following:

.container {
  transform: translate(-50%, -50%);
  position: absolute;
  left: 50%;
  top: 50%;
}

p {
  text-align: center;
  font-size: 32px;
  font-family: "Trebuchet MS", "Lucida Sans Unicode", "Lucida Grande",
    "Lucida Sans", Arial, sans-serif;
  font-weight: 700;
}

#result-text {
  font-size: 40px;
  font-weight: 900;
}

.button {
  outline: none;
  border: none;
  background: none;
  font-size: 60px;
  cursor: pointer;
  border-radius: 100px;
  padding: 10px 20px;
}

.button:hover {
  background-color: rgba(0, 0, 0, 0.133);
}

If you now open the index.html file in your web browser, you should see our basic Rock Paper Scissors game. As the player, you can select a move to throw and the computer will randomly select a move to play against you. Once a move has been picked, the page will update to display the result.

Adding Statsig

Let’s now add Statsig so we can start experimenting with our RPS game.

In our index.html file, let’s include the Statsig JS SDK by adding a script import to our head tag.

<head>
  <link rel="stylesheet" href="styles.css">
  
  <!-- Statsig JS -->
  <script src="https://cdn.jsdelivr.net/npm/[email protected]/build/statsig-prod-web-sdk.min.js"></script>
  
  <script src="scripts.js"></script>
</head>

Now in our scripts.js file, we can add calls to the Statsig SDK. The following code uses the Statsig SDK we loaded from the JSDelivr CDN to make an initialize that will fetch all data relevant to the given user Statsig’s servers. This initialize call hits the network and is asynchronous, so we must await it.

let moves = ["✊", "🖐", "✌"];

(async () => {
  await statsig.initialize(
    "<CLIENT_SDK_KEY>",
    {
      userID: "some_user_id",
    }
  );
})();

function onPick(playerIndex) {

// •••

Now at the end of the onPick function, let’s add a logEvent call to log that a game was played. The following code logs a game_played event containing the result of the game. This will be the metric we are attempting to move in all upcoming experiments. If an experimental feature is “Good”, it should increase the volume of this metric as more people play.

  document.getElementById("computer-move-text").innerHTML = "Computer picked " + randomMove;
  document.getElementById("result-text").innerHTML = "You " + result;

  statsig.logEvent("game_played", result.toLowerCase()); // new line
}

Now whenever a game is played, an event is logged with Statsig. We can verify this by going to the metrics page on console.statsig.com

Running an Experiment

We will now set up our first experiment to test new features added to our RPS game. Our first experiment will add new moves to the game in the form of Card Suits (♣️, ♥️, ♦️, ♠️). This doesn’t really make sense for an RPS game, but it will demonstrate how experimentation works.

Create a Layer

Login to console.statsig.com and we’ll add a new layer. If you haven’t already, you’ll need to create a project by selecting “Create New” in the top-left dropdown.

Once you have a project, navigate to the experiments page by selecting “Experiments” from the left navigation list. When the page loads, select the “Layers” tab, and then create a new layer. We’ll call our new layer “rps_experiments”.

Setup an Experiment

On the Experiments page, with the layers tab selected, you can click “Create new Experiment in Layer“.

Alternatively, from the Experiments page, with the experiments tab selected, you can click “Create”.

Both approaches will display the experiment creation dialog, but the latter will require you to set the layer yourself.

In the creation dialog, we’ll name our experiment “alternative_movesets”. Be sure that the layer field, below Advanced, is filled in with the layer we created above.

With our experiment created, we can now add parameters to our layer and then select them as part of this experiment. We are going to set up a Test and Control group for this experiment. The Control will keep the same moves as we have currently, but the Test group will use card suits instead of hand signs.

In the experiments page, select “+ Add Parameter” and add our new moves. They will just be an array of strings which are our card suit emojis.

Once done, your setup should look similar to below. With Control just use the current ✊, ✋ and ✌️ and Test using the updated ♠️, ♥️, ♣️, and ♦️.

Implement in Code

Let’s update our index.html so we can dynamically add the buttons at runtime. We’ll change the body to look like this, giving the buttons div an id of "actions”.

<body>
  <div class="container">
    <p id="computer-move-text">Pick a Move</p>
    <p id="result-text">...</p>
    
    <div id="actions"></div> <!-- updated line -->
  
  </div>
</body>

Then we’ll update the scripts.js file to pull the move set from Statsig and dynamically add the buttons to the DOM. The new setup function should look like the following. We first get a reference to our layer via getLayer(“rps_experiments”) and then get our moves by calling layer.get(“moves”, moves) . The first argument is the name of our moves parameter and the second is a default value should the parameter not be found.

let moves = ["✊", "🖐", "✌"];

async () => {
  await statsig.initialize(
    "<CLIENT_SDK_KEY>",
    {
      userID: "some_user_id",
    }
  );

  // -- new lines below

  const layer = statsig.getLayer("rps_experiments");
  moves = layer.get("moves", moves);

  const actions = document.getElementById("actions");

  // Dynamically add buttons to DOM
  moves.forEach((val, index) => {
    const button = document.createElement("button");
    button.textContent = val;
    button.onclick = () => onPick(index);
    button.className = "button";
    actions.appendChild(button);
  });

  // --

})();

Now if we reload the page, depending on what experiment group your user is in, you will either see no changes, or you will see the new moves displayed.

If you don’t see any changes, change the userID until you get one that is in the “Test” group. You can also use a Feature Gate to override what group a given user is in, but that is beyond this tutorial (learn more here).

Extending the Experiment

So we ran an experiment using getLayer but so far the flow hasn’t been that different from the getExperiment API. Let’s add some more parameters to experiment with to show the power of getLayer.

Giving our Opponent a Name

First, we’ll add a new parameter to our layer that will contain names to give to our AI Opponent. On our layers page (Experiments > Layers Tab > rps_experiments), add a parameter to the Layer by selecting “+ Add Parameter”. In the parameter creation dialog that appears, enter the values displayed below.

Now, in our scripts.js file, in the onPick function, let’s add a call to our new ai_name parameter. We will reference the same layer as before, but this time our parameter name is “ai_name“. We will again provide a default value (“Computer“) to fall back to incase the parameter cannot be found. With our aiName fetched from the layer, we then update the DOM to use our new name.

const aiName = layer.get("ai_name", "Computer"); // new line
document.getElementById("computer-move-text").innerHTML = `${aiName} picked ${randomMove}`; // updated

Adding a Scoreboard

Similar to our ai_name parameter, we will add a new score_board_enabled parameter to our layer. This time, however, we will use a simple boolean, instead of a string.

Now, at the top of scripts.js, we’ll add a new variable for holding our score. Then, in the onPick function, we’ll increment the score on a win and display it in the DOM. We will use our scoreboard_enabled boolean to control whether or not this is displayed to the user.

// Top of scripts.js

let moves = ["✊", "🖐", "✌"];
let score = 0; // new line

// onPick function in scripts.js

function onPick() {

  // •••

  let result = "Won";
  if (cpuIndex === playerIndex) {
    result = "Tied";
  } else if (cpuIndex === loseIndex) {
    result = "Lost";
  } else {
    score++;
  }
  
  // •••

  // new lines added below
  if (layer.get("scoreboard_enabled", false)) {
    document.getElementById("scoreboard").innerHTML = "Your Score: " + score;
  }
}

Running a Second Experiment

Now with these new parameters, we can run another experiment (even if the first is still running) that contains the original moves parameter as well as the newer ai_name and scoreboard_enabled parameters.

Because we are using getLayer rather than getExperiment, we don’t need to update our code to explicitly reference the experiment, getLayer automatically returns the correct values depending on which experiment, if any, the user is in.

Similar to our previous experiment, we will go to the Experiments page and add a new experiment (Again, be sure to set our rps_experiments layer under Advanced > Add Layer > Layer). Let’s call this new experiment “updated_features”.

For this experiment, we will create 3 groups (Control, Test_Pokemon and Test_AI).

Control will be the same as before. Test_Pokemon will contain Pokemon-themed emoji and opponent. Test_AI will also have different emoji and opponent name, but will also display the scoreboard. This should result in the setup below.

You can now run the updated_features experiment in parallel with the first alternative_movesets experiment. Depending on what group the user is in, they will be shown a different RPS game than other users. Change the userID if you wish to try out the different experiences.

Evaluating the Experiments

So our experiment has been running for a few days and it’s time to check on the results.

Going to the Results tab on our updated_features experiment we can evaluate how our experiment did.

We can see that 1.17K users have been exposed to our experiment. We have designated the game_played event as a Key Metric. Looking at the results, we can see that the Test Group “Test_Pokemon” has had a statistically significant lift. If this were a real feature, the Test_Pokemon group would be a clear ship.