If you just want to skip the tutorial and get the completed project, you can download it from https://tinyurl.com/rps-complete (You still need to update the to include your client SDK key). scripts.js Starter First, we need to set up the website we will be using Statsig on. The starter project can be downloaded from https://tinyurl.com/rps-starter, or you can follow the steps below. The following command creates the files we will need: touch index.html styles.css scripts.js In , paste the following: index.html <html lang="en">

<head>
  <link rel="stylesheet" href="styles.css">
  <script src="scripts.js"></script>
</head>

<body>
  <div class="container">
    <p id="computer-move-text">Pick a Move</p>
    <p id="result-text">...</p>
    <div>
      <button onclick="onPick(0)" class="button">✊</button>
      <button onclick="onPick(1)" class="button">🖐</button>
      <button onclick="onPick(2)" class="button">✌</button>
    </div>
  </div>
</body>

</html> In , paste the following: scripts.js let moves = ["✊", "🖐", "✌"];


function onPick(playerIndex) {
  const randomMove = moves[Math.floor(Math.random() * moves.length)];

  const cpuIndex = moves.indexOf(randomMove);
  const loseIndex = (playerIndex + 1) % moves.length;

  let result = "Won";
  if (cpuIndex === playerIndex) {
    result = "Tied";
  } else if (cpuIndex === loseIndex) {
    result = "Lost";
  }

  document.getElementById("computer-move-text").innerHTML =
    "Computer picked " + randomMove;
  document.getElementById("result-text").innerHTML = "You " + result;
} And in , paste the following: styles.css .container {
  transform: translate(-50%, -50%);
  position: absolute;
  left: 50%;
  top: 50%;
}

p {
  text-align: center;
  font-size: 32px;
  font-family: "Trebuchet MS", "Lucida Sans Unicode", "Lucida Grande",
    "Lucida Sans", Arial, sans-serif;
  font-weight: 700;
}

#result-text {
  font-size: 40px;
  font-weight: 900;
}

.button {
  outline: none;
  border: none;
  background: none;
  font-size: 60px;
  cursor: pointer;
  border-radius: 100px;
  padding: 10px 20px;
}

.button:hover {
  background-color: rgba(0, 0, 0, 0.133);
} If you now open the file in your web browser, you should see our basic Rock Paper Scissors game. As the player, you can select a move to throw and the computer will randomly select a move to play against you. Once a move has been picked, the page will update to display the result. index.html Adding Statsig Let’s now add Statsig so we can start experimenting with our RPS game. In our file, let’s include the Statsig JS SDK by adding a script import to our head tag. index.html <head>
  <link rel="stylesheet" href="styles.css">
  
  
  <script src="https://cdn.jsdelivr.net/npm/statsig-js@4.13.0/build/statsig-prod-web-sdk.min.js"></script>
  
  <script src="scripts.js"></script>
</head> Now in our file, we can add calls to the Statsig SDK. The following code uses the Statsig SDK we loaded from the JSDelivr CDN to make an that will fetch all data relevant to the given user Statsig’s servers. This initialize call hits the network and is asynchronous, so we must await it. scripts.js initialize let moves = ["✊", "🖐", "✌"];

(async () => {
  await statsig.initialize(
    "<CLIENT_SDK_KEY>",
    {
      userID: "some_user_id",
    }
  );
})();

function onPick(playerIndex) {

// ••• Now at the end of the function, let’s add a call to log that a game was played. The following code logs a event containing the result of the game. This will be the metric we are attempting to move in all upcoming experiments. If an experimental feature is “Good”, it should increase the volume of this metric as more people play. onPick logEvent game_played document.getElementById("computer-move-text").innerHTML = "Computer picked " + randomMove;
  document.getElementById("result-text").innerHTML = "You " + result;

  statsig.logEvent("game_played", result.toLowerCase()); // new line
} Now whenever a game is played, an event is logged with Statsig. We can verify this by going to the metrics page on console.statsig.com Running an Experiment We will now set up our first experiment to test new features added to our RPS game. Our first experiment will add new moves to the game in the form of Card Suits (♣️, ♥️, ♦️, ♠️). This doesn’t really make sense for an RPS game, but it will demonstrate how experimentation works. Create a Layer Login to console.statsig.com and we’ll add a new layer. If you haven’t already, you’ll need to create a project by selecting “Create New” in the top-left dropdown. Once you have a project, navigate to the experiments page by selecting “Experiments” from the left navigation list. When the page loads, select the “Layers” tab, and then create a new layer. We’ll call our new layer “rps_experiments”. Setup an Experiment On the Experiments page, with the layers tab selected, you can click “Create new Experiment in Layer“. Alternatively, from the Experiments page, with the experiments tab selected, you can click “Create”. Both approaches will display the experiment creation dialog, but the latter will require you to set the layer yourself. In the creation dialog, we’ll name our experiment “alternative_movesets”. Be sure that the layer field, below Advanced, is filled in with the layer we created above. With our experiment created, we can now add parameters to our layer and then select them as part of this experiment. We are going to set up a Test and Control group for this experiment. The Control will keep the same moves as we have currently, but the Test group will use card suits instead of hand signs. In the experiments page, select “+ Add Parameter” and add our new moves. They will just be an array of strings which are our card suit emojis. Once done, your setup should look similar to below. With Control just use the current ✊, ✋ and ✌️ and Test using the updated ♠️, ♥️, ♣️, and ♦️. Implement in Code Let’s update our so we can dynamically add the buttons at runtime. We’ll change the body to look like this, giving the buttons an id of "actions”. index.html div <body>
  <div class="container">
    <p id="computer-move-text">Pick a Move</p>
    <p id="result-text">...</p>
    
    <div id="actions"></div> 
  
  </div>
</body> Then we’ll update the file to pull the move set from Statsig and dynamically add the buttons to the DOM. The new setup function should look like the following. We first get a reference to our layer via and then get our moves by calling . The first argument is the name of our moves parameter and the second is a default value should the parameter not be found. scripts.js getLayer(“rps_experiments”) layer.get(“moves”, moves) let moves = ["✊", "🖐", "✌"];

async () => {
  await statsig.initialize(
    "<CLIENT_SDK_KEY>",
    {
      userID: "some_user_id",
    }
  );

  // -- new lines below

  const layer = statsig.getLayer("rps_experiments");
  moves = layer.get("moves", moves);

  const actions = document.getElementById("actions");

  // Dynamically add buttons to DOM
  moves.forEach((val, index) => {
    const button = document.createElement("button");
    button.textContent = val;
    button.onclick = () => onPick(index);
    button.className = "button";
    actions.appendChild(button);
  });

  // --

})(); Now if we reload the page, depending on what experiment group your user is in, you will either see no changes, or you will see the new moves displayed. If you don’t see any changes, change the userID until you get one that is in the “Test” group. You can also use a Feature Gate to override what group a given user is in, but that is beyond this tutorial (learn more ). here Extending the Experiment So we ran an experiment using but so far the flow hasn’t been that different from the API. Let’s add some more parameters to experiment with to show the power of . getLayer getExperiment getLayer Giving our Opponent a Name First, we’ll add a new parameter to our layer that will contain names to give to our AI Opponent. On our layers page (Experiments > Layers Tab > rps_experiments), add a parameter to the Layer by selecting “+ Add Parameter”. In the parameter creation dialog that appears, enter the values displayed below. Now, in our file, in the function, let’s add a call to our new parameter. We will reference the same layer as before, but this time our parameter name is “ai_name“. We will again provide a default value (“Computer“) to fall back to incase the parameter cannot be found. With our aiName fetched from the layer, we then update the DOM to use our new name. scripts.js onPick ai_name const aiName = layer.get("ai_name", "Computer"); // new line
document.getElementById("computer-move-text").innerHTML = `${aiName} picked ${randomMove}`; // updated Adding a Scoreboard Similar to our ai_name parameter, we will add a new score_board_enabled parameter to our layer. This time, however, we will use a simple boolean, instead of a string. Now, at the top of , we’ll add a new variable for holding our score. Then, in the function, we’ll increment the score on a win and display it in the DOM. We will use our scoreboard_enabled boolean to control whether or not this is displayed to the user. scripts.js onPick // Top of scripts.js

let moves = ["✊", "🖐", "✌"];
let score = 0; // new line // onPick function in scripts.js

function onPick() {

  // •••

  let result = "Won";
  if (cpuIndex === playerIndex) {
    result = "Tied";
  } else if (cpuIndex === loseIndex) {
    result = "Lost";
  } else {
    score++;
  }
  
  // •••

  // new lines added below
  if (layer.get("scoreboard_enabled", false)) {
    document.getElementById("scoreboard").innerHTML = "Your Score: " + score;
  }
} Running a Second Experiment Now with these new parameters, we can run another experiment (even if the first is still running) that contains the original moves parameter as well as the newer ai_name and scoreboard_enabled parameters. Because we are using rather than , we don’t need to update our code to explicitly reference the experiment, getLayer automatically returns the correct values depending on which experiment, if any, the user is in. getLayer getExperiment Similar to our previous experiment, we will go to the Experiments page and add a new experiment (Again, be sure to set our layer under Advanced > Add Layer > Layer). Let’s call this new experiment “updated_features”. rps_experiments For this experiment, we will create 3 groups (Control, Test_Pokemon and Test_AI). Control will be the same as before. Test_Pokemon will contain Pokemon-themed emoji and opponent. Test_AI will also have different emoji and opponent name, but will also display the scoreboard. This should result in the setup below. You can now run the updated_features experiment in parallel with the first alternative_movesets experiment. Depending on what group the user is in, they will be shown a different RPS game than other users. Change the userID if you wish to try out the different experiences. Evaluating the Experiments So our experiment has been running for a few days and it’s time to check on the results. Going to the Results tab on our updated_features experiment we can evaluate how our experiment did. We can see that 1.17K users have been exposed to our experiment. We have designated the game_played event as a Key Metric. Looking at the results, we can see that the Test Group “Test_Pokemon” has had a statistically significant lift. If this were a real feature, the Test_Pokemon group would be a clear ship.

Too Long; Didn't Read

Making security fun one episode at a time

A/B Experiments with Statsig Layers

Too Long; Didn't Read

Daniel Loomb

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

A/B Experiments with Statsig Layers

Too Long; Didn't Read

Daniel Loomb

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES