Running a successful online application is an exciting journey, But it is also full of challenges. It starts from product-market fit (PMF), continues with distribution challenges, pricing, optimization, and maintenance. Traditional ways to optimize games rely on a long exploration process, which holds a few significant cons: Executing a valid AB test requires proper product and technological capabilities to expose different groups of users to the different game experiences, make sure these groups don't mix, and that there were external or additional product changes that could affect the test results. Tests should be significant; otherwise, the results might be miss leading especially when measuring paying customers. Given an average of 5% paying customers, and let's say we need 200 paying customers in each group, means overall 8000 users in the test. In the User acquisition world, that could be anywhere between $20,000-$40,000 media spent. Also, small/medium-sized companies are limited with the amount of testing the run due to its limited DAU. The thing about AB tests is that the data gets more interesting a few weeks after the test is off and not on its early days. This long waiting time is expensive. A simulation approach and data-sandbox strategy is taking these barriers and makes them redundant: - simulation methodologies relays on data volume only on the training part, where the 'customer behavior' has been analyzed and different segments have been created. Statistical Significance While simulation has been executed, the number of users been imitated is basically unlimited, so every leaf on this huge decision tree( the product flow) gets a proper representation. There are numerous ways to build a simulation, which can provide the solution to rich and poor data environments. - while real-world testing relies on users volume(marketing is the biggest spent for online companies), in the sandbox approach, the Cost moves to IT(ML) and relies mostly on data processing, training, and execution. Lower Cost Efficient modeling with a scalable infrastructure can reduce the Cost by 100X. - applying a test to a live and running game contains risk - underperforming test results, time-consuming, in PvP games, the latest can break the fairness mechanism. No-Risk In the Sandbox, everything is possible. The limitation can be found mostly in creativity and operational barriers, like limited ability to expose different gameplay to different segments. AB testing has been, mostly, on a liner order with a limited pipe due to many reasons( some were described here). Using a simulation strategy enables measuring hundreds(!!) of different scenarios and evaluating the results within minutes to hours. Faster time to decision - I see that a lot, that once considering making changes in monetization features or economy objects in general, there are many concerns of 'braking something' or even disrupting a working money generating process. Nonstop Innovation - The power of Sandbox means that product and monetization teams can constantly look for creative ways to increase both retention, engagement, and monetization without interfering with a running product. true, this is the elephant in the room. Working with a simulation strategy means getting a good understanding of the potential impact of decisions taken. It is also clear that a simulation will never be 100% accurate, but 75%-85% accuracy provide a great horizon and limit potential mistakes. Accuracy -
