The subscription data platform for mobile-first companies. Cross-platform subscription infrastructure, revenue analytics, engagement automation, and integrations all in one place to help you grow your app faster.

Qonversion

The lead image for this article was generated by HackerNoon's AI Image Generator via the prompt "apple logo" 's platform presents an array of options for app developers and marketers to promote their apps. Among these, stand out as a robust tool for promoting subscription-based apps. But what makes Apple Search Ads so special? The secret lies in the rich, accurate marketing attribution data they provide. Apple Apple Search Ads In this article, I'll dig into the world of Apple Search Ads, how they function, and how you can leverage them to grow your app. Understanding Apple Search Ads Apple Search Ads is a mobile ad platform that allows developers to advertise their apps directly within the App Store search results. When users search for a specific keyword related to your app, your ad can appear at the top of the search results, thereby increasing your app's visibility and potential downloads. This targeted approach makes it a potent tool for promoting subscription-based apps. The Impact of Apple Search Ads The effectiveness of Apple Search Ads is supported by robust statistics that highlight the platform's potential for success. According to AppsFlyer's benchmark index (https://index.appsflyer.com/), Apple Search Ads is ranking number #1 for non-gaming apps user acquisition by retention, showing high value of users acquired through the platform. These underscore the potential of Apple Search Ads for promoting subscription-based apps, making it a key element in any subscription app marketing strategy. The Benefits of Accurate Marketing Attribution Data Marketing attribution data forms the backbone of any advertising campaign. It helps you determine which campaigns, keywords, or ads drive the most subscriptions or in-app purchases, allowing you to optimize your advertising budget for better results. One of the major benefits of Apple Search Ads is that they provide this data on a keyword level, making it easier to attribute in-app revenue and in-app subscriptions to the exact ad spend. However, accessing this rich data through Apple's API can be challenging without the right technical tools. Simplifying Data Access with Qonversion Most likely you don't have time and resources to build your attribution solution on top of Apple's API. You want to focus on buidling your app and growing it. So you need to use third party solution to jump start you Apple Search Ads user acquisition. We at Qonversion are building the powerful analytics platform specifically designed for subscription apps. For full disclosure, I'm the Co-Founder of Qonversion. Qonversion takes the complexity out of accessing Apple's API by offering this functionality out of the box, enabling you to easily track and analyze your marketing attribution data. Setting up Qonversion is straightforward. Once you've linked your app with Qonversion, you can start receiving accurate marketing attribution data from your Apple Search Ads campaigns. You are getting ASA data in a simple customizable dashboard with all metrics needed - from impressions and installs to subscriptions and revenue. You can also see the ROAS of your campaigns and even keywords. This empowers you to make informed decisions about your ad spend, understand your return on investment, and ultimately grow your subscription base. Apart from simplifying access to Apple Search Ads attribution data, Qonversion provides in-depth subscription analytics that help you understand your app's performance. It offers insights into subscription metrics like conversion rates, retention rates, and churn rates, giving you a holistic understanding of your app's health. It's not just an analytics tool; it's a comprehensive solution for managing and optimizing your subscription-based app. Conclusion Apple Search Ads, combined with the power of Qonversion, can greatly enhance the promotion of your subscription-based apps. By unlocking access to accurate marketing attribution data, you can better attribute in-app revenue and subscriptions to specific campaigns or keywords. This, in turn, empowers you to optimize your ad spend, making every dollar count towards acquiring valuable users for your app.

An Intro to Apple Search Ads for Subscription-Based Apps

AB testing is an essential aspect of mobile app development, especially when it comes to subscription-based iOS and Android apps. By conducting AB tests, app developers and marketers can determine which version of their app's paywall design, pricing strategy, and user experience is most effective at driving subscription revenue growth. In this blog post, I will discuss the importance of AB testing for subscription-based mobile apps and delve into the required infrastructure and technical details needed to conduct these tests accurately. I will also explore the metrics that matter for subscription apps and highlight the significance of speed in launching new tests. Finally, I will introduce Qonversion.io, a subscription revenue growth platform that is specifically designed to provide the right infrastructure and tools for mobile apps. Why Paywalls AB Testing is Important AB testing is crucial for subscription-based mobile apps because it allows developers and marketers to experiment with different paywall designs and pricing strategies to find the most effective one. With AB testing, two versions of the app's paywall design are presented to different segments of the user base. By analyzing user behavior and conversion rates, the app developers can determine which version performs better and adjust the paywall design accordingly. This process helps improve the conversion rate and revenue growth of the app. Infrastructure and Technical Details To conduct AB testing accurately, the required infrastructure must be in place. This infrastructure includes the ability to target specific segments of the user base and accurately measure the results of each test. By using user segmentation, app developers can present different versions of the paywall to specific groups of users based on their demographics, behavior, and preferences. Accurately measuring the results of each test is also critical to determine the effectiveness of each paywall design. The use of analytics tools, such as Google Analytics or Mixpanel is simply not enough to run and measure AB tests for subscription apps. , can help measure user behavior and conversion rates accurately. The most important metrics are hard to track without the analytics built for subscription AB testing. Metrics that Matter for Subscription Apps AB Testing When conducting AB testing for subscription-based apps, it is essential to focus on metrics that matter. The most crucial metrics for subscription apps include trial conversions, subscription renewals, and total revenue generated. These metrics help determine the success of the app's paywall design and pricing strategy. By focusing on these metrics, app developers can adjust their paywall design and pricing strategy to maximize revenue growth. Sometimes changing just a little detail in your paywall and pricing can lead to hundreds of percent of growth in a matter of months. Speed of Launching New Tests The speed of launching new AB tests is crucial for the success of subscription-based mobile apps. Speed of iteration is the most important skill for successful startups. As Mike Tyson once said, "Speed kills." Paul Graham, co-founder of Y Combinator, also emphasized the importance of speed, stating, "The most successful startups are the ones that manage to iterate quickly." So in the context of mobile app AB testing, the velocity is determined by the infrastructure you have. It is essential to have tools that allow the app to conduct AB tests without releasing new app versions. This way, you can make quick iterations and adjust the paywall design and pricing strategy accordingly without days and weeks of going through app stores reviews for each new test. Qonversion AB testing Capabilities Qonversion.io, the platform I co-founded in 2019, is a subscription revenue growth platform specifically designed to provide the infrastructure and tools required for subscription-based mobile apps to conduct AB testing and improve their revenue growth. With Qonversion, app developers can segment their user base and target specific groups of users with different paywall designs and pricing strategies. Qonversion also provides the best subscription analytics to measure the results of each test accurately. Additionally, Qonversion offers in-app purchase validation, revenue tracking, and subscription renewal tracking to provide a complete view of the app's revenue performance. The Qonversion AB testing tool is like Firebase AB testing but specifically designed to run tests for subscription app paywalls and subscription pricing. You can do all the stuff you used to do with Firebase AB testing, but you get more tools to accurately target what you need. Plus, Qonversion gives you a no-code paywall builder. No-code paywall builder is a game-changing tool for mobile app product owners. This tool enables product owners to create new paywalls without involving engineers in the process, significantly reducing the time and resources required to launch new tests. You can customize the paywall design to match the app's branding and experiment with different layouts, messaging, calls-to-action, subscription prices, and duration. This tool, combined with Qonversion's precise analytics and AB testing capabilities, can help product owners quickly iterate and optimize their paywall design, resulting in increased subscription revenue growth for their mobile app. In conclusion, AB testing is crucial for subscription-based mobile apps to improve their revenue growth. The required infrastructure and tools must be in place to conduct these tests accurately, and focusing on the right metrics is critical for success. The speed of launching new tests is essential. By using AB testing and the right tools, app developers and marketers can quickly get to a growth trajectory.

Paywall AB Testing for Subscription-Based IOS and Android Apps

Mobile in-app spending is a giant $130B+ market. The payments keep flowing through Apple and Google app store billing platforms. Apple and Google's control over in-app payments for virtual services remains intact despite significant external pressure by regulators and private companies to lower platform fees and allow third-party payment processors. Given the size of the opportunity, the trend of providing payment options bypassing the in-app monopoly of Apple and Google is on the rise, along with payment infrastructure companies facilitating this trend. Overview of in-app payments on the app stores Apple first introduced in-app purchases in 2009, the following year after the App Store was revealed to the public. The in-app purchases became a popular mobile app monetization strategy alongside ads. In-app payments made it simple to allow free app downloads and monetize by unlocking premium features in the app. Google followed this monetization model with its Google Play store. A little more than a decade later, consumer app store spending amounts to a massive $ billion (as of 2021 based on SensorTower data) and continues growing by more than 20% annually. 130 The rise of in-app subscriptions Historically games were driving the largest chunk of in-app spending, accounting for roughly just five years ago. But the growth rate of non-gaming apps has been higher in recent years, powered by in-app subscriptions. Developers of all sizes are widely adopting in-app subscriptions. Giants like , Snap, Adobe, Twitter, Telegram, Tinder, and are examples of companies that introduced consumer subscriptions on the app store, earning hundreds of millions in the app stores. 70% Microsoft Duolingo Even recently announced plans to introduce a subscription model similar to Twitter. Many others in the verticals like productivity, social media, streaming, dating, fitness, and education embraced the benefits of recurring monetization with auto-renewable in-app subscriptions. As a result of this trend, non-gaming apps' share of mobile in-app spending reached more than 50% for the first time in Q2 of 2022. Facebook Monopolistic approach and governmental scrutiny While Apple and Google's app stores became powerful mobile app distribution platforms, they also captured a significant share of in-app revenue while enforcing payments through their payment gates. This situation has become the subject of government scrutiny and numerous litigations from private companies. Apple and Google used to take a 30% commission from all in-app revenue. Applying that rate to $130 in-app spending, we get to 40B in net revenue. So there is a significant monetary incentive to keep things as they are for Apple and Google. So far, under external pressure, Apple has reduced the fee to 15% for small developers making under $1M in annual revenue. And Google followed Apple by reducing its fee to 15% of the of earnings each year. first $1M But even 15% is strikingly higher than the 3-4% fee charged by companies that provide payment services like Stripe, Adyen, and others. It's easy to see why there is a lot of pressure on Apple and Google to reduce the commission further and allow third-party payment options within the apps providing digital services. Epic games vs. Apple. Market expectations. The lawsuit by Epic Games was probably the most publicized story when a private company tried to strong-arm Apple on the subject of in-app payments. Apple removed Epic's Fortnite game from the App Store for implementing a third-party payment option. The case made many headlines, and many market participants expected it to end Apple's control over in-app payments. So much so that companies providing payment services, especially subscription management, have started preparing solutions to facilitate third-party in-app subscription payments. One of these companies was Paddle. Paddle provides subscription management for SaaS and web-based companies and raised $200 million at a $1.4 billion valuation in 2022. Paddle’s team to offer their subscription management solution for mobile apps to tap into lucrative mobile subscriptions. They even prepared the landing page for the launch, which is still available on their website. planned product Similarly, Revenuecat, a company managing $1B+ of in-app subscription revenue on app stores, hinted it was working on tools to facilitate quick in-app implementation of third-party payments. The company I run, , enabling in-app subscriptions and managing $400M+ in mobile subscription revenue on Apple and Google, was also eyeing supporting alternative payment options with our iOS and Android SDKs. Qonversion But the Epic vs. Apple case went differently than most expected. The court decided Apple was not acting as a monopolist and could control payment options on its platform. It became apparent that the market expectations needed to be revised, as any solution providing third-party in-app payments would violate the current App Store policy. Paddle it could not launch its mobile payments offer until further clarifications from Apple. And there has been no news ever since. Likewise, we at Qonversion postponed releasing tools to facilitate 3rd party in-app payments. announced Alternative payment options outside of apps At the same time, in the Epic vs Apple trial, the court ordered Apple to allow developers customer communication on external payment options outside an app. Arguably this is a significant opportunity missed by many developers at the moment and the direction where a lot of in-app spending started moving to. While third-party payments can't be integrated directly into an app selling virtual services, they can be used outside of an app. This does not violate Apple’s payment policies. Using an external payment option for one-off purchases is problematic though, as it creates an awful user experience. But providing a web-based onboarding and payment experience for a subscription-type app is more logical. Users can go through a web onboarding and subscribe before downloading an app. Once a user starts a subscription managed by Stripe or any other payment web payment processor, a developer provides him with the link to download an app, authenticate him on mobile and unlock premium access. While this requires additional development, the commission reduction from 15-30% to just 3-4% is usually worth it. Web-to-app subscription management trends So companies providing subscription management are introducing tools to bridge the gap between web and mobile subscription management. The required infrastructure includes a cloud back-end to store subscription states, APIs, and mobile SDKs to make it easy for mobile developers to get subscription data within the app. From my company's data, we see that 30% of mobile apps monetizing with in-app subscriptions have either introduced a web payment option already or looking to do it this year. And based on our client's feedback, the most popular web subscription provider across the board is Stripe. Stripe is winning due to its global footprint, simple interface, and light-weighted KYC account opening process. As this trend evolves, we expect to see more revenue shifts from native in-app subscriptions to external payment gates with the help of subscription management companies. While Apple and Google continue their fight to control app payments, they are already losing some revenue to the web. The massive difference between commissions on app stores and the web creates a significant incentive for this shift. And if there is demand, the technology will facilitate it. Payments control is not the only problem Pressure from governments on Apple and Google remains high. For example, South Korea has passed a law that Apple and Google must allow developers to use other payment systems on the app stores in 2021. European Economic Area (EEA) app stores to allow third-party payment options. But the app stores are being criticized not only for high payment fees. The lengthy and obscure app review process is another major concern of app developers. Some developers claim the process is biased, and apps can be rejected or removed from the App Store without a clear explanation. obligated Here is the message Telegram's founder Pavel Durov posted in August 2022 on his Telegram channel: .. we're often unable to distribute the new versions of Telegram due to the obscure "review process" imposed on all mobile apps by the tech monopolies. For example, our upcoming update – which is about to revolutionize how people express themselves in messaging – has been stuck in Apple's "review" for two weeks, without explanation or any feedback provided by Apple. If Telegram, one of the top 10 most popular apps globally, is receiving this treatment, one can only imagine the difficulties experienced by smaller app developers. So the battle with app stores is not only on the payments but also on the overall control of what's available and who can distribute the apps on the 7 billion mobile devices.

Is Apple’s Monopolistic Control Over the App Store About to End?

Introduction: RSS feeds have long been a popular means of delivering content from different sources, including websites, blogs, and news sites. Integrating and consuming RSS feeds can become much easier by converting them into JSON format. You can have a look , where we discussed the crucial advantages of RSS to JSON Conversion. here Now, let's explore where the RSS to conversion can be utilized. JSON Media companies and content aggregators can significantly benefit from the process of converting RSS feeds into JSON format. RSS feeds are a common method used by news websites and blogs to distribute their content in a standardized format. This simplifies the process of collecting, curating, and displaying news or articles from multiple sources on their platforms. Some known platforms that utilize RSS feeds are: : This developer-focused community aggregates content from various sources using RSS. Daily.dev : A content curation and magazine app that allows users to flip through news stories from various sources. Flipboard : An RSS feed reader that consolidates articles and media from blogs and websites. Feedly In this post, we look at how to convert RSS feeds to JSON using ApyHub’s API and integrate RSS feeds into your applications using NodeJS in just simple steps. Let’s start 🧑‍💻 1. Install the required libraries: To install all the required dependencies, run this command on the terminal of your project folder. npm install axios Once we run in the terminal, the command triggers the installation of the specified dependencies listed in the project's package.json file. The following dependencies are commonly installed. npm install : A popular HTTP client library for making HTTP requests in Node.js. It simplifies the process of sending HTTP requests and handling responses. In this case, Axios is used to make a POST request to the RSS to JSON API, sending the RSS URL. Axios 2. Create a new Project: The next step is to create a new file; for example, create a file and add the following code to import the required libraries we installed.: convertRSSToJSON.js import axios from "axios";

const axios = require('axios'); 3. Define the Conversion Function: will encapsulate the logic for API requests. This function will convert RSS feeds to JSON, handling URL preparation and error management. We will need apy-token which you can find in the . convertRSSToJSON documentation async function convertRSSToJSON(rssURL) {
  const apiURL = "https://api.apyhub.com/convert/rss-url/json?detailed=true";
  const requestData = {
    url: rssURL,
  };
  const options = {
    method: "POST",
    url: apiURL,
    headers: {
      "Content-Type": "application/JSON",
      "apy-token": "YOUR_APY_TOKEN", // Replace with your actual APY token
    },
    data: requestData,
  };
  try {
    const response = await axios.request(options);
    console.log("RSS to JSON conversion successful:", response.data);
    return response.data; // Or handle it as you need
  } catch (error) {
    console.error("An error occurred:", error.message);
    throw error; // Or handle the error as needed
  }
} 4. Invoke the conversion function After invoking the function, pass the sample .xml RSS feed URL. This function sends a request to the Convert RSS to JSON API, which returns the JSON Response. The successful conversion URL is logged to the console. If there is an error during the conversion process, it is caught, and the error message is logged to the console. convertRSSToJSON() const rssURL = 'https://apyhub.com/api/rss'; // Replace with your desired RSS feed URL
convertRSSToJSON(rssURL)
.then(jsonData => {
console.log('Converted JSON:', jsonData);
})
.catch(error => {
console.error('Error during conversion:', error.message);
}); 5. Save the file and execute the script Save the file, open your terminal, navigate to the project directory, and run the following command to execute the script: node convertRSStoJSON.js 6. Once the request is sent, the response will be returned in JSON. {
  "version": "https://jsonfeed.org/version/1",
  "title": "ApyHub",
  "description": "RSS Feed",
  "home_page_url": "https://apyhub.com/rss",
  "items": [
    {
      "title": "Enable Voice-based Interactions in your Retool Application",
      "summary": "This tutorial demonstrates a real-world use case..."
    }
    // More items...
  ]
} And that's it! It was easy, right? Now you can add multiple RSS feeds, simplifying the integration of diverse content sources into your applications. Also, do not forget to handle the errors in case the URL is malformed or there is no valid RSS feed accessible through the URL - in such cases, the API will return appropriate errors. Also published . here

RSS to JSON Conversion: A Step-by-Step Guide for Integration with NodeJS

Causal Analysis – Experimentation (AB Testing) and Statistical Techniques

From Pre-Seed to Series A: Navigating the Key Stages of Product Development

The Power of User Feedback in Product Development

Today, nearly all companies in the IT industry use AB testing to evaluate the value of a feature. They test all changes in applications and websites, allowing someone to determine which feature is the winner every day. As a result, numerous discussions revolve around experiments and AB tests, with a common misconception being the prohibition of applying t-tests for abnormal distributions. Let's delve into this topic and find the correct answer! Problem Formulation Let’s look at a familiar example Imagine a common experiment with different paywalls that suggest a purchase to a customer. We have two variants that show two different and independent groups of users. The decision will be made by a conversion rate to purchase. Every user was assigned to a control or test group randomly, so both samples are independent. Also, every user is assigned 1 or 0 meaning they either made a purchase or not. Thus, our samples have a bi-nominal distribution. In this case, the conversion rate to purchase equals the sample mean. So, we will compare p1n1 and p2n2. Is it possible to use a t-test here or not? Let’s generalize the problem We’re still considering the two independent distributed samples X1 and X2. We know that they have abnormal distributions with finite expected values μ1 and μ2 and finite variances σ1 and σ2. Now, we want to check a hypothesis about the equality of means: Theoretical Proof Let’s consider the Welch’s t-test in a case of different variances. When the variances of both samples are the same, then the Student’s and Welch’s t-tests have similar results. Thus, let’s consider a statistic: Can it be compared with critical values of Student’s distribution? By the central limit theorem, if n1,2 → ∞ Therefore, Also, per Slutsky’s theorem, this convergence in probability is the convergence in distribution. Thus, the constructed statistic t ∼ N(μ1 −μ2,1). Under the null hypothesis t ∼ N(0,1), which is asymptotically equivalent to the Student’s distribution. Empirical Proof Finally, we proved that a t-test can estimate the experiment with two variants of paywall. Also, it is worth mentioning that only two conditions were used for that example: n→∞
and finite of moments for distribution of random value. Let’s try to simulate a pair of distributions to check the prohibition to apply a t-test for them! Bernoulli data Bernoulli data is a widely used case for the experimentation process. A metric like CTR, any type of conversion rate, retention rate, etc., is often chosen as a target metric in an experiment. First, let’s simulate two samples of the Bernoulli distribution with the same parameters. After that, we started increasing the parameter for the second sample. For all samples, we calculated the p-value using Student’s t-test. We see that the p-value drops to zero almost immediately. import numpy as np
from scipy import stats

import matplotlib.pyplot as plt

p_diff_list = []
p_value_list = []

N = 10000
p_1 = 0.01
p_2 = p_1

while p_2 <= 1:
  p_diff_list.append(p_2-p_1)
  x_1 = np.random.binomial(1, p_1, N)
  x_2 = np.random.binomial(1, p_2, N)
  _, p_value = stats.ttest_ind(x_1, x_2, equal_var=False)
  p_value_list.append(p_value)
  p_2 += 0.01

fig, ax = plt.subplots(figsize=(10,4))

ax.plot(p_diff_list, p_value_list)
ax.set(xlabel='p_2 - p_1', ylabel='p-value',
       title='p-value changes for different Bernoulli distributions')

fig.tight_layout()
plt.show() Now, let’s simulate 100,000 times two samples with the same parameters and 100,000 times two samples with different parameters. After that, we can check the first and second-type errors. Indeed, the t-test is a powerful criterion for metrics with parameters like in our simulations—the first type error is around 5%, and the power is around 90%! import numpy as np
from scipy import stats

import matplotlib.pyplot as plt

n_attempts = 100000
N = 10000

# False Negative calculation
p_value_eq_list = []
p = 0.01

i = 1
while i <= n_attempts:
  x_1 = np.random.binomial(1, p, N)
  x_2 = np.random.binomial(1, p, N)
  _, p_value = stats.ttest_ind(x_1, x_2, equal_var=False)
  p_value_eq_list.append(p_value)
  i += 1

False_Negative = sum([i < 0.05 for i in p_value_eq_list])/len(p_value_eq_list)

# False Positive calculation
p_value_noneq_list = []
p_1 = 0.01
p_2 = 0.015

i = 1
while i <= n_attempts:
  x_1 = np.random.binomial(1, p_1, N)
  x_2 = np.random.binomial(1, p_2, N)
  _, p_value = stats.ttest_ind(x_1, x_2, equal_var=False)
  p_value_noneq_list.append(p_value)
  i += 1

False_Positive = 1 - sum([i < 0.05 for i in p_value_noneq_list])/len(p_value_noneq_list) Exponential data Now, let’s consider the exponential data prevalent in the real world, too. Two cases of exponential sample comparisons are shown below. In the first case, samples with the same parameters are considered. The p-value corresponds to reality; it says that samples are similar. In the right part, there is a case with two samples with different parameters and a p-value lower than 0.05. import numpy as np
from scipy import stats

import matplotlib.pyplot as plt

# Generate similar samples
N = 10000
beta = 3

xs_1 = np.random.exponential(beta, N)
xs_2 = np.random.exponential(beta, N)

_, p_value_s = stats.ttest_ind(xs_1, xs_2, equal_var=False)

# Generate different samples
N = 10000
beta_1 = 3
beta_2 = 10

xd_1 = np.random.exponential(beta_1, N)
xd_2 = np.random.exponential(beta_2, N)

_, p_value_d = stats.ttest_ind(xd_1, xd_2, equal_var=False)

n_bins = 100
fig, axs = plt.subplots(1, 2, figsize=(12,4))

axs[0].hist([xs_1, xs_2], n_bins, histtype='bar')
axs[0].set_title('Exponential similar samples')
axs[0].annotate(f'p-value = {p_value_s:.2f}', xy=(.6, .8), xycoords=axs[0].transAxes)

axs[1].hist([xd_1, xd_2], n_bins, histtype='bar')
axs[1].set_title('Exponential different samples')
axs[1].annotate(f'p-value = {p_value_d:.2f}', xy=(1.8, .8), xycoords=axs[0].transAxes)

fig.tight_layout()
plt.show() Now, we do the same calculations as with Bernoulli data for evaluating the first and second-type errors. The results are the same—the criteria are very powerful. import numpy as np
from scipy import stats

import matplotlib.pyplot as plt

n_attempts = 100000
N = 10000

# False Negative calculation
p_value_eq_list = []
beta = 1

i = 1
while i <= n_attempts:
  x_1 = np.random.exponential(beta, N)
  x_2 = np.random.exponential(beta, N)
  _, p_value = stats.ttest_ind(x_1, x_2, equal_var=False)
  p_value_eq_list.append(p_value)
  i += 1

False_Negative = sum([i < 0.05 for i in p_value_eq_list])/len(p_value_eq_list)

# False Positive calculation
p_value_noneq_list = []
beta_1 = 1
beta_2 = 1.05

i = 1
while i <= n_attempts:
  x_1 = np.random.exponential(beta_1, N)
  x_2 = np.random.exponential(beta_2, N)
  _, p_value = stats.ttest_ind(x_1, x_2, equal_var=False)
  p_value_noneq_list.append(p_value)
  i += 1

False_Positive = 1 - sum([i < 0.05 for i in p_value_noneq_list])/len(p_value_noneq_list) Conclusion Now we know the answer to the question about a prohibition on applying a t-test for abnormal distribution—yes, we can apply it to most distributions! Today, nearly all companies in the IT industry use AB testing to evaluate the value of a feature. They test all changes in applications and websites, allowing someone to determine which feature is the winner every day. As a result, numerous discussions revolve around experiments and AB tests, with a common misconception being the prohibition of applying t-tests for abnormal distributions. Let's delve into this topic and find the correct answer! Problem Formulation Let’s look at a familiar example Imagine a common experiment with different paywalls that suggest a purchase to a customer. We have two variants that show two different and independent groups of users. The decision will be made by a conversion rate to purchase. Every user was assigned to a control or test group randomly, so both samples are independent. Also, every user is assigned 1 or 0 meaning they either made a purchase or not. Thus, our samples have a bi-nominal distribution. In this case, the conversion rate to purchase equals the sample mean. So, we will compare p1n1 and p2n2. Is it possible to use a t-test here or not? Is it possible to use a t-test here or not? Let’s generalize the problem We’re still considering the two independent distributed samples X1 and X2. We know that they have abnormal distributions with finite expected values μ1 and μ2 and finite variances σ1 and σ2. Now, we want to check a hypothesis about the equality of means: Theoretical Proof Let’s consider the Welch’s t-test in a case of different variances. When the variances of both samples are the same, then the Student’s and Welch’s t-tests have similar results. Thus, let’s consider a statistic: Can it be compared with critical values of Student’s distribution? By the central limit theorem, if n1,2 → ∞ Therefore, Also, per Slutsky’s theorem, this convergence in probability is the convergence in distribution. Thus, the constructed statistic t ∼ N(μ1 −μ2,1). Under the null hypothesis t ∼ N(0,1), which is asymptotically equivalent to the Student’s distribution. Empirical Proof Finally, we proved that a t-test can estimate the experiment with two variants of paywall. Also, it is worth mentioning that only two conditions were used for that example: n→∞ and finite of moments for distribution of random value. n→∞ and finite of moments for distribution of random value. Let’s try to simulate a pair of distributions to check the prohibition to apply a t-test for them! Bernoulli data Bernoulli data is a widely used case for the experimentation process. A metric like CTR, any type of conversion rate, retention rate, etc., is often chosen as a target metric in an experiment. First, let’s simulate two samples of the Bernoulli distribution with the same parameters. After that, we started increasing the parameter for the second sample. For all samples, we calculated the p-value using Student’s t-test. We see that the p-value drops to zero almost immediately. import numpy as np
from scipy import stats

import matplotlib.pyplot as plt

p_diff_list = []
p_value_list = []

N = 10000
p_1 = 0.01
p_2 = p_1

while p_2 <= 1:
  p_diff_list.append(p_2-p_1)
  x_1 = np.random.binomial(1, p_1, N)
  x_2 = np.random.binomial(1, p_2, N)
  _, p_value = stats.ttest_ind(x_1, x_2, equal_var=False)
  p_value_list.append(p_value)
  p_2 += 0.01

fig, ax = plt.subplots(figsize=(10,4))

ax.plot(p_diff_list, p_value_list)
ax.set(xlabel='p_2 - p_1', ylabel='p-value',
       title='p-value changes for different Bernoulli distributions')

fig.tight_layout()
plt.show() import numpy as np
from scipy import stats

import matplotlib.pyplot as plt

p_diff_list = []
p_value_list = []

N = 10000
p_1 = 0.01
p_2 = p_1

while p_2 <= 1:
  p_diff_list.append(p_2-p_1)
  x_1 = np.random.binomial(1, p_1, N)
  x_2 = np.random.binomial(1, p_2, N)
  _, p_value = stats.ttest_ind(x_1, x_2, equal_var=False)
  p_value_list.append(p_value)
  p_2 += 0.01

fig, ax = plt.subplots(figsize=(10,4))

ax.plot(p_diff_list, p_value_list)
ax.set(xlabel='p_2 - p_1', ylabel='p-value',
       title='p-value changes for different Bernoulli distributions')

fig.tight_layout()
plt.show() Now, let’s simulate 100,000 times two samples with the same parameters and 100,000 times two samples with different parameters. After that, we can check the first and second-type errors. Indeed, the t-test is a powerful criterion for metrics with parameters like in our simulations—the first type error is around 5%, and the power is around 90%! import numpy as np
from scipy import stats

import matplotlib.pyplot as plt

n_attempts = 100000
N = 10000

# False Negative calculation
p_value_eq_list = []
p = 0.01

i = 1
while i <= n_attempts:
  x_1 = np.random.binomial(1, p, N)
  x_2 = np.random.binomial(1, p, N)
  _, p_value = stats.ttest_ind(x_1, x_2, equal_var=False)
  p_value_eq_list.append(p_value)
  i += 1

False_Negative = sum([i < 0.05 for i in p_value_eq_list])/len(p_value_eq_list)

# False Positive calculation
p_value_noneq_list = []
p_1 = 0.01
p_2 = 0.015

i = 1
while i <= n_attempts:
  x_1 = np.random.binomial(1, p_1, N)
  x_2 = np.random.binomial(1, p_2, N)
  _, p_value = stats.ttest_ind(x_1, x_2, equal_var=False)
  p_value_noneq_list.append(p_value)
  i += 1

False_Positive = 1 - sum([i < 0.05 for i in p_value_noneq_list])/len(p_value_noneq_list) import numpy as np
from scipy import stats

import matplotlib.pyplot as plt

n_attempts = 100000
N = 10000

# False Negative calculation
p_value_eq_list = []
p = 0.01

i = 1
while i <= n_attempts:
  x_1 = np.random.binomial(1, p, N)
  x_2 = np.random.binomial(1, p, N)
  _, p_value = stats.ttest_ind(x_1, x_2, equal_var=False)
  p_value_eq_list.append(p_value)
  i += 1

False_Negative = sum([i < 0.05 for i in p_value_eq_list])/len(p_value_eq_list)

# False Positive calculation
p_value_noneq_list = []
p_1 = 0.01
p_2 = 0.015

i = 1
while i <= n_attempts:
  x_1 = np.random.binomial(1, p_1, N)
  x_2 = np.random.binomial(1, p_2, N)
  _, p_value = stats.ttest_ind(x_1, x_2, equal_var=False)
  p_value_noneq_list.append(p_value)
  i += 1

False_Positive = 1 - sum([i < 0.05 for i in p_value_noneq_list])/len(p_value_noneq_list) Exponential data Now, let’s consider the exponential data prevalent in the real world, too. Two cases of exponential sample comparisons are shown below. In the first case, samples with the same parameters are considered. The p-value corresponds to reality; it says that samples are similar. In the right part, there is a case with two samples with different parameters and a p-value lower than 0.05. import numpy as np
from scipy import stats

import matplotlib.pyplot as plt

# Generate similar samples
N = 10000
beta = 3

xs_1 = np.random.exponential(beta, N)
xs_2 = np.random.exponential(beta, N)

_, p_value_s = stats.ttest_ind(xs_1, xs_2, equal_var=False)

# Generate different samples
N = 10000
beta_1 = 3
beta_2 = 10

xd_1 = np.random.exponential(beta_1, N)
xd_2 = np.random.exponential(beta_2, N)

_, p_value_d = stats.ttest_ind(xd_1, xd_2, equal_var=False)

n_bins = 100
fig, axs = plt.subplots(1, 2, figsize=(12,4))

axs[0].hist([xs_1, xs_2], n_bins, histtype='bar')
axs[0].set_title('Exponential similar samples')
axs[0].annotate(f'p-value = {p_value_s:.2f}', xy=(.6, .8), xycoords=axs[0].transAxes)

axs[1].hist([xd_1, xd_2], n_bins, histtype='bar')
axs[1].set_title('Exponential different samples')
axs[1].annotate(f'p-value = {p_value_d:.2f}', xy=(1.8, .8), xycoords=axs[0].transAxes)

fig.tight_layout()
plt.show() import numpy as np
from scipy import stats

import matplotlib.pyplot as plt

# Generate similar samples
N = 10000
beta = 3

xs_1 = np.random.exponential(beta, N)
xs_2 = np.random.exponential(beta, N)

_, p_value_s = stats.ttest_ind(xs_1, xs_2, equal_var=False)

# Generate different samples
N = 10000
beta_1 = 3
beta_2 = 10

xd_1 = np.random.exponential(beta_1, N)
xd_2 = np.random.exponential(beta_2, N)

_, p_value_d = stats.ttest_ind(xd_1, xd_2, equal_var=False)

n_bins = 100
fig, axs = plt.subplots(1, 2, figsize=(12,4))

axs[0].hist([xs_1, xs_2], n_bins, histtype='bar')
axs[0].set_title('Exponential similar samples')
axs[0].annotate(f'p-value = {p_value_s:.2f}', xy=(.6, .8), xycoords=axs[0].transAxes)

axs[1].hist([xd_1, xd_2], n_bins, histtype='bar')
axs[1].set_title('Exponential different samples')
axs[1].annotate(f'p-value = {p_value_d:.2f}', xy=(1.8, .8), xycoords=axs[0].transAxes)

fig.tight_layout()
plt.show() Now, we do the same calculations as with Bernoulli data for evaluating the first and second-type errors. The results are the same—the criteria are very powerful. import numpy as np
from scipy import stats

import matplotlib.pyplot as plt

n_attempts = 100000
N = 10000

# False Negative calculation
p_value_eq_list = []
beta = 1

i = 1
while i <= n_attempts:
  x_1 = np.random.exponential(beta, N)
  x_2 = np.random.exponential(beta, N)
  _, p_value = stats.ttest_ind(x_1, x_2, equal_var=False)
  p_value_eq_list.append(p_value)
  i += 1

False_Negative = sum([i < 0.05 for i in p_value_eq_list])/len(p_value_eq_list)

# False Positive calculation
p_value_noneq_list = []
beta_1 = 1
beta_2 = 1.05

i = 1
while i <= n_attempts:
  x_1 = np.random.exponential(beta_1, N)
  x_2 = np.random.exponential(beta_2, N)
  _, p_value = stats.ttest_ind(x_1, x_2, equal_var=False)
  p_value_noneq_list.append(p_value)
  i += 1

False_Positive = 1 - sum([i < 0.05 for i in p_value_noneq_list])/len(p_value_noneq_list) import numpy as np
from scipy import stats

import matplotlib.pyplot as plt

n_attempts = 100000
N = 10000

# False Negative calculation
p_value_eq_list = []
beta = 1

i = 1
while i <= n_attempts:
  x_1 = np.random.exponential(beta, N)
  x_2 = np.random.exponential(beta, N)
  _, p_value = stats.ttest_ind(x_1, x_2, equal_var=False)
  p_value_eq_list.append(p_value)
  i += 1

False_Negative = sum([i < 0.05 for i in p_value_eq_list])/len(p_value_eq_list)

# False Positive calculation
p_value_noneq_list = []
beta_1 = 1
beta_2 = 1.05

i = 1
while i <= n_attempts:
  x_1 = np.random.exponential(beta_1, N)
  x_2 = np.random.exponential(beta_2, N)
  _, p_value = stats.ttest_ind(x_1, x_2, equal_var=False)
  p_value_noneq_list.append(p_value)
  i += 1

False_Positive = 1 - sum([i < 0.05 for i in p_value_noneq_list])/len(p_value_noneq_list) Conclusion Now we know the answer to the question about a prohibition on applying a t-test for abnormal distribution—yes, we can apply it to most distributions!

QONVERSION

qonversion.io

11-50 employees

Since 2019

COMPANY RANKING

QONVERSION

EVERGREEN INDEX #8820

Representing this company? Click here to claim and customize this page!

Qonversion WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

QONVERSION

qonversion.io

11-50 employees

Since 2019

COMPANY RANKING

QONVERSION

EVERGREEN INDEX #8820

RELATED COMPANIESRANK

STORIES ABOUT QONVERSION

Representing this company? Click here to claim and customize this page!

Qonversion WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!