Utilizing testing in production should not be seen as a replacement for testing in staging or other preproduction environments. Instead, it should be viewed as a complementary technique within the quality assurance process. Important notice: In the fast-paced software development world, teams continually seek innovative ways to optimize their testing processes. One such approach gaining popularity is "Testing in Production" (TiP). This method has recently gained popularity due to its potential advantages in identifying elusive bugs and simulating actual user experiences. Let's start with a simple question: Why is QA in Production Necessary Despite the Presence of Preproduction Environments? : Despite the utmost efforts to replicate production environments, preproduction environments can never be a perfect match. Minor variations in configurations, data, or external integrations can exist, leading to unexpected behaviors when the software goes live. By conducting tests in the actual production environment, developers gain real-world insights and identify potential issues that might not have surfaced during preproduction testing. Closing the Gap : Preproduction environments often utilize synthetic or sanitized data, which may not accurately represent the variety and volume of data found in the live environment. Testing in production allows QA to work with authentic data, providing a more realistic assessment of the application's performance, scalability, and reliability. This ensures that the software can handle the actual data volume and usage patterns it will encounter in production. Real Data, Real Results : Production environments can be significantly more complex than preproduction environments due to increased user traffic, third-party integrations, and live data feeds. These complexities can introduce unique challenges and vulnerabilities that may remain hidden until the software is deployed to the live environment. By conducting tests in production, QA can validate the software's behavior under these intricate conditions and make necessary adjustments. Handling Complexity : Monitoring is crucial in various fields. In technology and software, monitoring refers to observing and tracking multiple metrics and performance indicators of systems, applications, networks, or services in real time or over time. The primary goal of monitoring is to ensure that everything is functioning as expected, identify potential issues, and take proactive measures to maintain optimal performance and reliability. The Lack of Monitoring in Preproduction Examples of Proper Monitoring Include: Website Performance Monitoring. Critical business scenarios (amount of registration, amount of any transactions, placing new orders, and so on) Server Infrastructure Monitoring. Application Performance Monitoring (APM). Network Monitoring. Security Event Monitoring. Database Monitoring. In some cases, resource limitations might be in establishing and upkeep a monitoring system in the preproduction environment. Nevertheless, the primary challenge lies in the need for real traffic and usage in the live production environment. Beneficial QA practices in production testing! Implement a gradual rollout strategy to limit the impact of changes. Start with a small percentage of users or traffic, and gradually increase as you gain confidence in the changes. Gradual Rollouts: Use feature flags to enable or disable specific features in the production environment. It allows you to turn off a feature if any issues arise quickly. Feature Flags: Implement robust monitoring and observability tools to monitor the system during testing closely. Real-time data helps quickly identify and address any anomalies or performance issues. Monitoring and Observability: Automate the deployment process to ensure consistency and reduce the chances of human error. Automation: Always have a well-defined rollback plan in case issues arise. Being prepared for unexpected scenarios is crucial for maintaining system stability. Fallback Plan: For instance, when deploying a new feature, once it is deployed to the production environment, the QA team should verify that the feature has been successfully released. To accomplish this, conducting a happy path scenario would be enough. Checking Completed Tasks to Guarantee Smooth Deployment and Proper Functioning in the Production Environment: Smoke testing is a type of preliminary testing aimed at determining whether the basic functionalities of an application work as expected after a new deployment. In a preproduction environment, smoke tests are traditionally performed to ensure that significant functionalities are not broken before moving forward with more comprehensive testing. However, smoke testing in production takes this approach one step further by validating the core functionalities directly in the live environment after the deployment has been rolled out. Smoke Tests in Production: When issues arise, logs provide a detailed record of what occurred, allowing developers and QA to trace the sequence of events leading to the problem. By analyzing the log data, developers can pinpoint the root cause of errors, leading to faster and more effective problem resolution. The QA team must verify that the new features are accompanied by appropriate logging. If they still need to, they should promptly notify the developers about the need for necessary enhancements. Effective Logging: Finally, I would like to share with you the outcomes and the practices that have been implemented in my company. Here is the list of them: - We have recently started applying this approach, which is why it is too early to discuss the outcomes. Feature Flags - QA plays an active role in the monitoring process. While writing System Requirement Specifications (SRS), QA examines which technical metrics, like method execution duration and event occurrence, could benefit the feature. Moreover, during the Business Requirement Specification (BRS) phase, just before the feature implementation, we keep an eye on business metrics, such as the number of users adopting the new feature. We promptly alert the feature owners if we do not find such metrics in the feature documentation during development. Monitoring and Observability - Our company has implemented a completely automated deployment process to minimize potential human-related errors. Automation CI/CD - Before rolling out any updates, the developer must assess the feasibility of a seamless rollback to the previous version in the event of widespread issues occurring in the production environment. Following each release, the assigned developer and the DevOps team diligently monitor logs and metrics for any irregularities in our established process. If any significant anomalies are identified, we immediately initiate the rollback procedure. Fallback Plan - Following the release, developers keep an eye on metrics and logs. At the same time, QA ensures the successful deployment of new features by conducting smoke tests on the implemented functionality. Checking Completed Tasks to Guarantee Smooth Deployment and Proper Functioning in the Production Environment - We categorize test cases into smoke, regression, full set, and feature-specific sets. After the production release, we automatically run production smoke tests, covering the system's most critical functionality. If a test case still needs to be automated, the tester responsible for the release performs it manually. Smoke Tests in Production Over the past four months, smoke production tests have uncovered three critical bugs. Among them, two were linked to discrepancies in settings between preproduction and production environments, particularly related to external providers. The third bug was attributed to the production environment having a different number of databases. While it's impossible to catch all errors in preproduction, efforts should still be made to minimize them. Adhering to the correct practices can lead to faster bug detection than waiting for users to encounter them, potentially safeguarding the company's reputation. Happy and productive testing!

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Why Testing in Production Will Transform Your Success Game

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Boosting Productivity: A Guide to Implementing a New QA Role for Faster Releases

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

The Noonification: The Four Horsemen of Flaky Tests (10/24/2023)

11 Best Automation Testing Tools to Try in 2021

12 Common Software Testing Misconceptions Debunked

Types of Crypto Wallets and How to Choose Best Crypto wallet

Boosting Productivity: A Guide to Implementing a New QA Role for Faster Releases

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

The Noonification: The Four Horsemen of Flaky Tests (10/24/2023)

11 Best Automation Testing Tools to Try in 2021

12 Common Software Testing Misconceptions Debunked

Types of Crypto Wallets and How to Choose Best Crypto wallet

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps