From Routine to Rewarding: Integrating ChatGPT Into Product Testing Workflow

If a product is complex and releases are substantial, there are multiple ways to ensure quality. One key tool I use in our development process is team testing: stepping into the customer’s shoes, traversing various user journeys, and ensuring all key interactions work as intended. This approach offers numerous benefits, from engaging the entire team in understanding the product to correcting flaws in the original business logic.

Despite its usefulness, team testing sessions tend to be tedious and generally disliked: building a product is far more enjoyable than repeatedly clicking through it.

In this post, I will describe how I made team testing more engaging and somewhat enjoyable by incorporating ChatGPT into the process, transforming it into a competitive activity.

How We Conduct Team Testing

The process was straightforward:

1. Deploy a feature to a testing environment.
2. Document what needs to be checked: main user journeys and definitions of success.
3. Gather on a call or in a room, use the product, discuss, and exchange feedback.
4. Organize and prioritize feedback, then act accordingly.

As a result, the team has usually compiled a list of issues to address.

Introducing ChatGPT

With the rise of ChatGPT in mid-2023, I decided to integrate it into our testing sessions. Initially, I used it to analyze and count issue reports in Slack—since Slack lacks convenient export functionality.

Here’s the prompt I tried:

The text below is the log of slack channel.

It contains bugs reported by people, each bug report starts with a first name, then timestamp, then the message itself (normally 1 or 2 lines).

Strings containing ".png", IMG" or ".mov"  could be ignored.

Read the text, count the number of reports per person and rank them from highest

ChatGPT struggled at first, glitching in about 60% of cases with verbose error messages.

I realized I needed a different approach.

Switching to a Script

Since ChatGPT ran a Python interpreter for our task, I decided to directly utilize the Python code it wrote.

I adjusted the script to account for the way Slack handles attached files as separate messages:

# Initialize counters for each name based on the list provided
# The names should match the displayed names in Slack
name_counts = {
    "John": 0,
    "Jane": 0,
    "Rosy": 0,
}

# Open the file and count the occurrences of each name
with open("testing_session_log.txt", "r") as file:
    lines = file.readlines()

# Filtering out garbage lines and counting names
for line in lines:
    # Ignore garbage lines
    if any(garbage in line for garbage in [".png", ".IMG", ".mov"]):
        continue
    for name in name_counts.keys():
        if name in line:
            name_counts[name] += 1

print(name_counts)

Returning to ChatGPT for Creativity and Recognition

Implementing the script to generate a leaderboard immediately after the session introduced a sense of competition.

I saw the engagement growing, and people reporting more bugs!

So, I continued to use ChatGPT for crafting personalized praise based on the team’s reported issues, acknowledging their contributions:

Using the attached testing session log, praise everyone for their impact and highligh one report of a choice for each person.

Participants names:
John
Jane
Rosy

Wrap the praise in a form of a short, sincere and inspirational message, no more than 4 sentences long.

I found it deeply satisfying to read the praise from a machine:

John: Your insight on UI details like the collapsed header buttons and gallery navigation enhances our project's usability.
Your contributions are crucial for a seamless user experience.

Jane: Your keen observations, especially on navigation issues, enhance our platform's intuitiveness.
Your ability to catch subtle bugs is invaluable.

Rosy:Your focus on mobile usability, from download behaviors to icon alignment, significantly polishes our mobile interface.
Your efforts make a substantial difference.

Conclusion

What I’ve implemented isn’t rocket science, but in the pre-LLM era, this would have taken considerably longer to achieve, if at all. While the future of AI-driven products is still shaping up, it’s clear that product teams can benefit greatly by integrating such technologies to simplify tasks, boost positive reinforcement, and free up time for critical projects.