If a product is complex and releases are substantial, there are multiple ways to ensure quality. One key tool I use in our development process is team testing: stepping into the customer’s shoes, traversing various user journeys, and ensuring all key interactions work as intended. This approach offers numerous benefits, from engaging the entire team in understanding the product to correcting flaws in the original business logic. Despite its usefulness, team testing sessions tend to be tedious and generally disliked: building a product is far more enjoyable than repeatedly clicking through it. In this post, I will describe how I made team testing more engaging and somewhat enjoyable by incorporating ChatGPT into the process, transforming it into a competitive activity. How We Conduct Team Testing The process was straightforward: Deploy a feature to a testing environment.
Document what needs to be checked: main user journeys and definitions of success.
Gather on a call or in a room, use the product, discuss, and exchange feedback.
Organize and prioritize feedback, then act accordingly. As a result, the team has usually compiled a list of issues to address. Introducing ChatGPT With the rise of ChatGPT in mid-2023, I decided to integrate it into our testing sessions. Initially, I used it to analyze and count issue reports in Slack—since Slack lacks convenient export functionality. Here’s the prompt I tried: The text below is the log of slack channel.

It contains bugs reported by people, each bug report starts with a first name, then timestamp, then the message itself (normally 1 or 2 lines).

Strings containing ".png", IMG" or ".mov"  could be ignored.

Read the text, count the number of reports per person and rank them from highest ChatGPT struggled at first, glitching in about 60% of cases with verbose error messages. I realized I needed a different approach. Switching to a Script Since ChatGPT ran a Python interpreter for our task, I decided to directly utilize the Python code it wrote. I adjusted the script to account for the way Slack handles attached files as separate messages: # Initialize counters for each name based on the list provided
# The names should match the displayed names in Slack
name_counts = {
    "John": 0,
    "Jane": 0,
    "Rosy": 0,
}

# Open the file and count the occurrences of each name
with open("testing_session_log.txt", "r") as file:
    lines = file.readlines()

# Filtering out garbage lines and counting names
for line in lines:
    # Ignore garbage lines
    if any(garbage in line for garbage in [".png", ".IMG", ".mov"]):
        continue
    for name in name_counts.keys():
        if name in line:
            name_counts[name] += 1

print(name_counts) Returning to ChatGPT for Creativity and Recognition Implementing the script to generate a leaderboard immediately after the session introduced a sense of competition. I saw the engagement growing, and people reporting more bugs! So, I continued to use ChatGPT for crafting personalized praise based on the team’s reported issues, acknowledging their contributions: Using the attached testing session log, praise everyone for their impact and highligh one report of a choice for each person.

Participants names:
John
Jane
Rosy

Wrap the praise in a form of a short, sincere and inspirational message, no more than 4 sentences long. I found it deeply satisfying to read the praise from a machine: John: Your insight on UI details like the collapsed header buttons and gallery navigation enhances our project's usability.
Your contributions are crucial for a seamless user experience.

Jane: Your keen observations, especially on navigation issues, enhance our platform's intuitiveness.
Your ability to catch subtle bugs is invaluable.

Rosy:Your focus on mobile usability, from download behaviors to icon alignment, significantly polishes our mobile interface.
Your efforts make a substantial difference. Conclusion What I’ve implemented isn’t rocket science, but in the pre-LLM era, this would have taken considerably longer to achieve, if at all. While the future of AI-driven products is still shaping up, it’s clear that product teams can benefit greatly by integrating such technologies to simplify tasks, boost positive reinforcement, and free up time for critical projects. Image credit: Midjourney, prompt “team of engineers coding late at night, Hyper-realistic textures, Precise details --version 6.0 --weird 1000” If a product is complex and releases are substantial, there are multiple ways to ensure quality. One key tool I use in our development process is team testing: stepping into the customer’s shoes, traversing various user journeys, and ensuring all key interactions work as intended. This approach offers numerous benefits, from engaging the entire team in understanding the product to correcting flaws in the original business logic. Despite its usefulness, team testing sessions tend to be tedious and generally disliked: building a product is far more enjoyable than repeatedly clicking through it. In this post, I will describe how I made team testing more engaging and somewhat enjoyable by incorporating ChatGPT into the process, transforming it into a competitive activity. How We Conduct Team Testing The process was straightforward: Deploy a feature to a testing environment. Document what needs to be checked: main user journeys and definitions of success. Gather on a call or in a room, use the product, discuss, and exchange feedback. Organize and prioritize feedback, then act accordingly. Deploy a feature to a testing environment. Document what needs to be checked: main user journeys and definitions of success. Gather on a call or in a room, use the product, discuss, and exchange feedback. Organize and prioritize feedback, then act accordingly. As a result, the team has usually compiled a list of issues to address. Introducing ChatGPT With the rise of ChatGPT in mid-2023, I decided to integrate it into our testing sessions. Initially, I used it to analyze and count issue reports in Slack—since Slack lacks convenient export functionality. Here’s the prompt I tried: The text below is the log of slack channel.

It contains bugs reported by people, each bug report starts with a first name, then timestamp, then the message itself (normally 1 or 2 lines).

Strings containing ".png", IMG" or ".mov"  could be ignored.

Read the text, count the number of reports per person and rank them from highest The text below is the log of slack channel.

It contains bugs reported by people, each bug report starts with a first name, then timestamp, then the message itself (normally 1 or 2 lines).

Strings containing ".png", IMG" or ".mov"  could be ignored.

Read the text, count the number of reports per person and rank them from highest ChatGPT struggled at first, glitching in about 60% of cases with verbose error messages. I realized I needed a different approach. Switching to a Script Since ChatGPT ran a Python interpreter for our task, I decided to directly utilize the Python code it wrote. Python interpreter I adjusted the script to account for the way Slack handles attached files as separate messages: # Initialize counters for each name based on the list provided
# The names should match the displayed names in Slack
name_counts = {
    "John": 0,
    "Jane": 0,
    "Rosy": 0,
}

# Open the file and count the occurrences of each name
with open("testing_session_log.txt", "r") as file:
    lines = file.readlines()

# Filtering out garbage lines and counting names
for line in lines:
    # Ignore garbage lines
    if any(garbage in line for garbage in [".png", ".IMG", ".mov"]):
        continue
    for name in name_counts.keys():
        if name in line:
            name_counts[name] += 1

print(name_counts) # Initialize counters for each name based on the list provided
# The names should match the displayed names in Slack
name_counts = {
    "John": 0,
    "Jane": 0,
    "Rosy": 0,
}

# Open the file and count the occurrences of each name
with open("testing_session_log.txt", "r") as file:
    lines = file.readlines()

# Filtering out garbage lines and counting names
for line in lines:
    # Ignore garbage lines
    if any(garbage in line for garbage in [".png", ".IMG", ".mov"]):
        continue
    for name in name_counts.keys():
        if name in line:
            name_counts[name] += 1

print(name_counts) Returning to ChatGPT for Creativity and Recognition Implementing the script to generate a leaderboard immediately after the session introduced a sense of competition. I saw the engagement growing, and people reporting more bugs! So, I continued to use ChatGPT for crafting personalized praise based on the team’s reported issues, acknowledging their contributions: Using the attached testing session log, praise everyone for their impact and highligh one report of a choice for each person.

Participants names:
John
Jane
Rosy

Wrap the praise in a form of a short, sincere and inspirational message, no more than 4 sentences long. Using the attached testing session log, praise everyone for their impact and highligh one report of a choice for each person.

Participants names:
John
Jane
Rosy

Wrap the praise in a form of a short, sincere and inspirational message, no more than 4 sentences long. I found it deeply satisfying to read the praise from a machine: John: Your insight on UI details like the collapsed header buttons and gallery navigation enhances our project's usability.
Your contributions are crucial for a seamless user experience.

Jane: Your keen observations, especially on navigation issues, enhance our platform's intuitiveness.
Your ability to catch subtle bugs is invaluable.

Rosy:Your focus on mobile usability, from download behaviors to icon alignment, significantly polishes our mobile interface.
Your efforts make a substantial difference. John: Your insight on UI details like the collapsed header buttons and gallery navigation enhances our project's usability.
Your contributions are crucial for a seamless user experience.

Jane: Your keen observations, especially on navigation issues, enhance our platform's intuitiveness.
Your ability to catch subtle bugs is invaluable.

Rosy:Your focus on mobile usability, from download behaviors to icon alignment, significantly polishes our mobile interface.
Your efforts make a substantial difference. Conclusion What I’ve implemented isn’t rocket science, but in the pre-LLM era, this would have taken considerably longer to achieve, if at all. While the future of AI-driven products is still shaping up, it’s clear that product teams can benefit greatly by integrating such technologies to simplify tasks, boost positive reinforcement, and free up time for critical projects. Image credit: Midjourney, prompt “team of engineers coding late at night, Hyper-realistic textures, Precise details --version 6.0 --weird 1000” Image credit: Midjourney, prompt “team of engineers coding late at night, Hyper-realistic textures, Precise details --version 6.0 --weird 1000”

Mind Mapping “How to Do Great Work”: My Take on Paul Graham's Essay (Part One)

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

From Routine to Rewarding: Integrating ChatGPT Into Product Testing Workflow

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Mind Mapping “How to Do Great Work”: My Take on Paul Graham's Essay (Part One)

100 Days of AI Day 4: Maximizing Productivity & Creativity with ChatGPT

10 Tips to Take Your ChatGPT Prompts to the Next Level

10 Tips to Get the Most out of ChatGPT

100 Days of AI Day 2: Enhancing Prompt Engineering for ChatGPT

Mind Mapping “How to Do Great Work”: My Take on Paul Graham's Essay (Part One)

100 Days of AI Day 4: Maximizing Productivity & Creativity with ChatGPT

10 Tips to Take Your ChatGPT Prompts to the Next Level

10 Tips to Get the Most out of ChatGPT

100 Days of AI Day 2: Enhancing Prompt Engineering for ChatGPT

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps