AI seems to help almost every job in some way.
As SDET / QA, what if your test cases could be auto-generated by AI and automatically verified using Pytest ?
Hence, I recently tried integrating OpenAI with Pytest to automate API test generation. Here’s what I learned.
What You Need to Know and Prepare
Before diving in, here some tools and resources you’ll need:
-
OpenAI Python Library
https://github.com/openai/openai-python -
Pytest (Python testing framework)
https://github.com/pytest-dev/pytest -
API Under Test — FakeStoreAPI (Cart endpoint)
Setup Overview
Let OpenAI Build and Verify Basic Test Cases
We’ll start by asking OpenAI to generate simple API test cases using structured prompts. Then, we use Pytest to run and validate those cases against the real API.
- Install and Import Required Packages
$ pip install openai
$ pip install pytest
https://gist.github.com/shyinlim/76bc4b8a37df414cccf29ea227ef6ab4?embedable=true
2. Define the Test Case Generator Function
We’ll create a function to send a prompt to OpenAI, instructing it to behave like a Senior SDET / QA and return structured API test cases in JSON format.
https://gist.github.com/shyinlim/7190ab613a342ce77c94bc906a55e258?embedable=true
3. Let OpenAI Build and Verify Basic Test Cases
Before defining the test function, we need to prepare a prompt that describes the API’s method, endpoint, and a sample of the request and response structure. This prompt will guide OpenAI to generate the test cases.
https://gist.github.com/shyinlim/1f6fe4fa0cc99edcff9e82504164ebfd?embedable=true
Once the prompt is ready, we define a test function that sends this information to OpenAI, retrieves the generated test cases, and runs them using Pytest.
https://gist.github.com/shyinlim/58334767807f41f76d5fbf73b4ac1f60?embedable=true
Okay, here’s a sample test cases of what OpenAI might return:
- create_cart_with_valid_single_product
- create_cart_with_multiple_products
- create_cart_with_minimum_valid_values
Expand with Human-Designed Cases
While AI can quickly generate basic test cases. However, edge cases, business-specific logic, or tricky validation rules often still require human insight.
Use AI as a starting point, then build on it with your domain knowledge.
https://gist.github.com/shyinlim/709a54c3b9491a8f2f0b6aa171145211?embedable=true
Knowing When to Let AI Handle It
AI-generated test functions work best when:
- API specs are clear and well-defined, including methods, parameters, and expected responses
- Field validations are needed (data types, required fields, ranges).
- Request / response flows are standard without complex business logic.
- Need to quickly bootstrap test coverage, especially for repetitive or CRUD endpoints
You’ll still want to write tests manually when:
- Business logic is complex or conditional, which AI may not fully understand.
- Heavily DB-dependent tests, they often require manual setup and domain knowledge to ensure correctness.
- Mocking or stubbing other services is needed, especially async or dependent ones.
- Test results depend on side effects beyond response values, like logs or DB updates.
- Security testing such as auth, permission, or injection checks is involved.
Final Thoughts
This experiment shows a great way to boost testing efficiency by combining OpenAI and Pytest.
It’s not about replacing SDET / QA Engineers, but about helping us get started faster, cover more ground, and focus on the tricky stuff.
The key takeaway ?
The magic isn’t just in the AI, it’s in the prompt.
Good prompts don’t just show up by magic, they come from your mix of :
- Testing experience
- Testing techniques
- Testing mindset
- Communication skills
- Product understanding
Here’s my GitHub repo showing a straightforward example:
Test smart, not hard. And, Happy Testing :)
Sh-Yin Lim (