Can SQA Engineers Rely on ChatGPT in Writing Test Cases?

I am a Software Quality Assurance (SQA) engineer with 12 years of experience who spent hundreds of hours writing, reviewing, and running test cases. I know that sometimes the process of creating a test design can be boring, but without a well-crafted test case design, the QA engineers will not be able to test the software efficiently.

They will have little or no visibility on testing coverage, which could lead to poor-quality software filled with defects and bugs. I know that AI-powered tools like ChatGPT can become a revolution and will change the way of work for SQA Engineers.

Here, in this experiment, I asked ChatGPT to write test cases for different applications and scenarios to see if SQA Engineers can fully outsource the test design job to ChatGPT or not. So, let's go!

1) Writing basic positive and negative test cases for Login functionality. The app is well known, and the functionality is common.

Sample Scenario: Write Positive and Negative test cases for login into Facebook.com

ChatGPT Response:

My conclusion: these test cases can definitely be used for basic functionality checks. They are written good, structure is perfect. The issue I see is that they are very general. So, for example, the expected result written by ChatGPT for a negative test case, when a password is not entered, says that “the user should see an error message indicating that the password field is required.”

For 50% of apps, this test case is valid, but sometimes, it is important to include more details into your test case, especially when wording is important or was changed recently, and the idea of the test case is to verify the change.

Also, another issue with ChatGPT test cases, which is probably even more important, is that if you run this test case manually on the facebook.com website, you will see that the expected result can be completely different.

Leaving an empty password field leads you to another screen where the avatar and name of the user with the entered email are shown. The message shown is, “The password you’ve entered is incorrect. Forgot Password?”

So, as you can see, the expected result is completely different from what ChatGPT has provided. So the test cases written even for basic scenarios can’t be used straightforwardly. ChatGPT will definitely save you time, but still, deep review and rework are required.

2) Writing test cases for more complex scenarios

Sample Scenario: Write test cases for the “Monthly payment calculator” functionality at zillow.com.

ChatGPT Response:

My conclusion: I found a couple of critical issues in ChatGPT test cases which I listed below:

• The expected result is too general for such complex functionality, no formulas are provided, and it is not clear how to verify that calculations are correct.

• The fields names in test cases are different than on the real website, Zillow.com

• The amount of fields in test cases is different than on the real website, Zillow.com

According to the issues I see in the ChatGPT test cases, I think using ChatGPT for complex functionality is not yet possible. You probably need to write more exact prompts to be able to receive more detailed test cases.

But for me, it looks like now, it is still easier to write test cases by yourself in the case of complex and unique functionality. Also, I am not sure how it is possible to write test cases using ChatGPT for the functionality that is not in production yet. Still, writing prompts, in this case, will take too much time.

Again, it will be easier to write test cases by yourself.

3) Writing test cases according to best practices and test design techniques.

This task is a common interview question. It is not related to any app; it is more about test design techniques knowledge. A test design technique is a standardized method to derive, from a specific test basis, test cases that realize a specific coverage.

Exhaustive testing for each set of test data is practically impossible due to time and budget constraints, especially when there is a large set of input combinations. SQA engineers should judiciously design test cases that will take care of all possible test data combinations.

Test case design techniques should select test cases that are most likely to find bugs, from a large number of available test cases. Boundary value analysis can be used to pick the appropriate test data and test cases.

Sample Scenario: Write test cases for a numeric field “amount” using the boundary values test design technique.

“Amount” field description: values less than 10 are rejected, values between 10 and 21 are accepted, and values greater than or equal to 22 are rejected.

ChatGPT Response:

My conclusion: According to software testing theory, the basic principle behind this technique is to choose input data values:

• Just below the minimum value
• Minimum value
• Just above the minimum value
• A Normal (nominal) value
• Just below the maximum value
• Maximum value
• Just above the maximum value

In general, the answer from ChatGPT looks good, all the required scenarios are written, and the expected output is correct. I would definitely recommend using ChatGPT for boundary value analysis technique test design.

Of course, in real life, such types of test cases are probably 1% from others. Because usually, it is more important to spend time by testing the main functionality rather than exhaustively verifying each field using test design techniques.

But this is a great opportunity to prepare yourself for the interview questions. Such types of questions are often asked in interviews.

Can SQA Rely on ChatGPT in Writing Test Cases?

As a general conclusion and answering shortly - no, still not. Can ChatGPT save some time for SQA Engineer: yes, of course.

ChatGPT can assist SQA in its work by providing insights and suggestions based on its vast knowledge base, helping to identify potential issues, and providing solutions to problems.

Additionally, ChatGPT can help answer common questions and provide guidance to those who are new to the field.

ChatGPT can be a useful tool to support the work of a SQA, but it cannot replace the skills and expertise of a human professional in this field.