Large language models and AI agents have already transformed many fields and are changing our lives in fundamental ways. In the testing domain, AI agents have a clear path for making immediate improvements in process and quality, and ultimately for producing reliable, performant, secure, and compliant software. Check out Demystifying Agentic Test Automation: What It Means for QA Teams.But it’s not obvious how to take advantage of these capabilities. While AI agents are not fully predictable, they can be managed reliably via robust control mechanisms. Let's see how. testing domain Demystifying Agentic Test Automation: What It Means for QA Teams What does it mean to manage AI agents in QA? There are several important aspects to managing AI agents, both in general and specifically in the testing domain. Configuration and guardrailsinvolve setting agent autonomy levels and boundaries, defining test objectives through prompts and constraints, and specifying which areas require human approval. These steps ensure that the system operates within controlled parameters while meeting goals.For example, you may allow your agentic AI system to write test code and even tracking and reporting systems, but not necessarily modify your production code.
Model selection and updates focus on deciding when to upgrade models versus maintaining stability. This includes testing model changes before rolling them out to production to avoid unexpected issues. If the quality of testing and coverage deteriorates when you upgrade models, it's a clear signal your AI agents are too finetuned to a specific model version.
Oversight and validation encompass quality gates and verification protocols, monitoring for flaky tests and low-value coverage, and managing the budget for execution costs. These practices help maintain test reliability and cost-effectiveness throughout the development lifecycle. Configuration and guardrailsinvolve setting agent autonomy levels and boundaries, defining test objectives through prompts and constraints, and specifying which areas require human approval. These steps ensure that the system operates within controlled parameters while meeting goals.For example, you may allow your agentic AI system to write test code and even tracking and reporting systems, but not necessarily modify your production code. Configuration and guardrails Model selection and updates focus on deciding when to upgrade models versus maintaining stability. This includes testing model changes before rolling them out to production to avoid unexpected issues. If the quality of testing and coverage deteriorates when you upgrade models, it's a clear signal your AI agents are too finetuned to a specific model version. Model selection and updates Oversight and validation encompass quality gates and verification protocols, monitoring for flaky tests and low-value coverage, and managing the budget for execution costs. These practices help maintain test reliability and cost-effectiveness throughout the development lifecycle. Oversight and validation How to manage AI agents in practice With those aspects in mind, let's see how to actually control your agents. Consider a scenario where your team is migrating to agentic AI testing. You want to configure your agent's autonomy carefully. Here is a concrete example of how to go about it. First, configure the agent system prompt. For example, you might specify: "You may generate API and UI test code for the /checkout and /payment flows. You may create test reports and update tracking dashboards. You must NOT modify production code or database schemas. All destructive operations (deleting tests, changing CI/CD pipelines) require human approval."
Next is tool integration. Connect your agent to your test management platform via APIs so it can read existing test cases and understand coverage gaps. This may be done through an MCP server like the Tosca MCP server or via custom tools you build.
Next, it’s time for a progressive rollout:Week 1 - Agent generates tests in "suggestion mode" only.Week 2-3 - Agent executes tests in an isolated staging environment.Week 4+ - Agent runs tests in pre-production with human review of failures.
Finally, set up quality gates in your CI/CD pipeline that require a minimum pass rate (for example, 80%) before agent-generated tests can block deployments. Monitor false positive rate weekly and tune prompts accordingly. First, configure the agent system prompt. For example, you might specify: "You may generate API and UI test code for the /checkout and /payment flows. You may create test reports and update tracking dashboards. You must NOT modify production code or database schemas. All destructive operations (deleting tests, changing CI/CD pipelines) require human approval." configure the agent system prompt Next is tool integration. Connect your agent to your test management platform via APIs so it can read existing test cases and understand coverage gaps. This may be done through an MCP server like the Tosca MCP server or via custom tools you build. tool integration Tosca MCP server custom tools you build. Next, it’s time for a progressive rollout:Week 1 - Agent generates tests in "suggestion mode" only.Week 2-3 - Agent executes tests in an isolated staging environment.Week 4+ - Agent runs tests in pre-production with human review of failures. progressive rollout Finally, set up quality gates in your CI/CD pipeline that require a minimum pass rate (for example, 80%) before agent-generated tests can block deployments. Monitor false positive rate weekly and tune prompts accordingly. set up quality gates How to Control AI Agents: prompts, tools, and feedback loops Learning to control your agents is critical. AI agents, contrary to common beliefs, don’t actually have intelligence or agency. Agents are best understood as a system prompt combined with state/memory and a selection of tools. All the intelligence lies within the large language model (LLM), which accepts the agent (system prompt plus tools) and the user prompt as context. It then decides which tools to invoke and how to process both prompts until generating a final answer. There are three ways to help control your AI agents: prompt engineering, tool integration, and performance monitoring/feedback loops. Prompt engineering serves as the primary interface with these agents. This process involves writing effective test objectives and acceptance criteria, building a library of proven prompts, and iterating based on the outputs generated by the agent. Well-crafted prompts guide the agent to deliver precise and useful results.
Tool integration can be achieved through the MCP server, connecting agents to source control, design documents, and CI/CD pipelines. This integration provides essential context that enhances the agent’s intelligence.  Additionally, emerging platforms like Applitools (for visual AI testing), Katalon Studio (for codeless test automation), and Selenium combined with AI extensions provide varying levels of autonomy and control for different organizational needs.
Performance monitoring and feedback loops track key metrics such as coverage, bug detection, false positives, and maintenance time. Real-time monitoring and alerting systems help determine when to reconfigure or retrain the agent for continued optimal performance. Prompt engineering serves as the primary interface with these agents. This process involves writing effective test objectives and acceptance criteria, building a library of proven prompts, and iterating based on the outputs generated by the agent. Well-crafted prompts guide the agent to deliver precise and useful results. Prompt engineering Tool integration can be achieved through the MCP server, connecting agents to source control, design documents, and CI/CD pipelines. This integration provides essential context that enhances the agent’s intelligence.  Additionally, emerging platforms like Applitools (for visual AI testing), Katalon Studio (for codeless test automation), and Selenium combined with AI extensions provide varying levels of autonomy and control for different organizational needs. Tool integration Performance monitoring and feedback loops track key metrics such as coverage, bug detection, false positives, and maintenance time. Real-time monitoring and alerting systems help determine when to reconfigure or retrain the agent for continued optimal performance. Performance monitoring and feedback loops Let's next consider the trajectory from traditional testing to agentic AI testing How to migrate from traditional testing to agentic AI testing Transitioning from traditional testing to agentic AI testing will be different for each organization as they start with their own unique combination of test automation, skill, and culture. Organizations with existing automation typically run agentic AI tests alongside their legacy scripts, adopting gradual migration strategies and increasing the system’s autonomy as confidence grows. Organizations with existing automation This approach allows for a smoother shift without disrupting existing processes. The primary value here is flexibility and the ability to adapt quickly to changes. Organizations that rely primarily on manual testing often start the process with low-risk regression suites to minimize risk. They focus on codifying tribal knowledge, transforming informal, human-centered test expertise into formalized, repeatable tests. Organizations that rely primarily on manual testing Additionally, for these organizations there is a cultural shift within QA teams, moving from maintaining scripts towards overseeing autonomous agents. But, in some ways it is easier to effect this change because the value is so much greater compared to already automated organizations. Practical integration typically involves hybrid testing methods that combine manual, AI-assisted, and fully agentic approaches. Platforms like Tricentis Tosca and qTest enable unified management across these methods. Tricentis Tosca qTest Example: Evolving a Traditional Test to an Agentic Approach Let's compare how a typical test scenario might evolve from traditional to agentic approaches. In a traditional Selenium test (using manual scripting) a QA engineer might write an explicit WebDriver code like this: driver.findElement(By.id("username")).sendKeys("test@example.com") driver.findElement(By.id("username")).sendKeys("test@example.com") But this test breaks when the UI changes (e.g., ID becomes class-based selector). Manual updates are required for every locator change. And with dynamic UI changes (very common when practicing A/B testing) the maintenance overhead and chance for errors is significant. However, with AI-Assisted Testing (with tools like Tricentis Tosca or mabl): mabl The QA engineer records user actions via visual test builder.
Self-healing locators adapt to minor UI changes automatically.
This approach still requires human intervention for test design and assertion logic. The QA engineer records user actions via visual test builder. Self-healing locators adapt to minor UI changes automatically. This approach still requires human intervention for test design and assertion logic. And with fully Agentic AI Testing: fully The QA engineer provides high-level intent: "Test the login flow with valid and invalid credentials, edge cases, and security scenarios."
The agent autonomously discovers UI elements, generates test cases, and creates assertions.
It self-adapts to UI changes and refactors test logic without human input.
It learns from failures and adjusts test strategies in real-time. The QA engineer provides high-level intent: "Test the login flow with valid and invalid credentials, edge cases, and security scenarios." The agent autonomously discovers UI elements, generates test cases, and creates assertions. It self-adapts to UI changes and refactors test logic without human input. It learns from failures and adjusts test strategies in real-time. The key insight is that traditional testing requires constant developer/QA time. Agentic AI shifts effort from execution and maintenance to strategic oversight and prompt engineering. This is all great. But it’s also good to remember that agentic AI testing is a new paradigm, and migration to the bleeding edge always comes with challenges. So next, let's review some of the common challenges and solutions. Common challenges in agentic AI testing A common challenge with implementing agentic testing is calibrating trust in the AI, which is best done through incremental rollout. calibrating trust Teams can introduce AI agents into the process gradually, validate their behavior on constrained scopes, and expand their responsibilities as confidence grows based on real outcomes and monitored performance.Another persistent challenge isdealing with flaky tests, which can undermine trust and inflate maintenance costs. dealing with flaky tests Effective strategies include isolating unstable scenarios, tagging and quarantining flaky tests, and using AI agents to surface patterns in failures so teams can harden those areas. At the same time, cost control and coverage deduplication require continuously pruning redundant scenarios and ensuring that additional test coverage actually adds incremental value rather than just increasing execution spend. cost control and coverage deduplication Finally, maintaining clear accountability is critical as agents take on more work. Even when agents generate, execute, and update tests, QA must own all agent outputs and remain the ultimate decision-maker on what ships. This means establishing review workflows, audit trails, and sign-off checkpoints so that autonomous activity is always paired with human oversight and responsibility. maintaining clear accountability But, even here AI agents can help as they are perfectly capable of performing review tasks. Moreover, it is possible to try multi-agent review: run three review agents, let them debate their findings, come to an agreement, then have a human receive the final distilled report and perform their own sanity checks.The end game is integrating agentic AI testing directly into the software development lifecycle where agents don't just test code written by human developers. Instead they operate as part of a multi-agent AI development team where: AI engineers write code
that AI testers develop and execute tests for
and together they iterate until the task is complete. AI engineers write code that AI testers develop and execute tests for and together they iterate until the task is complete. Conclusion The key takeaways? To move to agentic AI: Start with low autonomy and constrained scope.
Use MCP/tools to give agents context without giving them production write access.
Track failure rate, coverage, and costs.
Expand autonomy only after reliability is proven. Start with low autonomy and constrained scope. Use MCP/tools to give agents context without giving them production write access. Track failure rate, coverage, and costs. Expand autonomy only after reliability is proven. Managing agentic AI for testing shifts the focus from traditional script maintenance to strategic oversight. It requires QA teams to embrace a new mindset where they guide, monitor, and refine autonomous agents rather than manually updating scripts. To get started with agentic AI testing, organizations should begin small, validating AI agents on low-risk scenarios and gradually expanding their scope as confidence builds. By following best practices and leveraging robust tools, QA teams can transform their testing processes, accelerate releases, and improve software quality in an increasingly complex digital landscape. Have a really great day!

Demystifying Agentic Test Automation for QA Teams

I ♥ writing code. Find me on GitLab.

Nominated for 2022 - HackerNoon Contributor of the Year - Javascript

Nominated for 2022 - Hackernoons Tech Storyteller of the Year

Nominated for 2022 - HackerNoon Contributor of the Year - Cloud Computing

Nominated for 2022 - HackerNoon Contributor of the Year - Heroku

Nominated for 2022 - HackerNoon Contributor of the Year - Design

Nominated for 2022 - HackerNoon Contributor of the Year - Salesforce

Too Long; Didn't Read

Effectively Managing AI Agents for Testing

Effectively Managing AI Agents for Testing

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

5 Signs You’ve Built a Secretly Bad Architecture (And How to Fix It)

Daniel Saks Predicts Agentic AI Will Empower Individuals and Boost Productivity

Active Inference AI: Here's Why It's The Future of Enterprise Operations and Industry Innovation

The Trick That Agentic Frameworks Pulled On Us

Exploring the Spatial Web Protocol at the 4th Annual Applied Active Inference Symposium

What Is Agentic AI?

5 Signs You’ve Built a Secretly Bad Architecture (And How to Fix It)

Daniel Saks Predicts Agentic AI Will Empower Individuals and Boost Productivity

Active Inference AI: Here's Why It's The Future of Enterprise Operations and Industry Innovation

The Trick That Agentic Frameworks Pulled On Us

Exploring the Spatial Web Protocol at the 4th Annual Applied Active Inference Symposium

What Is Agentic AI?

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps