From Test Automation to Agentic Quality Engineering: MCP, Playwright and RAG

[Ritesh_M]

From Test Automation to Agentic Quality Engineering: A Practical Journey with Playwright, MCP and RAG Introduction

Software testing is going through another major transformation.

Traditional automation has already moved testing from manual execution to repeatable, CI/CD-driven automation. The next evolution is AI-driven and agentic testing, where AI is not limited to generating code but can also understand a test objective, interact with an application, generate automation, execute tests and analyze failures.

This approach is increasingly referred to as Agentic Quality Engineering.

I explored this approach through a practical proof of concept using Playwright, MCP agents and Retrieval-Augmented Generation (RAG). The objective was to understand how AI could be integrated into an existing automation framework and eventually executed through a CI/CD pipeline.

The implementation covered the complete flow—from use-case understanding and application exploration to test generation, execution, failure analysis and headless CI/CD execution.

---

What Is Agentic Testing?

Traditional automation generally follows:

Requirement
    ↓
Test Case
    ↓
Automation Script
    ↓
Execution
    ↓
Pass / Fail
    ↓
Script Maintenance

In an agentic approach, the workflow becomes:

Requirement
    ↓
AI Agent
    ↓
Retrieve Relevant Context
    ↓
Explore Application
    ↓
Generate Test Assets
    ↓
Execute
    ↓
Analyze Result
    ↓
Recover / Recommend / Escalate

The major difference is that the AI agent participates in the reasoning and decision-making process, rather than only producing automation code.

---

Our Practical Implementation

The most interesting part of this approach was implementing it against a real automation workflow.

The POC used Playwright as the browser automation layer, an MCP-based AI agent for tool interaction, and RAG for project-specific knowledge.

The overall flow was:

Test Use Case
     ↓
Markdown File
     ↓
AI Agent
     ↓
Application / DOM Exploration
     ↓
Locator Identification
     ↓
Method Generation
     ↓
Test Case Generation
     ↓
Playwright Execution
     ↓
Failure Analysis / Recovery
     ↓
CI/CD Headless Execution

The goal was to move beyond a simple AI coding assistant and build an AI-assisted testing workflow that could work with the existing automation framework.

---

1. Start With a Business Use Case

Instead of directly writing automation code, the functional requirement was maintained as a structured Markdown use case.

For example:

Booking Use Case

1. Login to the application.
2. Navigate to the booking module.
3. Search using a registration number.
4. Select the required service.
5. Enter mandatory information.
6. Submit the booking.
7. Validate booking confirmation.

The use case becomes the starting point for the AI agent.

This creates a separation between:

What needs to be tested

and

How it should be automated.

---

2. Let the Agent Understand the Application

The next step was connecting the AI agent to the application through Playwright MCP.

The agent could interact with the application and inspect the DOM/page structure to identify the elements required for the requested workflow.

Conceptually:

Use Case
   ↓
Open Application
   ↓
Inspect DOM / Page
   ↓
Identify Elements
   ↓
Map Elements to Actions

For a booking workflow, this could include identifying registration fields, service selection, customer information, submit actions and confirmation messages.

This reduced the amount of manual effort required during initial automation development.

---

3. Generate Locators, Methods and Tests

Once the relevant elements were identified, the next step was converting that understanding into automation assets.

The workflow was:

DOM Understanding
      ↓
Locator Identification
      ↓
Page Method Generation
      ↓
Test Case Generation

The objective was not to generate isolated scripts.

The generated automation was designed to fit into the existing Playwright framework structure and reusable automation approach.

The agent could also derive positive, negative and exception scenarios from the use case.

For example:

Positive: valid registration, valid details and successful booking.

Negative: invalid registration, missing mandatory information.

Exception: duplicate booking, expired registration, session timeout or backend failure.

---

4. RAG as the Knowledge Layer

Browser interaction alone is not enough for enterprise testing.

A project also contains:

• Existing test cases

• Business rules

• API specifications

• Domain knowledge

• Test data requirements

• Known defects

• Historical failures

• Framework conventions

This is where RAG becomes important.

In the POC, RAG was explored as a way of providing the AI agent with relevant project-specific context.

                AI Agent
                   │
                   ▼
                  RAG
                   │
      ┌────────────┼────────────┐
      ↓            ↓            ↓
 Test Cases   Documentation   Domain Rules
      │            │            │
      └────────────┼────────────┘
                   ▼
               Context
                   ↓
            AI + Playwright

A useful way to think about the architecture is:

RAG       → What does the agent need to know?
MCP       → What can the agent do?
AI Agent  → How should it reason and act?

This effectively gives the automation framework a project knowledge or memory layer.

---

5. Intelligent Failure Analysis

A major benefit of the agentic approach is the ability to investigate failures instead of simply reporting them.

Traditional automation may report:

FAILED
Element not found

An agentic workflow can examine additional evidence:

Test Failure
    ↓
DOM Inspection
    ↓
Locator Analysis
    ↓
Application State
    ↓
Network / API Information
    ↓
Historical Context
    ↓
Failure Classification

The issue could be:

• Locator changes

• Test-data problems

• Environment instability

• Synchronization issues

• Backend failures

• Workflow changes

• Actual application defects

This enables a broader form of self-healing, where the system attempts to understand the reason for failure before recommending or applying a correction.

That distinction is important because automatically changing a locator could otherwise hide a genuine application defect.

---

6. CI/CD Integration and Headless Execution

A major part of the implementation was taking the automation beyond local execution and making it suitable for CI/CD execution in a headless environment.

To support this, the framework was packaged with the required execution components, including:

• test.bat file for standardized test execution

• Test sequence files to control execution order and test flow

• Docker files to containerize the execution environment

• Required Playwright/browser dependencies

• Headless execution configuration

• CI/CD pipeline integration

The overall execution flow became:

Git Repository
      ↓
CI/CD Pipeline
      ↓
Docker Environment
      ↓
test.bat / Test Sequence
      ↓
Playwright
      ↓
Headless Browser
      ↓
Test Execution
      ↓
Results / Logs

This was an important step because the POC was no longer dependent on a developer's local machine.

The same automation could be packaged into a controlled environment and executed through the pipeline using headless browsers.

Docker also helped provide consistency between local and CI environments by packaging the required execution dependencies.

---

7. Integrating AI With the Existing Framework

The objective was not to replace the existing automation framework.

The framework continued to provide components such as:

• Page Objects

• Test cases

• Utilities

• Test data

• Reporting

• Version control

• CI/CD

AI was introduced as an intelligence layer around these existing components.

Existing Playwright Framework
          │
          ├── Page Objects
          ├── Test Cases
          ├── Utilities
          ├── Test Data
          └── CI/CD
                  │
                  ▼
             AI Layer
             ┌────┴────┐
             ▼         ▼
            RAG       MCP
             │         │
         Knowledge   Tools

This makes the approach more practical for enterprise adoption.

---

8. From AI Automation to Intelligent CI/CD

With the framework integrated into CI/CD, the next evolution is making the pipeline more intelligent.

A future workflow could be:

Code Change
    ↓
AI Impact Analysis
    ↓
Identify Affected Areas
    ↓
Select Relevant Regression Tests
    ↓
Dockerized Execution
    ↓
Playwright Headless Execution
    ↓
AI Failure Analysis
    ↓
Evidence + Recommendation
    ↓
Quality Gate

Instead of simply running automation, the goal is to make the pipeline more context-aware and intelligent.

The long-term vision is not just faster execution, but smarter decisions about what to test, what failed and why.

---

9. Human-in-the-Loop and Security

Agentic testing should not mean unrestricted automation.

An agent can be allowed to:

• Read requirements

• Explore test environments

• Generate tests

• Execute approved tests

• Collect evidence

• Analyze failures

Higher-risk actions should remain controlled.

Enterprise implementations therefore need:

restricted permissions, approved environments, tool controls, auditability and human approval where required.

---

10. Testing the AI Tester

Another important question is:

How do we know the AI agent itself is producing good results?

The agent should be evaluated using measures such as:

Area Example Requirement understanding Correct interpretation Test generation Valid scenarios Coverage Critical scenario coverage Locator generation Reliability Failure analysis Root-cause accuracy Self-healing Recovery success Tool usage Correct tool selection Stability Repeatability

This creates two quality layers:

Application Quality
        +
Agent Quality
        ↓
Overall Quality

---

Key Learnings From the Practical Implementation

The project highlighted several important lessons.

Context is critical.
AI can generate technically valid automation but still misunderstand business behavior without project-specific context.

RAG and MCP complement each other.
RAG provides knowledge, MCP provides tools, and the AI agent provides reasoning and orchestration.

Existing automation architecture remains important.
AI should enhance established practices such as Page Objects, reusable utilities, CI/CD and version control.

Self-healing needs diagnosis.
A failure should be understood before automation is changed.

CI/CD is essential for real-world adoption.
Packaging the solution with test sequence files, test.bat, Docker and headless execution helped move the POC from a local experiment toward a repeatable engineering workflow.

Human expertise remains important.
AI can accelerate testing, but quality engineering still requires domain knowledge, risk assessment and engineering judgment.

---

What This Means for QA Engineers

The role of the QA engineer is evolving from simply writing and maintaining scripts toward designing intelligent quality systems.

The skillset is expanding from:

Testing + Automation

to:

Testing + Automation + AI + RAG + MCP + CI/CD + Agent Evaluation

This does not mean replacing QA engineers.

It means reducing repetitive work and allowing engineers to focus more on test strategy, product risk, complex failures and quality decisions.

---

Conclusion

The combination of Playwright, MCP and RAG creates a new direction for software testing.

In our practical implementation, the journey started with a Markdown-based use case, followed by AI-driven application exploration and DOM understanding, locator and method generation, test creation, Playwright execution, and failure analysis.

RAG added project-specific knowledge, while MCP provided the agent with the ability to interact with testing tools.

The solution was then taken beyond local execution by integrating it with CI/CD, creating test sequence files, a test.bat execution layer and Docker-based execution, and running the Playwright tests in headless mode.

This demonstrates an important shift:

AI can become an intelligence layer around an existing automation framework—not just a tool for generating test scripts.

The future of QA may therefore not be about writing more automation.

It may be about building intelligent quality systems that can understand requirements, interact with software, generate and execute tests, analyze failures and continuously assist engineers throughout the software delivery lifecycle.

[Thaufique_Mod]

Thanks for posting this informative topic. It is very helpful.