how does the AI agent work?

We deploy AI to simulate user behavior to automatically generate end-to-end tests. It's followed up by deterministic systems that generate and run the tests.

see the agent in action

Octomind AI agent in a nutshell

Octomind AI agent mimics human users (i.e., clicks input fields, signs up for newsletter) to navigate apps, interprets app intent, and identifies all relevant user flows. This is how it runs when you sign up:

The agent looks at your site.
It looks for cookies and a 'required' login first. 'Required' is the type of login you need to get further in the app.
When a 'required' login flow is found, we will ask you for your login test credentials.
The agent will put these 2 tests as pre-requisites (dependencies) to all new tests.
Next, the agent automatically generates up to 3 meaningful tests.
If successful, it validates the tests and files them as 'active'.
We will run all active cases.
You can have the agent generate more tests sequential to the active test (up to 3 on each run) they were launched from.
You can prompt the AI agent to generate a specific test case.
You can run the AI agent to re-generate a broken test.

AI agent knows what to test

Our AI agent traverses the publicly accessible code in the DOM and uses the vision capability of the underlying multimodal LLM to add visual context when code insight is insufficient. It tries to understand the purpose of your site to generate relevant end-to-end tests.

[data-wf-bgvideo-fallback-img] { display: none; } @media (prefers-reduced-motion: reduce) { [data-wf-bgvideo-fallback-img] { position: absolute; z-index: -100; display: inline-block; height: 100%; width: 100%; object-fit: cover; } }

a screenshot of octomind AI agent generating test steps

AI agent generates tests, step by step

Our AI mimics user behavior to interact with apps as a human would to reach the goal of a user flow. These interactions are represented in test steps.

We record and store each test case's interaction chain and generate the corresponding Playwright code deterministically on the fly immediately prior to test execution.

AI auto-maintenance

work in progress

Octomind automatically determines if test failures are caused by user flow behavioral changes, the test code itself, or bugs in the code.

In the case of a behavioral change, we pinpoint failing interactions, and deploy the AI Agent to detect new desired interaction that will allow us to achieve the test case's goals.

fight flaky tests with AI

work in progress

We already deploy a variety of techniques to fight test flakiness. Yet it's not enough.

That's why we are investigating the best ways to implement AI-based analysis of unexpected site behavior, like temporary pop-ups or toasts, and further improve testing performance.

blogtopus: on AI, agents and app testing

50% developer rants, 50% tech news commentary, 100% cephalopod opinions on all things software. It's worth the read. Guaranteed.

October 10, 2024

How to start with e2e testing from scratch?

An Octomind tutorial for developers who want start end-to-end testing. Let me show you what getting up to speed with Octomind looks like so you can see if it’s something you’d want to try out for your own websites or apps.

Why we no longer use LangChain for our AI agents

When abstractions do more harm than good - lessons learned using LangChain in production and what we should’ve done instead.

Did you break your code or is the test flaky?

Flaky e2e tests are frustrating for QA and development teams, causing constant disruptions and eroding trust in test outcomes. A fail-safe checklist for how you can tell the difference.