We are using cookies for secure log-in and to improve our app.
First party only, no data krakens 🦑 Learn more in our privacy policy.
Writing a single test is fun. Writing a test suite that scales is… sometimes less fun.
The more tests you write the less it becomes a matter of individual tests, and more a matter of designing the whole testing system. There are countless decisions to be made.
The bad news is that even if you are a well seasoned tester, practices from one company may not be directly applicable to another company. Every application is different and therefore each requires a different strategy. There really is no best practice.
The good news is, that there are at least some areas, with challenges that many have been through before and ones we can learn from. In this blogpost, I’d like to discuss one particular category of these challenges - test isolation.
There are many test automation tools that offer automatic script generation. Playwright has codegen features, Cypress has its Studio, Selenium offers a full IDE. There are even whole products based on the idea of recording and replaying tests.
While these tools are fun to use and can be great for learning the basics, they quickly fall short, when it comes to the repeated use of code that they generated.
Even just re-running a newly recorded test can be a problem, because the data created in the first recording may get in the way of the second run.
But it’s not just about data. Modern websites these days use cookies, local storage, indexed databases and other forms of local in-browser storage that provides important context to the application’s frontend code, or servers.
The simplest example of this are personalized cookie settings. These settings are usually saved in (you guessed it) cookies, but once a user opens a new browser or enters incognito mode, these settings are gone.
This same problem then translates to test automation. Usually, testing frameworks clean up the state of the browser to avoid polluting the context of individual tests. But as the mentioned example shows, there are many situations where this might actually become a problem.
describe('Cookie Consent Tests', () => {
test('new user sees cookie banner', async () => {
await page.goto('https://yourpage.com');
// A new user should see the cookie consent banner
const banner = await page.locator('.cookie-banner');
expect(await banner.isVisible()).toBe(true);
});
test('returning user does not see banner', async () => {
// First visit to set the cookie
await page.goto('https://yourpage.com/')
await page.click('.cookie-banner-accept');
// Simulate returning to the site
await page.reload();
const banner = await page.locator('.cookie-banner');
expect(await banner.isVisible()).toBe(false);
});
});
In practice this means that there are two different perspectives or user flows that need to be tested in order to get a good coverage:
There’s a tendency to think in user scenarios when creating test automation, but even the same scenario can have different outcomes given a difference in context.
When it comes to test isolation, I feel like it’s one of those principles that every test automation beginner learns about. A test should not interfere with any other tests. Simple as that.
But then when it comes to reality, it’s much, much harder to stick to this. Let’s take a simple example where we want to test a to-do app. We want to:
In principle, these should of course be two separate tests. But you can see how tempting it is to merge them together to speed up the execution. Because after all, if we really want to isolate the second test, we’ll need to create an item anyway.
// tests depend on each other
describe('Todo List', () => {
test('create todo item', async () => {
await createTodo('Buy milk');
});
test('delete todo item', async () => {
await deleteTodo('Buy milk');
});
});
// each test is independent
describe('Todo List', () => {
test('can create new todo item', async () => {
const todoText = 'Buy milk';
await createTodo(todoText);
const newItem = await page.locator('.todo-item', { hasText: todoText });
expect(await newItem.isVisible()).toBe(true);
});
test('can delete existing todo item', async () => {
// Setup: Create item specifically for this test
const todoText = 'Delete me';
await createTodo(todoText);
// Actual deletion test
await deleteTodo(todoText);
const deletedItem = await page.locator('.todo-item', { hasText: todoText });
expect(await deletedItem.isVisible()).toBe(false);
});
});
In real-life scenarios, we of course deal with much more complex scenarios, but the decisions that need to be made are similar in principle. When deciding on whether to merge tests together, create dependencies or to fully isolate them, I personally always side with isolation.
This is mostly because I want to be able to run tests in parallel. While full test isolation might create a slight increase in execution time, parallelization will decrease it exponentially.
It is good to consider parallelization from day one of creating test automation. It’s much more complicated to achieve it if tests are not properly isolated. But what is proper isolation?
Consider testing an arbitrary SaaS platform - The most basic entity that decides how the page looks would usually be a single user account. So for a fully parallel test execution, each parallel process must run as a different user. In that case, when running 10 parallel processes, we must create 10 testing accounts.
In the case of an e-commerce application things might get a bit more complicated, because even if we create separate customer accounts, we still need to deal with available items in the store. In case they run out, the test automation would unexpectedly fail. This is a situation in which the basic entity would be the store itself. Before test execution, data of the whole store must be ready to “fulfil orders” of the whole test run. This of course might get pretty complicated, but it is a huge part of creating good test automation.
When dealing with test automation, data is being thrown around everywhere. That’s what should happen. When your test automation resembles the way your app is going to be used, it will create, modify and delete data during the process. The main question is how to design a test suite in such a way that this data movement does not become a problem for both your system nor your test stability. These are some of the concerns, just to name a few:
The data isolation problem ties all of the previous problems together. While it might be tempting to reuse existing data across tests, this approach can become a hellish nightmare once tests run in parallel. Whenever possible, I’d advise for as much data isolation as possible.
One of the common approaches is to create a data factory pattern that generates isolated data for each test. For example, when testing a user profile feature, rather than sharing a single test account, your data factory might look something like this:
const createTestUser = async (prefix: string = ''): Promise<User> => {
const user = {
username: `test_user_${prefix}_${Date.now()}`,
email: `test_${prefix}_${Date.now()}@example.com`,
preferences: {
theme: 'light',
notifications: 'enabled'
}
};
await backend.createUser(user);
return user;
}
This approach ensures that each test has its own unique data set, eliminating potential conflicts between parallel test runs. However, when debugging, it’s beneficial to have a way to track data that was created. The example above randomizes the user name and email using Date.now(), but in case of debugging they might be a bit hard to find. An effective way of solving this is to produce an output of a test run that stores these names.
While creating isolated data is important, cleaning up that data is equally crucial. Test data can accumulate and cause various problems. If your tests run highly parallel and on every code change, databases quickly fill up, slowing down the system under test and potentially even leading to increasing costs.
There are often discussions on whether a data cleanup should happen before test execution starts, or after it finishes. I’d argue for doing it before the test execution, ideally completely isolated from the test script itself, as this means you can still access the test environment when debugging after a specific test run and ensure that a potentially failed teardown from before won't affect your next test run.
All things considered, data isolation strategies should evolve with your test suite. It’s valuable to plan ahead, but it’s pretty much impossible to plan everything as the application under test evolves. Premature optimization can be a costly endeavor, with questionable results, so your test suite and isolation maturity should grow together with your product needs.
___
At Octomind, we ensure test isolation by a variation of the test factory pattern from before. The core idea is that every test run generates new, unique data - each entity (such as documents, users, or transactions) is created with distinct identifiers and names. This means that even if multiple tests rely on an operation like "create a new document," each test instance will produce an independent document when executed, preventing unintended dependencies or data collisions.
This approach enables parallel execution since tests are not competing for the same resources, significantly reducing overall test runtime. Moreover, it makes running tests against multiple environments seamless, as each test case remains self-contained and does not introduce cross-environment conflicts.
