Part IV — Win Some. Lose Some.

Quick recap: In previous parts we covered the notion of E2E testing, extended to the mobile world and discussed the tooling required.

Software design is the art of trade offs. You make some decisions that will gain some value but will likely preclude some other benefits. A famous example is the Project Management Triangle, balancing between cost, time and scope.

A similar, but more specific triangle can be applied on E2E testing (although the above is still true for any project). I would claim that when you design your tests, you may balance between 3 factors:

Coverage Velocity

The coverage and the coverage velocity defines how many of your business flows are covered by the tests, and how fast you can achieve this coverage. As your requirements and hence your software functionality is likely to change often, it also means how fast you can adapt your tests to those changes.

The exact number of tests vary according to the specific domain. 20 test scenarios may provide a good coverage for a small consumer app, but will barely scratch the surface for a large enterprise web software.

Based on the resources you have, you can design your tests in a manner that will allow reuse of testing actions when creating new tests easily or tailor craft each testing scenario.

Resilience

E2E tests are infamous for being flaky. Changes in the underlying software is one example, but also careless design that does not take into considerations the activity that occur in the system under the hoods, such as the time it takes to get the data from the server, can make the tests extremely brittle. Verifying each screen element prior to handling it and waiting until it is available reduces the chances of tests to break, but takes it costs elsewhere.

Performance

This simply says how long the tests will run. Releasing multiple times a day requires that the tests will run often and relatively fast. But with less frequent releases, extra minutes during the run may not be so critical, or may be compensated with additional hardware.

What Does It mean?

Writing a very small set of E2E tests, verifying basic scenarios such as login and a main flow, will probably run relatively fast (high performance) and will not break often (high resilience) but is only testing small part of functionality (low coverage). This is sometimes referred to as Smoke Tests (or Sanity tests).

Using automated recording tools (such as Selenium IDE or Katalon) can achieve a large number of tests in a short time (high coverage) and the tests may run fast with very explicit selectors (high performance), but are likely to be very brittle and prone to break on every small change in the system (low resilience).

Creating tests with lots of generic functions to select the elements can expedite the creation of additional tests (high coverage) and can make them relatively stable (high resilience), but will have a cost at execution time (low performance).

I will demonstrate the last example in a piece of code we use in our project. We have a function called SafeClick (code is in webdriver.io syntax):

export function safeClick(el, options?, timeout?) {
    options = options || {};
    browser.waitForExist(el, timeout);
    if (!options.skipVisible) {
        browser.waitForVisible(el, timeout);
    }
    let tagName = browser.getTagName(el);
    if (['button', 'input', 'textarea', 'a'].indexOf(tagName) > -1 )     {
        browser.waitForEnabled(el, timeout);
    }
    browser.click(el);
}

This code is waiting for the element to exist, then checking the type of the element and if it is an element that can be enabled (i.e. not a div or a custom element), it will wait for it to become enabled and eventually perform the click.

This function enables us to write tests faster. We do not need to repeat in every test the wait for the element to become visible or enabled, and we are hiding this complexity away from the test scenarios, but at the same time we are paying an extra cost of superfluous browser access. In the above case, we will access the browser at least 4 times:

• Finding that the element exists
• Finding the element tag name
• Checking the element enable state
• Clicking on the element.

What is Right For me?

Needless to say, there is no one correct answer for that. It is common to focus on performance of the tests and ignore other aspects, but eventually the correct answer is the one that serves your business goals the best.

Need someone to help you around the E2E testing world? Read the whole series. If you are happy and joyful — you may clap!