Different types of automated tests


There is an abundance of terms related to automated testing: unit, integration, functional, end-to-end, smoke, client, acceptation… These terms are quite fuzzy and overlapping (consider e.g. client and acceptation tests or integration and functional tests) . However, it doesn’t matter how do you name them – it matters when, how and, especially, why you use them.

In this post I describe 4 kinds of test that I find the most useful. The nomenclature I use is by no means official (different people may use different names for these 4 test types) nor important. What is really important, are their underlying goals; understanding the reason for each type of tests. This is what I’ll be focusing on.

Unit Tests

Tests of this kind exercise single components (units) of an application. Such unit is usually a single class or, less often, a small group of strongly related classes, forming a single logical concept. The characteristic attribute of unit tests is complete isolation. If a tests includes other classes, DB or network connections, system calls etc. it is not a true unit test. All dependencies of a tested class should be stubbed, so only the logic of that single class is exercised.

The necessity for such strong isolation is driven by two key goals:

The first goal is an application composed of bug-free components.

To rest assured that your app is correct, your want as high test coverage as possible. This implies that you’ll have a lot of tests – tests that first must be coded and then run again and again. What, in turn, implies that they must be easy to code and fast.

Avoiding instantiation of a vast trees of chained classes in each test, and avoiding time consuming connections to external resources (DB, web services, file system etc.) is exactly what makes tests fast. Isolation from external resources also simplifies tests’ setup (no need to load fixtures, clean up files, read configuration etc.) what further improves tests’ speed.

Full isolation also makes coding tests easier. The complexity of a test (possible combinations of input parameters, boundary conditions etc.) depends exponentially on number of interrelated classes participating in such test. Narrowing all tests to a single class, without dependencies, can lower number of tests cases needed to thoroughly test a system by an order of magnitude. Additionally, cutting out all dependencies to external resources, greatly lowers effort required to prepare test environment.

The second goal is an application that is easy to modify.

Application evolves over time. Requirements change. Design changes. Code is refactored. This is especially emphasized if you’re into XP-like emergent design and constant, aggressive refactoring. To make such continuous evolution possible, your code and tests can’t be brittle nor time consuming to change.

The key here is high cohesion and low coupling. Change of any single concept in your application should require modification of only single class’ code (or a very narrow group of classes). This allows you to modify your application with a relatively low effort. Also, change in one of your classes shouldn’t break tests for any other classes – this allows you to modify your application in a controlled way (and also reduces effort).

Full isolation of your unit tests is a big step towards achieving such modular design: if you’re able to thoroughly test a class without touching any of its dependencies, you usually should also be able to modify this class without having to modify or breaking tests for any other classes.

How many unit tests should I have?

As many as possible. Except of maybe plain getters and setters without any additional logic, you should test every part of public interface of every class. You should take into account all possible edge cases, boundary conditions etc.

Focused Integration Tests

Tests of this kind check connections between components and an external resources (DB, web service etc.). The characteristic attribute of focused integration tests is testing only a single connection. So, for a class with connections to many external resources there should be separate focused integration test for each such dependency.

The goal is an application talking correctly with its environment.

Unit tests guarantee that your classes internally work as expected. They verify that your class properly parses config file contents or correctly applies business logic to data received from a web service. However, can you be sure that this config file is opened properly or that a web service connection request is correctly formatted? Are you sure this class will be able to talk to a real service or file system? That’s what focused integration tests verify.

Integration tests focus on external connections, which are slow. This implies that they should be limited in scope and shouldn’t test business logic.

Limiting each integration test to a single connection helps avoiding exponential growth of complexity in a similar way like full isolation does in case of unit tests. Avoiding testing business logic (which already should be thoroughly tested by unit tests) and limiting integration test to checking only the connection itself, reduces the number of calls that must be made to an external resource in your tests, making your test suite faster.

How many focused integration tests should I have?

In general, as many as external connections you code has. In reality the number of tests may be higher, as some connections may have several different states you need to handle (e.g. a normal, successful call to a web service and a time-outed call). However, you should strive to keep numbers of external connection to a minimum: e.g. if more than one of your classes needs to read a config file, you should centralize file access in a single config parser class, and create focused integration tests only for this single class.

End-To-End Tests

These tests exercise all the application components – UI, business logic and connections to external  resources (DB, web services etc.) – together, in a single test. The characteristic attribute of end-to-end test is a black-box approach. They should exercise an application from an end user perspective, providing input and receiving output only via application GUI, without accessing any of its internals.

The goal is an application correctly glued together.

All pieces of your application are already well tested: every piece of business logic is covered by unit tests and every resource connection by focused integration tests. But you also need to be sure that all the pieces will collaborate together as expected. That’s what end-to-end tests are for. They should check that the whole application, from UI to external resources, is correctly composed of its building blocks: that all dependencies and configuration are correctly set, that input and output data flow properly up and down through the whole application stack and so on.

This implies that end-to-end tests are the only kind of tests that should not strive for isolation. It is exactly the opposite – the sole purpose of end-to-end tests is to exercise all applications components together at the same time. That’s why such tests should rather access application “from outside”, via tools like Webrat or Watir, than to play with its internals. It implies also that they are even slower than focused integration tests, so they too shouldn’t test business logic nor boundary conditions.

How many end-to-end tests should I have?

As few as possible to ensure that the application is glued correctly. Because end-to-end tests access application via its GUI, they are the slowest and the most brittle of all kinds of tests. Therefore, they should check only primary paths through an application (plus maybe a few of most important error scenarios) leaving all edge cases to other, more focused tests.

Customer Tests

These tests are technically very similar to end-to-end tests. They also exercise the complete application from a black-box perspective, working only through its GUI. However, their most characteristic attribute is that they are created in close cooperation with customers or sometimes even directly by customers. That’s because – despite technical similarities – they have a different goal than end-to-end tests.

The goal is code which does what user expects.

All kinds of tests mentioned before verify programmer’s expectations. They ensure technical correctness. However, the fact that an applications is build from bug-free components, can talk with its environment and is correctly glued together doesn’t automatically mean that it does what its users want. Therefore, we need an additional kind of tests, designed to clarify requirements. This implies that these tests should exercise an application only through its GUI and be written in a language understood by a customer, not in technical lingo.

The reason for testing only via GUI is a bit different than in case of end-to-end tests. End-to-end tests do it because they must exercise the whole application to check if all layers collaborate correctly. The customer tests do it because they verify user requirements, and user, not being involved in design or coding, can have requirements only regarding the application’s external interface – she doesn’t care how the application works internally, she cares only what visible results it produces.

For similar reasons it is important to avoid technical jargon in customer tests. Requirements, from the customer point of view, are centered around actions she can accomplish with an application not around low level steps needed to perform these actions nor underlying technology. Therefore, they should use vocabulary like “sign-in”, “set meeting date”, “print an invoice”, not “fill in login and password input fields”, “choose a value from a date picker widget” or “add an invoice to a background print queue” (which may be perfectly valid for other, programmer-centric test types).

How many customer tests should I have?

No more than necessary to understand how the application should work. Keep in mind that customer tests are not acceptance tests and therefore shouldn’t cover all usage scenarios. Customer tests are meant to work as live examples, clarifying most complicated parts of an application logic where needed. If the application domain is simple or well understood by programmers it is perfectly valid to have only a few (or even none) customer tests.


1 Comment

  1. Good Article, thanks.

    I shows the important points for the different testing types. Thats always a problem I have. How much and which kind of tests i should do with the different test types.

RSS feed for comments on this post

Comments are closed.

%d bloggers like this: