Different ways of code reuse in RSpec

Hierarchical structure of RSpec, with nested context/describe blocks and hierarchical before/after hooks, topped with a bit of syntactic sugar like let/let! methods, already goes a long way towards clean and readable tests. However, more complex suites still can quickly go un-DRY and unwieldy without code reuse between example groups.

Fortunately, RSpec provides several different tools for reusing test code: helper methods and modules, custom matchers, shared examples and shared contexts. It also allows to create examples programmatically (eg. in a loop). In this post I’ll briefly introduce all these options and discuss use cases for each of them.

RSpec helper methods and modules

To encapsulate common, complex logic RSpec allows you to write arbitrary helper methods directly within an example group (or in a parent example group):

describe "parent context" do
  def parent_helper_method
    # ...
  end

  describe "nested context" do
    def nested_helper_method
      # ...
    end

    it "should see both inherited and local helper methods" do
      parent_helper_method
      nested_helper_method
    end
  end
end

To make helper methods even more reusable (between multiple, not nested example groups), it is possible to define them in a module:

module Helpers
  def helper_method
    # ...
  end
end

and then include it in your example:

describe "some context" do
  include Helpers

  it "should see helper methods from included module" do
    helper_method
  end
end

You can also mix-in helpers via extend instead of include or include them through RSpec configuration instead of directly in an example group, but I won’t dig into all the details right now.

When should you use RSpec helper methods and modules?

Two areas where helper methods are most useful are example initialization and execution.

The primary use of helper methods is to hide implementation details by grouping several low-level statements into more meaningful, higher level abstractions (with clear, expressive names). There is really no magic here, this is analogical to the function of methods in OO programming. E.g. if you want to test different outcomes of the last round in a perfect darts game, instead of manually go through the whole play like this:

describe "last round in a perfect darts game"
  before do
    2.times do
      3.times do
        player_1.shoot(20, triple)
      end
      3.times do
        player_2.shoot(20, triple)
      end
    end
  end
end

you can do it this way:

describe "last round in a perfect darts game"
  before do
    2.times do
      shoot_a_perfect_round(player_1)
      shoot_a_perfect_round(player_2)
    end
  end
end

or even:

describe "last round in a perfect darts game"
  before do
    shoot_a_perfect_two_rounds_for_both_players
  end
end

using helper methods to hide all the nitty-gritty details of what constitutes a perfect round in darts.

In a similar manner, you can also use helper methods as factories, hiding construction of complex objects (although this use case is largely superseded by dedicated libraries like factory_girl or machinist).

RSpec custom matchers

RSpec also allows you to easily create new matchers to encapsulate custom assertion logic into well named methods:

RSpec::Matchers.define :be_thrice_as_big_as do |expected|
  match do |actual|
    actual == 3 * expected
  end
end

describe "number nine" do
  it "should be three times three" do
    9.should be_thrice_as_big_as 3
  end
end

As in the case of RSpec helpers, this is only a basic example. RSpec custom matchers are way more powerful than this, providing chaining, custom failure messages, their own internal helper methods and so on. They can be also defined as modules instead of RSpec::Matchers.define method.

When should you use RSpec custom matchers?

Use cases for custom matchers are quite similar to these of helper methods (hiding low-level details and providing meaningful naming). They differ in their area of use – while helper methods are used for example initialization and execution, custom matchers are used for example verification.

Custom matchers also provide an option to define custom failure messages, what (especially in the case of more complex, multi-step assertions) is a great way to improve readability and expressiveness not only of the test code but also of the test output.

RSpec shared examples

Another powerful approach to code reuse offered by RSpec is to extract complete sections of system under test specifications into shared examples, like this:

shared_examples_for User do
  it "behaves in some way" do
    # ...
  end

  it "behaves in some other way" do
    # ...
  end
end

You can then use such extracted specification to define behavior of other classes:

describe Customer do
  it_behaves_like User

  it "also behaves in special, Customer-y way" do
    # ...
  end
end

describe Admin do
  it_behaves_like User

  it "also behaves in special, Admin-y way" do
    # ...
  end
end

Such inclusion of shared examples works exactly like they were hand-coded inside a host example:

describe Customer do
  context "behaves like User" do
    it "behaves in some way" do
      # ...
    end

    it "behaves in some other way" do
      # ...
    end
  end

  it "also behaves in special, Customer-y way" do
    # ...
  end
end

Once more, this is only a tip of an iceberg – RSpec shared examples offer a lot of additional flexibility, e.g. you may pass them extension blocks or parameters, they may use methods defined in their host and they can be included without nesting – but the above examples illustrate the core idea.

When should you use RSpec shared examples?

Shared examples are especially useful for specifying inheritance hierarchies (see the example above with User, Customer and Admin), mixins and “interfaces”:

shared_examples_for Serializable do
  # ...
end

describe Dictionary do
  it_behaves_like Serializable
  # ...
end

describe TreeGraph do
  it_behaves_like Serializable
  # ...
end

Shared examples can also be used when specifying different states of a single object, that have some behavior in common, e.g.:

shared_examples_for Account do
  # ...
end

describe "new account" do
  it_behaves_like Account
  # ...
end

describe "confirmed account" do
  it_behaves_like Account
  # ...
end

describe "blocked account" do
  it_behaves_like Account
  # ...
end

RSpec shared contexts

Shared contexts are the newest addition to the RSpec code reuse mechanisms’ spectrum (added in RSpec 2.6). You can think of them as the opposite to shared examples – while shared examples allow you to inject examples that use their host’s context (before hooks, helper methods etc.), shared contexts allow you to inject context that will be used by their host’s examples.

Let’s look at how this is done. First, we define shared context:

shared_context "shared stuff" do
  before do
    @shared_var = :some_value
  end

  def shared_method
    # ...
  end

  let(:shared_let) { :some_value }
end

Then, we include it in our example group like this:

describe "group that includes a shared context" do
  include_context "shared stuff"

  it "has access to everything defined in shared context" do
    @shared_var
    shared_method
    shared_let
  end
end

Again, there is more to it, e.g. you can include shared context via metadata instead of explicit include_context, but I skip it for the sake of example brevity.

When should you use RSpec shared contexts?

Shared contexts are mostly useful when there are several examples that share some initial state, but they are too many, too complex or too unrelated to specify them in a single describe block through nesting.

This could be to some degree achieved also by using helper methods, but helper methods and shared contexts differ a bit in scope – shared contexts are higher level, containing several different initialization elements (before hooks, let and subject methods etc.) at once, while helpers are more narrowly focused and are used rather inside these initialization elements.

Constructing RSpec examples programmatically

One more way to help you DRY your examples is to create them programmatically. This is especially useful when done in a loop. Instead of creating a lot of almost identical examples by hand:

describe "pluralize" do
  it "should map mouse to mice" do
    "mouse".pluralize.should == "mice"
  end

  it "should map cactus to cacti" do
    "cactus".pluralize.should == "cacti"
  end

  it "should map tooth to teeth" do
    "tooth".pluralize.should == "teeth"
  end
end

You can do it in a much more concise way:

describe "pluralize" do
  {"mouse" => "mice", "cactus" => "cacti", "tooth" => "teeth"}.each do |singular, plural|
    it "should map #{singular} to #{plural}" do
      singular.pluralize.should == plural
    end
  end
end

When should you create RSpec examples programmatically?

Examples can be generated programmatically when they are data-driven, that is when there are several examples that express almost identical behaviour, but with different input/output values that all need to be tested to cover all important boundary conditions.

Coding all examples for such cases explicitly, by hand is too tedious and not DRY and testing them using only a single example with a loop inside, like this:

describe "pluralize" do
  it "should map singulars to plurals" do
    {"mouse" => "mice", "cactus" => "cacti", "tooth" => "teeth"}.each do |singular, plural|
      singular.pluralize.should == plural
    end
  end
end

although very concise, will result in harder to debug tests, because the example will be stopped at the first error encountered, instead of running through all the data every time.

Final words

My final words are simple: with all the various reuse tools RSpec puts in your tool-belt, there is no excuse to allow any kind of duplication to creep into your tests. Happy and DRY coding!

PS: There is one more code reuse tool provided by RSpec: macros. However, after recent enhancements to the shared examples mechanism (passing parameters, extension blocks etc.) they are no longer necessary (see RSpec author’s notes on this topic here) so I don’t cover them in this post.

Comments (15)

Custom jQuery matchers in Jasmine

Implementing your custom matchers (or custom assertions in case of xUnit-like frameworks) is a very good testing practice. In theory, as you’re verifying if a given expectation is correct, ‘.toBeTrue()’ is the only matcher you need. In practice however, using convoluted constructs like:

expect(someVariable != undefined).toBeTrue();

severely hinders readability of your tests. Thats why more clear and explicit matchers like:

expect(someVariable).toBeDefined();

come in handy. Such neat matchers are of course more comfortable to use, but their real power comes from revealing the intent of a test – what is one of primary goals (and strengths) of the whole Behavior Driven Development concept.

Different frameworks provide different sets of built in matchers. The one provided by Jasmine is relatively bare-bone, but it is not a bad thing, as it offers neat and robust mechanism for creating your own matchers when needed without polluting an API with a myriad of predefined matchers for all occasions (most of which you’ll never use). In this post I’ll show how to built a custom jQuery matcher library for Jasmine (provided by my jasmine-jquery plugin).

The simplest way to create a custom matcher, presented in Jasmine docs, is to use “.addMatchers()” method in your “beforeEach()” block:

beforeEach(function() {
  this.addMatchers({
    toHaveClass: function(className) {
      return this.actual.hasClass(className);
    }
  });
});

“this.actual” is automatically set by Jasmine to the variable on which we set expectations via “expect(…)” method. Other parameters for your matcher can be set as usual, as input parameters of your callback function (see “className” in above example). Your matcher function must return a boolean value indicating if it succeeded or not.

Now, when you have your first jQuery matcher defined, you can use it in your tests like this:

it('should have class "my-class"', function() {
  var myElement = $(<div class="my-class"></div>');
  expect(myElement).toHaveClass("my-class");
});

instead of much less readable:

it('should have class "my-class"', function() {
  var myElement = $('<div class="my-class"></div>');
  expect(myElement.hasClass("my-class")).toBeTruthy();
});

Jasmine also automatically allows your custom matchers to be inverted by prepending them with “.not”, so you can also use your freshly created jQuery matcher like this:

it('should have class "my-class"', function() {
  var myElement = $('<div class="my-class"></div>');
  expect(myElement).not.toHaveClass("other-class");
});

So far so good. Your tests are now much more readable and their intent is expressed more clearly. However, when written in such a way (placed in “beforeEach()”), your matchers are local to your test file. In more specific cases this may be not so bad, but in case of e.g. jQuery related matchers, we will want to re-use them in many tests across many projects, so making a local copy each time is definitely not an option.

The solution to the above problem is simple – as you could see, “addMatchers()” method takes a JS object as a parameter. So instead of declaring this object locally, lets move it to a separate file:

in my_jquery_matchers.js:

jQueryMatchers = {
  toHaveClass: function(className) {...},
  toBeVisible: function() {...},
  ...
};

in my_spec.js:

beforeEach(function() {
  this.addMatchers(jQueryMatchers);
});

Now the matchers are neatly separated into their own file, allowing you to reuse this “library” among may test files. And as Jasmine allows you to use “beforeEach()” at arbitrarily high level – not only in your “describe” block but also at a level global to your whole test file, or even at a level of the test runner itself (global for all test files executed by this runner), you need to invoke “addMatchers()” only once, so your tests stay reasonably DRY.

When creating custom matchers specific to your code, above solution is sufficient. However, a matcher library for an external framework like jQuery feels more like a part of the test framework core, not as a part of test suite. Therefore, two problems that don’t bother me that much in case of my custom matchers, make me not fully satisfied in case of jQuery matcher library. First, variable containing matchers (jQueryMatchers in the above example) pollutes global namespace. Second, having to instantiate a part of the core library inside one of your “beforeEach()” blocks is awkward – these blocks should rather contain code related to your SUT (system under tests), not a framework’s boilerplate.

So, let’s take it one step further and “hack” our jQuery matchers into Jasmine framework directly. The idea may seem scary at first, but it is really straightforward. Jasmine keeps all its core matchers in a separate object called “Matchers”. All you need to do is to add your new matchers directly to the “Matchers” object’s prototype instead of passing them via “addMatchers()” method:

jasmine.Matchers.prototype.toHaveClass: function(className) {
  return this.actual.hasClass(className);
};

Now the only thing you need to do is to include the file in a test runner (the same way you include Jasmine core or jQuery library) and our jQuery matchers are transparently available in all test files, without polluting the namespace or any of “beforeEach()” blocks. (I’d like to reiterate here that for matchers related to your custom SUT only, this may be a little overkill, but it is a neat and elegant solution when creating “core” matcher libraries, reused among many projects).

OK, we’re almost there with our solution, however, I’ll apply two more tweaks that will make our library more robust.

First annoyance to fix is the way Jasmine displays failed assertions for our jQuery matchers. Inside our matcher, this.actual is a jQuery element, and Jasmine by default can’t do anything smarter than dump it to string in an error message. So when you try:

var myElement = $(<div class="wrong-class"></div>');
expect(myElement).toHaveClass("some-class");

you’ll get:

Expected { 0 : HTMLNode, length : 1, context : HTMLNode, selector : '#sandbox', init : Function, jquery : '1.4.2', size : Function, toArray : Function, get : Function, pushStack : Function, each : Function,
...
a _LOT_ of similar jQuery element dump code here
...
, width : Function } to have class 'some-class'.

This makes an error message almost impossible to comprehend. However, it may be quite easily fixed by using a simple trick – modifying “actual” variable inside our matcher before returning:

jasmine.Matchers.prototype.toHaveClass: function(className) {
  var result = this.actual.hasClass(className);
  this.actual =
      $('<div />').append(this.actual.clone()).html();
  return result;
};

What we do with “this.actual” above, is conversion from jQuery element into a plain HTML string. This results in much more explanatory message, leaving no doubt why our test didn’t pass:

Expected '<div class="wrong-class"></div>' to have class 'some-class'.

The last problem left for us to solve is a situation when there is naming collision between our jQuery matchers and built-in Jasmine matchers. E.g. I’d like to have a matcher called “toContain()” that will check if a Query element contains another element, specified by a jQuery selector, but Jasmine already has a “toContain()” matcher that checks if an array contains specified item.

The simplest way to solve this issue is just to check what matcher names are already taken by Jasmine and to avoid potential collisions by renaming our jQuery matcher to e.g. “toContainElement()” or “toContainSelector()”; However, I’m quite stubborn and I want a simple “toContain()”. I’m also lazy, and I don’t want to check all built-in Jasmine matchers – and what’s even more important, I don’t want a naming collision when new matchers are added to Jasmine core in future.

Another simple trick will do it:

var builtInMatcher = jasmine.Matchers.prototype.toContain;

jasmine.Matchers.prototype.toContain = function() {
  if (this.actual instanceof jQuery) {
    // our jQuery matcher code here
    // return results
  }

  return builtInMatcher.apply(this, arguments);
};

If the tested variable is a jQuery element our jQuery matcher will be used, otherwise we pass the control to the original Jasmine matcher. Also, invoking original matcher by using “apply(this, arguments)” ensures that even if our overlapping jQuery matcher and original Jasmine matcher expect different number of arguments the above code will still work correctly.

And that’s all. The full, more polished implementation can be found in my jasmine-jquery library (it contains also support for HTML fixtures – see my previous post about fixtures in Jasmine for a more detailed overview).

Comments (3)

HTML fixtures in Jasmine (using jasmine-jquery)

My comparison of JavaScript BDD frameworks resulted in choosing Jasmine to be my JS testing framework of choice. However, I’m doing a lot of jQuery based development, while Jasmine by design tries to stay DOM agnostic. It is not that big problem, as Jasmine is really flexible and open (I’ll show in a moment how to manipulate DOM with jQuery using plain-vanilla Jasmine), but with a couple of tweaks packed into my jasmine-jquery plugin, jQuery users can make their life even easier.

First, let’s start with the simplest approach possible to test jQuery code in your Jasmine specs. The most straightforward way is just to create a “fixture” jQuery element inside your spec, invoke your plugin on it and then set your expectations against the fixture element (to verify if the plugin modified fixture element correctly):

describe('simplest jQuery plugin test', function () {
  it('should use inline fixture', function () {
    var fixture = $('<div>some HTML code here</div>');
    fixture.myTestedJqueryPlugin();
    expect(fixture).toSomething();
  });
});

Of course, for any non trivial plugin you’ll need more than one spec to fully test its behavior, so the above example in reality will look more like this:

describe('more realistic jQuery plugin test', function () {
  var fixture;

  beforeEach(function () {
    fixture = $('<div>some HTML code here</div>');
  });

  it('should use inline fixture', function () {
    fixture.myTestedJqueryPlugin();
    expect(fixture).toSomething();
  });

  it('should use inline fixture again', function () {
    fixture.myTestedJqueryPlugin();
    expect(fixture).toSomethingElse();
  });
});

Above solution is still far from perfect though. One of the problems is that jQuery fixture element is created only in memory and never added to DOM. In effect, many tests (e.g. related to element visibility or dimensions) won’t work correctly. In order to solve this, we need to add our fixture to the DOM and, to avoid test interdependencies, we must clear the DOM between tests.

describe('DOM based jQuery plugin test', function () {
  beforeEach(function () {
    $('#fixture').remove();
    $('body').append('<div id="fixture">...</div>');
  });

  it('should use DOM fixture', function () {
    $('#fixture').myTestedJqueryPlugin();
    expect($('#fixture')).toSomething();
  });

  it('should use DOM fixture again', function () {
    $('#fixture').myTestedJqueryPlugin();
    expect($('#fixture')).toSomethingElse();
  });
});

Now it should work for all kinds of jQuery related tests, and the beforeEach doesn’t look that bad (only 2 lines of code), but unfortunately such a simple one-liner fixtures will rarely suffice for non trivial, real-life plugins. And believe me, inlining complex, multiline HTML strings in your test code is ugly, turning your well factored test code into one big tangled mess. Also, to keep your tests clean, you may want to reuse fixtures not only between tests in one file, but also between different test files. The answer here is loading fixtures from files. We can change our code to something like this:

describe('test with fixture files', function () {
  beforeEach(function () {
    $('#fixture').remove();
    $.ajax({
      async: false, // must be synchronous to guarantee that no tests are run before fixture is loaded
      dataType: 'html',
      url: 'my_fixture.html',
      success: function(data) {
        $('body').append($(data));
      }
    });
  });

  it('should use DOM fixture', function () {
    $('#fixture').myTestedJqueryPlugin();
    expect($('#fixture')).toSomething();
  });

  it('should use DOM fixture again', function () {
    $('#fixture').myTestedJqueryPlugin();
    expect($('#fixture')).toSomethingElse();
  });
});

Now your fixtures can be neatly organized into separate files and don’t pollute your test code. An added benefit is that they don’t have to be static – my_fixture.html file can be now e.g. generated via normal Rails action etc. However, beforeEach have grown a little. And this is just a tip of an iceberg. Right now the fixture file is reloaded via AJAX before each of your tests, which results in a major penalty to your test suite’s speed – some kind of fixture caching would be welcome. Also, what if you want to load more than one fixture? You’ll then have to duplicate fixture loading and DOM clean-up code. And what if for some reason you want to load fixture from external file, but don’t want to append it automatically to DOM, but rather assign to a variable inside your test? Again, this can be easily done, but it will add another variation to your beforeEach code. Even worse, you’ll probably use fixtures in more than one test file, so your fixture loading code will end-up copy pasted all over the place. As you can see this can quickly go out of control for any non trivial project.

jasmine-jquery to the rescue! ;)

It extends Jasmine framework with a robust set of methods to handle HTML fixtures. It supports also fixture caching to speed-up your test suite and auto clean-up of DOM between tests. With jasmine-jquery you can write your tests like this:

describe('test with jasmine-jquery', function () {
  it('should load many fixtures into DOM', function () {
    loadFixtures('my_fixture_1.html', 'my_fixture_2.html');
    expect($('#jasmine-fixtures')).toSomething();
  });

  it('should only return fixture', function () {
    var fixture = readFixtures('my_fixture_3.html');
    expect(fixture).toSomethingElse();
  });
});

And that’s all. Yes, there doesn’t have to be beforeEach at all. Fixtures are automatically appended to DOM by loadFixtures method and automatically cleaned-up between tests. Apart from loadFixtures and readFixtures jasmine-jquery provides other useful methods, so don’t forget to check jasmine-jquery GitHub page for the full documentation.

Apart of HTML fixtures support, jasmine-jquery provides also a robust set of Jasmine custom matchers for jQuery – I’ll present them in more detail in next post.

Comments (34)

Different types of automated tests

There is an abundance of terms related to automated testing: unit, integration, functional, end-to-end, smoke, client, acceptation… These terms are quite fuzzy and overlapping (consider e.g. client and acceptation tests or integration and functional tests) . However, it doesn’t matter how do you name them – it matters when, how and, especially, why you use them.

In this post I describe 4 kinds of test that I find the most useful. The nomenclature I use is by no means official (different people may use different names for these 4 test types) nor important. What is really important, are their underlying goals; understanding the reason for each type of tests. This is what I’ll be focusing on.

Unit Tests

Tests of this kind exercise single components (units) of an application. Such unit is usually a single class or, less often, a small group of strongly related classes, forming a single logical concept. The characteristic attribute of unit tests is complete isolation. If a tests includes other classes, DB or network connections, system calls etc. it is not a true unit test. All dependencies of a tested class should be stubbed, so only the logic of that single class is exercised.

The necessity for such strong isolation is driven by two key goals:

The first goal is an application composed of bug-free components.

To rest assured that your app is correct, your want as high test coverage as possible. This implies that you’ll have a lot of tests – tests that first must be coded and then run again and again. What, in turn, implies that they must be easy to code and fast.

Avoiding instantiation of a vast trees of chained classes in each test, and avoiding time consuming connections to external resources (DB, web services, file system etc.) is exactly what makes tests fast. Isolation from external resources also simplifies tests’ setup (no need to load fixtures, clean up files, read configuration etc.) what further improves tests’ speed.

Full isolation also makes coding tests easier. The complexity of a test (possible combinations of input parameters, boundary conditions etc.) depends exponentially on number of interrelated classes participating in such test. Narrowing all tests to a single class, without dependencies, can lower number of tests cases needed to thoroughly test a system by an order of magnitude. Additionally, cutting out all dependencies to external resources, greatly lowers effort required to prepare test environment.

The second goal is an application that is easy to modify.

Application evolves over time. Requirements change. Design changes. Code is refactored. This is especially emphasized if you’re into XP-like emergent design and constant, aggressive refactoring. To make such continuous evolution possible, your code and tests can’t be brittle nor time consuming to change.

The key here is high cohesion and low coupling. Change of any single concept in your application should require modification of only single class’ code (or a very narrow group of classes). This allows you to modify your application with a relatively low effort. Also, change in one of your classes shouldn’t break tests for any other classes – this allows you to modify your application in a controlled way (and also reduces effort).

Full isolation of your unit tests is a big step towards achieving such modular design: if you’re able to thoroughly test a class without touching any of its dependencies, you usually should also be able to modify this class without having to modify or breaking tests for any other classes.

How many unit tests should I have?

As many as possible. Except of maybe plain getters and setters without any additional logic, you should test every part of public interface of every class. You should take into account all possible edge cases, boundary conditions etc.

Focused Integration Tests

Tests of this kind check connections between components and an external resources (DB, web service etc.). The characteristic attribute of focused integration tests is testing only a single connection. So, for a class with connections to many external resources there should be separate focused integration test for each such dependency.

The goal is an application talking correctly with its environment.

Unit tests guarantee that your classes internally work as expected. They verify that your class properly parses config file contents or correctly applies business logic to data received from a web service. However, can you be sure that this config file is opened properly or that a web service connection request is correctly formatted? Are you sure this class will be able to talk to a real service or file system? That’s what focused integration tests verify.

Integration tests focus on external connections, which are slow. This implies that they should be limited in scope and shouldn’t test business logic.

Limiting each integration test to a single connection helps avoiding exponential growth of complexity in a similar way like full isolation does in case of unit tests. Avoiding testing business logic (which already should be thoroughly tested by unit tests) and limiting integration test to checking only the connection itself, reduces the number of calls that must be made to an external resource in your tests, making your test suite faster.

How many focused integration tests should I have?

In general, as many as external connections you code has. In reality the number of tests may be higher, as some connections may have several different states you need to handle (e.g. a normal, successful call to a web service and a time-outed call). However, you should strive to keep numbers of external connection to a minimum: e.g. if more than one of your classes needs to read a config file, you should centralize file access in a single config parser class, and create focused integration tests only for this single class.

End-To-End Tests

These tests exercise all the application components – UI, business logic and connections to external  resources (DB, web services etc.) – together, in a single test. The characteristic attribute of end-to-end test is a black-box approach. They should exercise an application from an end user perspective, providing input and receiving output only via application GUI, without accessing any of its internals.

The goal is an application correctly glued together.

All pieces of your application are already well tested: every piece of business logic is covered by unit tests and every resource connection by focused integration tests. But you also need to be sure that all the pieces will collaborate together as expected. That’s what end-to-end tests are for. They should check that the whole application, from UI to external resources, is correctly composed of its building blocks: that all dependencies and configuration are correctly set, that input and output data flow properly up and down through the whole application stack and so on.

This implies that end-to-end tests are the only kind of tests that should not strive for isolation. It is exactly the opposite – the sole purpose of end-to-end tests is to exercise all applications components together at the same time. That’s why such tests should rather access application “from outside”, via tools like Webrat or Watir, than to play with its internals. It implies also that they are even slower than focused integration tests, so they too shouldn’t test business logic nor boundary conditions.

How many end-to-end tests should I have?

As few as possible to ensure that the application is glued correctly. Because end-to-end tests access application via its GUI, they are the slowest and the most brittle of all kinds of tests. Therefore, they should check only primary paths through an application (plus maybe a few of most important error scenarios) leaving all edge cases to other, more focused tests.

Customer Tests

These tests are technically very similar to end-to-end tests. They also exercise the complete application from a black-box perspective, working only through its GUI. However, their most characteristic attribute is that they are created in close cooperation with customers or sometimes even directly by customers. That’s because – despite technical similarities – they have a different goal than end-to-end tests.

The goal is code which does what user expects.

All kinds of tests mentioned before verify programmer’s expectations. They ensure technical correctness. However, the fact that an applications is build from bug-free components, can talk with its environment and is correctly glued together doesn’t automatically mean that it does what its users want. Therefore, we need an additional kind of tests, designed to clarify requirements. This implies that these tests should exercise an application only through its GUI and be written in a language understood by a customer, not in technical lingo.

The reason for testing only via GUI is a bit different than in case of end-to-end tests. End-to-end tests do it because they must exercise the whole application to check if all layers collaborate correctly. The customer tests do it because they verify user requirements, and user, not being involved in design or coding, can have requirements only regarding the application’s external interface – she doesn’t care how the application works internally, she cares only what visible results it produces.

For similar reasons it is important to avoid technical jargon in customer tests. Requirements, from the customer point of view, are centered around actions she can accomplish with an application not around low level steps needed to perform these actions nor underlying technology. Therefore, they should use vocabulary like “sign-in”, “set meeting date”, “print an invoice”, not “fill in login and password input fields”, “choose a value from a date picker widget” or “add an invoice to a background print queue” (which may be perfectly valid for other, programmer-centric test types).

How many customer tests should I have?

No more than necessary to understand how the application should work. Keep in mind that customer tests are not acceptance tests and therefore shouldn’t cover all usage scenarios. Customer tests are meant to work as live examples, clarifying most complicated parts of an application logic where needed. If the application domain is simple or well understood by programmers it is perfectly valid to have only a few (or even none) customer tests.

Comments (1)

Java Script mock frameworks comparison

In this short series I’ve already looked at xUnit and BDD style testing frameworks for JS. As mocking is a fundamental part of testing and only two of JS frameworks (Jasmine and JSpec) have mock frameworks built-in, now it’s turn to look at standalone mock frameworks.

Jasmine’s and JSpec’s built-in solutions will be also tested here to see how they compare to their stand-alone counterparts.

Two flavors of mock frameworks

There are two approaches to verifying expectations when using mock objects as spies:

  • setting expectations before tested code is run
    expect(Klass.method).toBeCalledWith('foo argument');
    Klass.method('foo argument');
    verify();
    
  • running tested code first and specifying assertions afterwards
    spyOn(Klass, 'method');
    Klass.method('foo argument');
    expect(Klass.method).wasCalledWith('foo argument');
    

The difference between both methods is mostly cosmetic, although I prefer second one (assertions after the code is run) for two reasons:

  • it is more consistent with the standard flow of test code – you normally invoke tested class’ methods first and then use “assert” statements on their return values, not the other way round.
  • it makes moving parts of mock / class initialization code into “before_each” easier, thus allowing you to keep your test code more DRY.

JsMockito
http://jsmockito.org

FLAVOR

Expectations after tested code.

PROS

  • Easy integration with many testing frameworks.
    JsMockito provides special API calls for flawless integration with ScrewUnit, JsTestDriver, JsUnitTest, YUITest, QUnit and JsUnity.
  • Ability to mock both object methods and functions.
  • Correctly handles expectations set on tested code using “call” and “apply”.
  • Robust input arguments matchers.

CONS

  • Almost none. The only con I have is that it depends on jsHamcrest for input parameter matching instead of being fully self-enclosed, but this is really very minor one.

CONCLUSION

A very solid framework. If your testing framework of choice doesn’t have built-in mock framework, JsMockito is definitely your best option.

jqMock
http://code.google.com/p/jqmock

FLAVOR

Expectations first, before tested code.

PROS

  • Allows mocking native methods (e.g. window.alert).
  • Extends jqUnit with powerful set of additional assertions (that can be used for matching input parameters in mocked methods as well as standalone).

CONS

  • Works only with jqUnit.

CONCLUSION

JqMock is quite robust framework. However, it it designed especially for jqUnit (a wrapper for QUnit library, which, although quite popular, is rather far from being my framework of choice). So, if you use jqUnit, jqMock is a decent and quite obvious choice. However, if you don’t use jqUnit (just like me) you’re out of luck (and jqMock is not that good to justify switching to jqUnit especially for it).

Jasmine’s built-in mock framework
http://github.com/pivotal/jasmine

FLAVOR

Expectations after tested code.

PROS

  • “callThrough” option.
    Ability to spy on a method (so number of calls, passed parameters etc. can be verified as usual) but passing the call to (and get the returned value from) the original method afterwards.
  • “callFake” option.
    Ability to specify a custom callback function to be invoked when the spy is called (awesome for e.g. mocking jQuery AJAX calls) .
  • Additional properties for spys (“callCount”, “mostRecentCall”, “argsForCall”) helpful for making assertions after mocked method was called.

CONS

  • Works only in Jasmine.

CONCLUSION

Definitely my mock framework of choice. Especially the “callFake” method is an icing on a tasty cake. The major drawback is that it is built into Jasmine – so if you use other framework you are unfortunately out of luck. But hey, as you can read in my BDD JS frameworks comparison, you have no good excuse to not use Jasmine in the first place.

JSpec’s built-in mock framework
http://visionmedia.github.com/jspec

FLAVOR

Expectations first, before tested code.

PROS

  • Mocking AJAX requests.
  • Mocking timers (“setTimeout” and similar methods).
  • No need to explicitly call “verify” at the end of a test.

CONS

  • Works only in JSpec.

CONCLUSION

Quite solid framework with a couple of interesting features, although not so polished as Jasmine’s. If you’re using JSpec, there is no reason to replace it with other, external framework, but there’s also no reason to switch to JSpec just for it’s mocking capabilities.

Other Frameworks

  • JSMock (http://jsmock.sourceforge.net)
    It’s mature (current version is 1.2.1) and the basic features expected from a mock framework are in, however the documentation is lacking (there are a few basic examples, but I couldn’t find API docs anywhere) and it is quite stale (the newest version is from May 2007) so it is not worth investing your time.
  • Amok (http://code.google.com/p/amok)
    It seems quite interesting, with some nice features like ability to mock AJAX calls and decent documentation (although a bit lacking in the area of aforementioned AJAX mocking). However, current version is 0.3 and was not updated for almost 18 months. On the other hand, documentation was updated much more recently (about 3 months ago). So my final verdict it to avoid it for now, with option to check in future if this project is live again.
  • Jack (http://boss.bekk.no/display/BOSS/Jack)
    The latest version is 0.0.2-alpha and not updated for over a  year. Probably an abandoned project.
  • JJ (http://github.com/nakajima/jj)
    It is still a work in progress. Right now it is practically only a basic stubbing framework, missing fundamental features like mocks / spies. Not really usable yet.

The Final Verdict

To make it short: my favorite framework is Jasmine’s built-in one. It is as good and polished as the rest of this top-notch framework. If you are just selecting JS testing framework (or are able to switch) Jasmine is your best choice.

If you are using either jqUnit or JSpec and are not willing to switch, use jqMock or JSpec’s built-in one, respectively, as there is no strong reason to substitute them with an external solution.

If you’re using any other xUnit or BDD framework (especially JsTestDriver, YUITest or ScrewUnit) your top choice is, hands down, jsMockito.

Comments (3)

Java Script BDD style frameworks comparison

In the previous installment of the series I’ve looked at xUnit style frameworks. In this one, I’ll compare BDD style frameworks.

JSpec
http://visionmedia.github.com/jspec

JSpec, influenced strongly by Ruby’s RSpec, takes a very interesting route: test cases are not written in plain Java Script, but rather in a custom DSL, closely resembling RSPec syntax (e.g. parentheses are optional).

SYNTAX

With a custom grammar:

describe 'ShoppingCart'
  before_each
    cart = new ShoppingCart
  end

  describe 'addProducts'
    it 'should add several products'
      cart.addProduct('cookie')
      cart.addProduct('icecream')
      cart.should.have 2, 'products'
    end
  end

  describe 'checkout'
    it 'should throw an error when checking out with no products'
      -{ cart.clear().checkout() }.should.throw_error EmptyCart
    end
  end
end

Alternatively in plain Java Script:

JSpec.describe('ShoppingCart', function(){
  before_each(function{
    cart = new ShoppingCart
  })

  describe('addProducts', function(){
    it ('should add several products', function(){
      cart.addProducts('cookie')
      cart.addProducts('icecream')
      expect(cart).to(have, 2, 'products')
    })
  })

  describe('checkout', function(){
    it ('should throw an error when checking out with no products', function(){
      expect(function(){ cart.clear().checkout() }).to(throw_error, EmptyCart)
    })
  })
})

PROS

  • RSPec like DSL.
    It gives you a very clean and concise syntax, more readable than normally possible in JS (you’ll love it, especially if you also code in Ruby).
  • Support for shared behaviors.
  • Pretty HTML runner.
  • Very robust server-based runner.
    Allows aggregating test results from several browsers at once. Support also non-browser platforms like Rhino or Env.js. Very nice for Continuous Integration.
  • Built-in mock framework.
  • Support for mocking AJAX requests.
  • Support for mocking timers.
  • HTML fixtures.
  • Nested describes.
  • Very robust set of matchers (including matchers for jQuery).
  • Ruby / Rails integration (with auto-test like functionality: automatic tests invocation when source file changes).
  • Extensible via modules and hooks.

CONS

  • RSPec like DSL.
    Unfortunately, pretty custom syntax comes at a cost. Test code must first go through a custom parser, which tends to break sometimes (e.g. embedding JS function declaration in test cases is tricky). It is no major problem – it happens only in some edge cases – but be prepared for an extensive debugging from time to time. Also, custom grammar doesn’t play well with syntax coloring in many IDEs, which may make test using custom grammar actually less readable, not more.
    It is possible to alternatively use regular Java Script syntax instead (what makes it IDE coloring friendly), but still the code goes through the parser, so all the problems with mis-interpreted syntax remain.
  • Shared behaviors implementation.
    JSpec provides its own mechanism (similar to the one used in RSpec) for sharing behaviors among many specifications.  However, its implementation is a bit awkward – shared behavior is not only used for sharing, but it is also automatically invoked as a regular, standalone behavior, what makes using it as a pure virtual set of tests rather awkward and error prone.
  • A little overcomplicated extensibility.
    Instead of using plain Java Script code for implementing helpers, you must use special module syntax. This again is the result of using custom grammar, which doesn’t allow you to intermix plain JS code with JSpec code as freely as you want.

CONCLUSION

JSpec is a very solid platform. It is probably the most feature packed framework on the market. However, the custom Ruby-like grammar is a mixed blessing – it gives you a very nice syntax, but unfortunately not without the cost. Anyway, this is a really good framework, one of the best available – and definitely recommended.

Jasmine
http://github.com/pivotal/jasmine

Jasmine is another RSpec like framework, created by authors of jsUnit and Screw.Unit. It takes a bit different approach than JSpec though, utilizing Java Script syntax to the maximum instead of trying to mimic Ruby.

SYNTAX

describe('some suite', function () {
  var suiteWideFoo;

  beforeEach(function () {
    suiteWideFoo = 0;
  });

  afterEach(function () {
    //...
  });

  describe('some nested suite', function() {
    var nestedSuiteBar;

    beforeEach(function() {
      nestedSuiteBar=1;
    });

    afterEach(function() {
      //...
    });

    it('nested expectation', function () {
      expect(suiteWideFoo).toEqual(0);
      expect(nestedSuiteBar).toEqual(1);
    });
  });

  it('top-level describe', function () {
    expect(suiteWideFoo).toEqual(0);
    expect(nestedSuiteBar).toEqual(undefined);
  });
});

PROS

  • Very clean syntax with well thought scope handling.
    And the side effect for it is that many interesting features just work, in plain Java Script, without need to use special API etc. E.g. you can create shared behaviors just by encapsulating any part of your spec (tests only, or the whole describe clause, with tests, beforeEach filters etc.) in a JS function, and then calling that function from anywhere inside your describe block, without need to use any fancy syntax for shared behavior declaration. The same goes for helper methods etc.
  • Built-in top-notch mocking framework.
  • Support for asynchronous tests.
  • Nested describes.
  • Very nice HTML runner.
  • Good support for Continuous Integration.
    Either via jasmine-ruby (http://github.com/pivotal/jasmine-ruby) or via integration with jsTestDriver (http://code.google.com/p/js-test-driver/wiki/XUnitCompatibility)

CONS

  • From the version number (current one is 0.10.3) it seems it may be a bit young. However, it is feature packed and very stable, so this con is probably over-defensive.

CONCLUSION

This is my favorite JS testing framework. I don’t see practically any cons. Definitely recommended.

JSSpec

http://jania.pe.kr/aw/moin.cgi/JSSpec

This is one of the first BDD style frameworks for Java Script. Therefore, it gained some traction, being used by such projects like e.g. MooTools, but nowadays it is starting to show its age.

SYNTAX

describe('Plus operation', {
    before_each : function() {
        // ...
    },

    'should concatenate two strings': function() {
        value_of("Hello " + "World").should_be("Hello World");
    },

    'should add two numbers': function() {
        value_of(1 + 2).should_be(3);
    },

    after_each : function() {
        // ...
    }
});

PROS

  • HTML runner.
  • Conditional execution for different browsers.
    There’s an interesting feature that lets you embed special condition clauses in your test names that tell the framework to run given test only in specified browsers (although I have some doubts about using it, as I think my JS code should be rather portable, and the exact same set of test should pass cross-browser).

CONS

  • A bit awkward syntax.
    The framework uses “value_of” instead of commonly established “expect” and tests names are specified directly as function names instead of an established “it” method with a string parameter.
  • No nested describes.
  • No support for Continuous Integration.
  • Not updated for some time – it doesn’t seem to be actively maintained any more.

CONCLUSION

It is decent and usable framework, but aged, lacking in features when compared to its younger counterparts. Its very basic capabilities, together with a bit awkward syntax, not matching established modern standards, doesn’t make it a compelling choice. Not recommended.

Other Frameworks

  • Screw.Unit (http://github.com/nkallen/screw-unit)
    A quite decent, mature framework. Maybe there are no exceptional features that distinguish it from the crowd, but anyway it is a solid contender. However, it seems to be abandoned in favor of Jasmine framework, so most probably you cannot expect active development of Screw.Unit any more.
  • Inspec (http://github.com/aq1018/inspec)
    This framework seems to be at an early stage. Documentation is practically nonexistent. Even its own test suite (to which author sends back for more examples) is fairly incomplete. Not usable in production for now.
  • jShoulda (http://jshoulda.scriptia.net)
    It is officially abandoned by the author in favor of SugarTest.
  • SugarTest (http://sugartest.scriptia.net)
    It is immature still (current version is 0.2, feature set is relatively basic) and it doesn’t seem to be very actively developed (most recent commit is about a year old).

The Final Verdict

It is quite easy to choose top two contenders, as both JSpec and Jasmine clearly stand out from the crowd. It is much harder though to select between these two. Both are mature, actively supported and robust, and choke-full of very cool features.

JSpec seems more feature complete, however its custom grammar becomes problematic at times. On the other hand, Jasmine’s clear syntax and scoping, allow to easily re-create many of missing features (e.g. shared behaviors or helpers can be implemented just as plain JS functions), enabling at the same time rock-solid test code that doesn’t need to rely on the whims of custom grammar parser. Also, Jasmine offers no half-baked features. If it is in, it is polished (a good example of this is its built-in mocking framework, which is probably the most robust one on the market).

My final decision? Jasmine.

It is a real joy to work with this framework.

Comments (4)

Java Script xUnit style frameworks comparison

In this installment of the series I’ll look at xUnit frameworks. Comparison of BDD frameworks and standalone mock frameworks will follow in subsequent posts.

JsTestDriver
http://code.google.com/p/js-test-driver

This one is a very strange beast. Its main focus, as claimed by authors, is command line and continuous integration – you can think of jsTestDriver more as a robust test runner platform rather than a typical assertion framework. This is very visible in its design – its client-server architecture based runner is as robust as it may get, definitely one of (or the) best on the market, while assertion framewrok, although solid, is relatively bare bone.

SYNTAX

TestCase("MyTestCase", {
    setUp: function() {
        //...
    },

    testAssertionWithMessage: function() {
        assertTrue("this is not true", false);
    },

    testAssertionWithoutMessage: function() {
        var actual = 5;
        assertEqual(5, actual);
    },

    tearDown: function() {
        //...
    }
};

PROS

  • Robust Continuous Integration support: parallel execution in many browsers at once (both local and remote) with aggregated result reporting; designed with speed in mind (keeps everything permanently loaded in browser, reloads only files that changed – what gives a huge speed boost by avoiding frequent re-parsing of code)
  • Allows replacing its assertion layer with another framework. This way jsTestDriver acts only as a test runner while actual test suites are written using alternative syntax. This offers possibility to replace not so pretty jsTestDriver syntax with something more fancy (even changing its flavor from xUnit to BDD in case of integrating with Jasmine) and also makes porting your existing code to jsTestDriver a breeze if you are lucky to use one of supported frameworks. Currently though, only a few integrations are supported (as of time of this writing only QUnit, YUI Test and Jasmine).
  • Integrates with Eclipse and IntelliJ IDEA. If you are using one of these, it may compensate for the lack of HTML runner (bad luck for NetBeans users though).
  • Automatically resets DOM between tests. Also provides some support for HTML fixture loading (although rather very basic, based on comments: if you have HTML inside a comment block in your test, it will get automatically loaded into DOM).
  • No need to create any HTML wrappers to run tests.
  • There is a code coverage plugin for jsTestDriver available (http://code.google.com/p/js-test-driver/wiki/CodeCoverage).

CONS

  • No HTML runner. If you happen to use IDE that is not supported by jsTestDriver, you must read test results from the console, with its ugly and hard to read formatting of error messages.
  • A bit over-complicated to run on a developer machine. For CI this is no problem, but having to keep open two consoles (one to control a server, second to display test results) plus one or more browsers on a local workstation is a little overkill for me.

CONCLUSION

This is a really solid framework, with simple, but nice and clean syntax and a couple of unique features. And although its CI oriented client-server architecture may feel too heavy for some usage scenarios, it is a real deal for bigger, more complex projects. Among xUnit style frameworks this is definitely one of the best choices.

QUnit
http://docs.jquery.com/QUnit

This is an “unofficial official” testing framework for jQuery. It’s mature and quite established on the market (also partly due to the popularity of jQuery). It is rather simplistic, containing by design only most core set of features (with two notable exceptions: support for HTML fixtures and testing of asynchronous calls).

SYNTAX

test("a basic test without module", function() {
    var value = "hello";
    equals("hello", value, "We expect value to be hello");
});

module("Some Module", {
    setup: function() {
        //...
    },
    teardown: function() {
        //...
    }
});

test("first test within module", function() {
    ok(true, "all pass");
});

test("second test within module", function() {
    ok( true, "all pass" );
});

PROS

  • HTML runner with nice output formatting.
  • Automatic clean-up of DOM between tests. Some support for HTML fixtures – content of predefined DIV, if present inside runner’s page HTML, is re-created between tests (although this is not fully clean when you run multiple suites at once – all fixtures for all suites must be cluttered in HTML in such case, instead of being stored separately and loaded on demand).
  • Support for testing asynchronous code.
  • Integrates with jsTestDriver.

CONS

  • Non standard (and non intuitive) syntax. Test methods are placed outside, instead of inside of module. Assertion have very custom naming (e.g. “ok” instead of “assertTrue”). Surprising “expect” method (setting expectations at the beginning of a test on how many calls to other assertions will be made in this test).
  • Absolutely barebone set of assertions (only 3 assertions available: “ok”, “equals” and “same”).
  • No command line runner (and no support for CI).
  • Tests pollute global namespace.

CONCLUSION

Being very simple and being a standard platform for jQuery core tests may seem tempting at first, but this framework is so simplistic, and has such a strange, custom syntax, that I’ll not recommend it. Built-in support for HTML fixtures is a plus, but the way it is implemented makes it useful only for a small test suites, so this doesn’t change the final verdict.

jqUnit
http://code.google.com/p/jqunit

jqUnit is just a wrapper around QUnit framework. It doesn’t add any new features, only wraps the QUnit with a different, JSUnit compatible syntax.

PROS

  • The same as for QUnit (although I’m not sure about jsTestDriver integration).
  • Fixes a bit “weird” QUnit syntax. Allows easy porting of your old JsUnit code.

CONS

  • The same as for QUnit.

CONCLUSION

If you really need to port your huge code base from JsUnit, this may be an interesting choice. Otherwise it is not recommended.

YUI Test
http://developer.yahoo.com/yui/yuitest

YUI Test is a part of Yahoo YUI library. It is mature and very actively maintained (as well as YUI library itself). Although it uses YUI library namespace, it can be used also standalone.

SYNTAX

var oTestCase = new YAHOO.tool.TestCase({
    name: "TestCase Name",

    // Special instructions
    _should: {
        ignore: {
            testSomethingUnimportant: true //ignore this test
        },

        error: {
            testSomethingExceptional: true //this should throw an error
        }
    },

    setUp : function () {
        //...
    },

    tearDown : function () {
        //...
    },

    testSomething: function () {
        YAHOO.util.Assert.areEqual("Something", this.someValue, "Value should be 'Something'");
    },

    testSomethingUnimportant: function () {
        //this test is ignored
    },

    testSomethingExceptional: function () {
        methodThatThrowsException();
    }
});

PROS

  • Robust set of assertions.
  • Built-in mocking of mouse and keyboard events.
  • Supports asynchronous tests.
  • Not only test cases but also test suites can have setUp and tearDown methods – this allows almost BDD like nesting (however with not so nice syntax as in case of BDD frameworks).
  • Integrates with jsTestDriver.

CONS

  • Very ugly HTML runner. It tries to emulate the look of a console – with all its readability problems. (However, YUI Test runner provides very robust support for test result serialization, with option to pass serialized results to a given URL, so it should be possible to create your own pretty printer).
  • No command line runner. (Although, again – robust support for test result serialization should allow you to code your own support for CI / command line invocation).
  • YUI related namespacing. Having to write “YAHOO.util.Assert.areEqual” instead of just “assertEqual” is awkward. It may be mitigated a little though, by doing something like:
    var Assert = YAHOO.util.Assert;
    Assert.areEqual(...);
    
  • Strange, non standard “_should” syntax for “special” instructions (like ignoring tests or expecting exceptions to be thrown).
  • No support for HTML fixtures.

CONCLUSION

YUI Test is a very solid framework. If the lack of robust Continuous Integration support is not a show stopper for you, it is definitely worth considering. A couple of minor syntax annoyances are a small crack on the surface of otherwise polished product, however they are far from being serious enough to discourage you from using it.

JsUnit
http://www.jsunit.net

One of the first, most “traditional” xUnit type frameworks. It is still actively maintained, but it’s starting to show its age.

SYNTAX

function setUp() {
    //...
}

function testAddition() {
    assertEquals("2 plus 3 is 5", 5, add(2, 3));
}

function tearDown() {
    //...
}

PROS

  • Based on JUnit for Java, so it has “standard” syntax.
  • HTML runner.
  • There is some support for server-based testing in several browsers at once (via JUnit or Ant).

CONS

  • Tests methods not encapsulated in classes (pollute global namespace).
  • Very simplistic regarding features (no fixtures, no async testing etc.)

CONCLUSION

Nothing interesting here. This framework might be popular at a time, but now there are other, younger and much better contenders. Not recommended.

Other frameworks

  • jsUnity (http://jsunity.com)
    Seems quite promising, but for now it is a bit too young (as the time of this writing current version is 0.6). It has a clean, readable syntax with a nice set of assertions, but apart from this it is still in a very basic shape: no advanced runners, no CI integration etc. – just a minimal set of  functionality you can expect from a testing framework. On the one hand, there were several updates to this framework, on the other, the newest one is about a year old – so it is difficult to tell if this project is still actively maintained. If so, it may be worth looking at in future.
  • JsUnit (berlios) (http://jsunit.berlios.de)
    Not updated for ages… Probably no more actively maintained.
  • JsUnitTest (http://jsunittest.com)
    Never reached full maturity (latest version is 0.7.3, documentation is sparse) while at the same time not updated for a long time – seems an abandoned project.
  • J3Unit (http://j3unit.sourceforge.net)
    Declared to be in beta stage, but last update was 4 years ago… Seems abandoned.

The Final Verdict

There are two xUnit frameworks for Java Script that stand out from the crowd: jsTestDriver and YUI Test. YUI Test has a more feature packed assertion framework (with user input events mocking, async tests etc.) while jsTestDriver distinguishes itself with its awesome support for multi-browser server-based runner.

The final verdict?

JsTestDriver.

While YUI Test features may seem attractive, most of them is not so hard to mimic by hand, especially when using jQuery or a similar JS framework (e.g. simulating mouse and keyboard events in jQuery is dumb simple). On the other hand, hand-coding jsTestDriver’s support for distributed, parallel testing is a completely different pair of boots. And it’s a must have on any reasonable complex project, especially when using CI.

QUnit also stands out from the crowd in one aspect – its very neat and readable HTML test runner, which may be quite tempting. However, lack in other areas make it suitable only for small, simple projects.

It is worth noting that both QUnit and YUI Test integrations are supported by jsTestDriver – so you can have best of both words if you want – e.g. jsTestDriver’s runner with YUI Test’s robust set of assertions. However, none of xUnit frameworks provides enough advantages over standard jsTestDriver’s assertion framework to justify additional hassle – so, my final verdict is a plain jsTestDriver (at least as far as xUnit style frameworks go – BDD style frameworks are a completely different story).

Also, for a complete solution you need a mocking framework (not built-in any of xUnit frameworks).

Standalone mocking frameworks, as well as BDD flavor frameworks, will be compared in detail in next installments of this series.

Comments (6)

Older Posts »
Follow

Get every new post delivered to your Inbox.