Archive for Java Script

Custom jQuery matchers in Jasmine

!!! THIS BLOG IS NO LONGER ACTIVE – CHECK OUT MY CURRENT BLOG AT VELESIN.IO !!!

Implementing your custom matchers (or custom assertions in case of xUnit-like frameworks) is a very good testing practice. In theory, as you’re verifying if a given expectation is correct, ‘.toBeTrue()’ is the only matcher you need. In practice however, using convoluted constructs like:

expect(someVariable != undefined).toBeTrue();

severely hinders readability of your tests. Thats why more clear and explicit matchers like:

expect(someVariable).toBeDefined();

come in handy. Such neat matchers are of course more comfortable to use, but their real power comes from revealing the intent of a test – what is one of primary goals (and strengths) of the whole Behavior Driven Development concept.

Different frameworks provide different sets of built in matchers. The one provided by Jasmine is relatively bare-bone, but it is not a bad thing, as it offers neat and robust mechanism for creating your own matchers when needed without polluting an API with a myriad of predefined matchers for all occasions (most of which you’ll never use). In this post I’ll show how to built a custom jQuery matcher library for Jasmine (provided by my jasmine-jquery plugin).

The simplest way to create a custom matcher, presented in Jasmine docs, is to use “.addMatchers()” method in your “beforeEach()” block:

beforeEach(function() {
  this.addMatchers({
    toHaveClass: function(className) {
      return this.actual.hasClass(className);
    }
  });
});

“this.actual” is automatically set by Jasmine to the variable on which we set expectations via “expect(…)” method. Other parameters for your matcher can be set as usual, as input parameters of your callback function (see “className” in above example). Your matcher function must return a boolean value indicating if it succeeded or not.

Now, when you have your first jQuery matcher defined, you can use it in your tests like this:

it('should have class "my-class"', function() {
  var myElement = $(<div class="my-class"></div>');
  expect(myElement).toHaveClass("my-class");
});

instead of much less readable:

it('should have class "my-class"', function() {
  var myElement = $('<div class="my-class"></div>');
  expect(myElement.hasClass("my-class")).toBeTruthy();
});

Jasmine also automatically allows your custom matchers to be inverted by prepending them with “.not”, so you can also use your freshly created jQuery matcher like this:

it('should have class "my-class"', function() {
  var myElement = $('<div class="my-class"></div>');
  expect(myElement).not.toHaveClass("other-class");
});

So far so good. Your tests are now much more readable and their intent is expressed more clearly. However, when written in such a way (placed in “beforeEach()”), your matchers are local to your test file. In more specific cases this may be not so bad, but in case of e.g. jQuery related matchers, we will want to re-use them in many tests across many projects, so making a local copy each time is definitely not an option.

The solution to the above problem is simple – as you could see, “addMatchers()” method takes a JS object as a parameter. So instead of declaring this object locally, lets move it to a separate file:

in my_jquery_matchers.js:

jQueryMatchers = {
  toHaveClass: function(className) {...},
  toBeVisible: function() {...},
  ...
};

in my_spec.js:

beforeEach(function() {
  this.addMatchers(jQueryMatchers);
});

Now the matchers are neatly separated into their own file, allowing you to reuse this “library” among may test files. And as Jasmine allows you to use “beforeEach()” at arbitrarily high level – not only in your “describe” block but also at a level global to your whole test file, or even at a level of the test runner itself (global for all test files executed by this runner), you need to invoke “addMatchers()” only once, so your tests stay reasonably DRY.

When creating custom matchers specific to your code, above solution is sufficient. However, a matcher library for an external framework like jQuery feels more like a part of the test framework core, not as a part of test suite. Therefore, two problems that don’t bother me that much in case of my custom matchers, make me not fully satisfied in case of jQuery matcher library. First, variable containing matchers (jQueryMatchers in the above example) pollutes global namespace. Second, having to instantiate a part of the core library inside one of your “beforeEach()” blocks is awkward – these blocks should rather contain code related to your SUT (system under tests), not a framework’s boilerplate.

So, let’s take it one step further and “hack” our jQuery matchers into Jasmine framework directly. The idea may seem scary at first, but it is really straightforward. Jasmine keeps all its core matchers in a separate object called “Matchers”. All you need to do is to add your new matchers directly to the “Matchers” object’s prototype instead of passing them via “addMatchers()” method:

jasmine.Matchers.prototype.toHaveClass: function(className) {
  return this.actual.hasClass(className);
};

Now the only thing you need to do is to include the file in a test runner (the same way you include Jasmine core or jQuery library) and our jQuery matchers are transparently available in all test files, without polluting the namespace or any of “beforeEach()” blocks. (I’d like to reiterate here that for matchers related to your custom SUT only, this may be a little overkill, but it is a neat and elegant solution when creating “core” matcher libraries, reused among many projects).

OK, we’re almost there with our solution, however, I’ll apply two more tweaks that will make our library more robust.

First annoyance to fix is the way Jasmine displays failed assertions for our jQuery matchers. Inside our matcher, this.actual is a jQuery element, and Jasmine by default can’t do anything smarter than dump it to string in an error message. So when you try:

var myElement = $(<div class="wrong-class"></div>');
expect(myElement).toHaveClass("some-class");

you’ll get:

Expected { 0 : HTMLNode, length : 1, context : HTMLNode, selector : '#sandbox', init : Function, jquery : '1.4.2', size : Function, toArray : Function, get : Function, pushStack : Function, each : Function,
...
a _LOT_ of similar jQuery element dump code here
...
, width : Function } to have class 'some-class'.

This makes an error message almost impossible to comprehend. However, it may be quite easily fixed by using a simple trick – modifying “actual” variable inside our matcher before returning:

jasmine.Matchers.prototype.toHaveClass: function(className) {
  var result = this.actual.hasClass(className);
  this.actual =
      $('<div />').append(this.actual.clone()).html();
  return result;
};

What we do with “this.actual” above, is conversion from jQuery element into a plain HTML string. This results in much more explanatory message, leaving no doubt why our test didn’t pass:

Expected '<div class="wrong-class"></div>' to have class 'some-class'.

The last problem left for us to solve is a situation when there is naming collision between our jQuery matchers and built-in Jasmine matchers. E.g. I’d like to have a matcher called “toContain()” that will check if a Query element contains another element, specified by a jQuery selector, but Jasmine already has a “toContain()” matcher that checks if an array contains specified item.

The simplest way to solve this issue is just to check what matcher names are already taken by Jasmine and to avoid potential collisions by renaming our jQuery matcher to e.g. “toContainElement()” or “toContainSelector()”; However, I’m quite stubborn and I want a simple “toContain()”. I’m also lazy, and I don’t want to check all built-in Jasmine matchers – and what’s even more important, I don’t want a naming collision when new matchers are added to Jasmine core in future.

Another simple trick will do it:

var builtInMatcher = jasmine.Matchers.prototype.toContain;

jasmine.Matchers.prototype.toContain = function() {
  if (this.actual instanceof jQuery) {
    // our jQuery matcher code here
    // return results
  }

  return builtInMatcher.apply(this, arguments);
};

If the tested variable is a jQuery element our jQuery matcher will be used, otherwise we pass the control to the original Jasmine matcher. Also, invoking original matcher by using “apply(this, arguments)” ensures that even if our overlapping jQuery matcher and original Jasmine matcher expect different number of arguments the above code will still work correctly.

And that’s all. The full, more polished implementation can be found in my jasmine-jquery library (it contains also support for HTML fixtures – see my previous post about fixtures in Jasmine for a more detailed overview).

Comments (3)

HTML fixtures in Jasmine (using jasmine-jquery)

!!! THIS BLOG IS NO LONGER ACTIVE – CHECK OUT MY CURRENT BLOG AT VELESIN.IO !!!

My comparison of JavaScript BDD frameworks resulted in choosing Jasmine to be my JS testing framework of choice. However, I’m doing a lot of jQuery based development, while Jasmine by design tries to stay DOM agnostic. It is not that big problem, as Jasmine is really flexible and open (I’ll show in a moment how to manipulate DOM with jQuery using plain-vanilla Jasmine), but with a couple of tweaks packed into my jasmine-jquery plugin, jQuery users can make their life even easier.

First, let’s start with the simplest approach possible to test jQuery code in your Jasmine specs. The most straightforward way is just to create a “fixture” jQuery element inside your spec, invoke your plugin on it and then set your expectations against the fixture element (to verify if the plugin modified fixture element correctly):

describe('simplest jQuery plugin test', function () {
  it('should use inline fixture', function () {
    var fixture = $('<div>some HTML code here</div>');
    fixture.myTestedJqueryPlugin();
    expect(fixture).toSomething();
  });
});

Of course, for any non trivial plugin you’ll need more than one spec to fully test its behavior, so the above example in reality will look more like this:

describe('more realistic jQuery plugin test', function () {
  var fixture;

  beforeEach(function () {
    fixture = $('<div>some HTML code here</div>');
  });

  it('should use inline fixture', function () {
    fixture.myTestedJqueryPlugin();
    expect(fixture).toSomething();
  });

  it('should use inline fixture again', function () {
    fixture.myTestedJqueryPlugin();
    expect(fixture).toSomethingElse();
  });
});

Above solution is still far from perfect though. One of the problems is that jQuery fixture element is created only in memory and never added to DOM. In effect, many tests (e.g. related to element visibility or dimensions) won’t work correctly. In order to solve this, we need to add our fixture to the DOM and, to avoid test interdependencies, we must clear the DOM between tests.

describe('DOM based jQuery plugin test', function () {
  beforeEach(function () {
    $('#fixture').remove();
    $('body').append('<div id="fixture">...</div>');
  });

  it('should use DOM fixture', function () {
    $('#fixture').myTestedJqueryPlugin();
    expect($('#fixture')).toSomething();
  });

  it('should use DOM fixture again', function () {
    $('#fixture').myTestedJqueryPlugin();
    expect($('#fixture')).toSomethingElse();
  });
});

Now it should work for all kinds of jQuery related tests, and the beforeEach doesn’t look that bad (only 2 lines of code), but unfortunately such a simple one-liner fixtures will rarely suffice for non trivial, real-life plugins. And believe me, inlining complex, multiline HTML strings in your test code is ugly, turning your well factored test code into one big tangled mess. Also, to keep your tests clean, you may want to reuse fixtures not only between tests in one file, but also between different test files. The answer here is loading fixtures from files. We can change our code to something like this:

describe('test with fixture files', function () {
  beforeEach(function () {
    $('#fixture').remove();
    $.ajax({
      async: false, // must be synchronous to guarantee that no tests are run before fixture is loaded
      dataType: 'html',
      url: 'my_fixture.html',
      success: function(data) {
        $('body').append($(data));
      }
    });
  });

  it('should use DOM fixture', function () {
    $('#fixture').myTestedJqueryPlugin();
    expect($('#fixture')).toSomething();
  });

  it('should use DOM fixture again', function () {
    $('#fixture').myTestedJqueryPlugin();
    expect($('#fixture')).toSomethingElse();
  });
});

Now your fixtures can be neatly organized into separate files and don’t pollute your test code. An added benefit is that they don’t have to be static – my_fixture.html file can be now e.g. generated via normal Rails action etc. However, beforeEach have grown a little. And this is just a tip of an iceberg. Right now the fixture file is reloaded via AJAX before each of your tests, which results in a major penalty to your test suite’s speed – some kind of fixture caching would be welcome. Also, what if you want to load more than one fixture? You’ll then have to duplicate fixture loading and DOM clean-up code. And what if for some reason you want to load fixture from external file, but don’t want to append it automatically to DOM, but rather assign to a variable inside your test? Again, this can be easily done, but it will add another variation to your beforeEach code. Even worse, you’ll probably use fixtures in more than one test file, so your fixture loading code will end-up copy pasted all over the place. As you can see this can quickly go out of control for any non trivial project.

jasmine-jquery to the rescue! ;)

It extends Jasmine framework with a robust set of methods to handle HTML fixtures. It supports also fixture caching to speed-up your test suite and auto clean-up of DOM between tests. With jasmine-jquery you can write your tests like this:

describe('test with jasmine-jquery', function () {
  it('should load many fixtures into DOM', function () {
    loadFixtures('my_fixture_1.html', 'my_fixture_2.html');
    expect($('#jasmine-fixtures')).toSomething();
  });

  it('should only return fixture', function () {
    var fixture = readFixtures('my_fixture_3.html');
    expect(fixture).toSomethingElse();
  });
});

And that’s all. Yes, there doesn’t have to be beforeEach at all. Fixtures are automatically appended to DOM by loadFixtures method and automatically cleaned-up between tests. Apart from loadFixtures and readFixtures jasmine-jquery provides other useful methods, so don’t forget to check jasmine-jquery GitHub page for the full documentation.

Apart of HTML fixtures support, jasmine-jquery provides also a robust set of Jasmine custom matchers for jQuery – I’ll present them in more detail in next post.

Comments (34)

Java Script mock frameworks comparison

!!! THIS BLOG IS NO LONGER ACTIVE – CHECK OUT MY CURRENT BLOG AT VELESIN.IO !!!

In this short series I’ve already looked at xUnit and BDD style testing frameworks for JS. As mocking is a fundamental part of testing and only two of JS frameworks (Jasmine and JSpec) have mock frameworks built-in, now it’s turn to look at standalone mock frameworks.

Jasmine’s and JSpec’s built-in solutions will be also tested here to see how they compare to their stand-alone counterparts.

Two flavors of mock frameworks

There are two approaches to verifying expectations when using mock objects as spies:

  • setting expectations before tested code is run
    expect(Klass.method).toBeCalledWith('foo argument');
    Klass.method('foo argument');
    verify();
    
  • running tested code first and specifying assertions afterwards
    spyOn(Klass, 'method');
    Klass.method('foo argument');
    expect(Klass.method).wasCalledWith('foo argument');
    

The difference between both methods is mostly cosmetic, although I prefer second one (assertions after the code is run) for two reasons:

  • it is more consistent with the standard flow of test code – you normally invoke tested class’ methods first and then use “assert” statements on their return values, not the other way round.
  • it makes moving parts of mock / class initialization code into “before_each” easier, thus allowing you to keep your test code more DRY.

JsMockito
http://jsmockito.org

FLAVOR

Expectations after tested code.

PROS

  • Easy integration with many testing frameworks.
    JsMockito provides special API calls for flawless integration with ScrewUnit, JsTestDriver, JsUnitTest, YUITest, QUnit and JsUnity.
  • Ability to mock both object methods and functions.
  • Correctly handles expectations set on tested code using “call” and “apply”.
  • Robust input arguments matchers.

CONS

  • Almost none. The only con I have is that it depends on jsHamcrest for input parameter matching instead of being fully self-enclosed, but this is really very minor one.

CONCLUSION

A very solid framework. If your testing framework of choice doesn’t have built-in mock framework, JsMockito is definitely your best option.

jqMock
http://code.google.com/p/jqmock

FLAVOR

Expectations first, before tested code.

PROS

  • Allows mocking native methods (e.g. window.alert).
  • Extends jqUnit with powerful set of additional assertions (that can be used for matching input parameters in mocked methods as well as standalone).

CONS

  • Works only with jqUnit.

CONCLUSION

JqMock is quite robust framework. However, it it designed especially for jqUnit (a wrapper for QUnit library, which, although quite popular, is rather far from being my framework of choice). So, if you use jqUnit, jqMock is a decent and quite obvious choice. However, if you don’t use jqUnit (just like me) you’re out of luck (and jqMock is not that good to justify switching to jqUnit especially for it).

Jasmine’s built-in mock framework
http://github.com/pivotal/jasmine

FLAVOR

Expectations after tested code.

PROS

  • “callThrough” option.
    Ability to spy on a method (so number of calls, passed parameters etc. can be verified as usual) but passing the call to (and get the returned value from) the original method afterwards.
  • “callFake” option.
    Ability to specify a custom callback function to be invoked when the spy is called (awesome for e.g. mocking jQuery AJAX calls) .
  • Additional properties for spys (“callCount”, “mostRecentCall”, “argsForCall”) helpful for making assertions after mocked method was called.

CONS

  • Works only in Jasmine.

CONCLUSION

Definitely my mock framework of choice. Especially the “callFake” method is an icing on a tasty cake. The major drawback is that it is built into Jasmine – so if you use other framework you are unfortunately out of luck. But hey, as you can read in my BDD JS frameworks comparison, you have no good excuse to not use Jasmine in the first place.

JSpec’s built-in mock framework
http://visionmedia.github.com/jspec

FLAVOR

Expectations first, before tested code.

PROS

  • Mocking AJAX requests.
  • Mocking timers (“setTimeout” and similar methods).
  • No need to explicitly call “verify” at the end of a test.

CONS

  • Works only in JSpec.

CONCLUSION

Quite solid framework with a couple of interesting features, although not so polished as Jasmine’s. If you’re using JSpec, there is no reason to replace it with other, external framework, but there’s also no reason to switch to JSpec just for it’s mocking capabilities.

Other Frameworks

  • JSMock (http://jsmock.sourceforge.net)
    It’s mature (current version is 1.2.1) and the basic features expected from a mock framework are in, however the documentation is lacking (there are a few basic examples, but I couldn’t find API docs anywhere) and it is quite stale (the newest version is from May 2007) so it is not worth investing your time.
  • Amok (http://code.google.com/p/amok)
    It seems quite interesting, with some nice features like ability to mock AJAX calls and decent documentation (although a bit lacking in the area of aforementioned AJAX mocking). However, current version is 0.3 and was not updated for almost 18 months. On the other hand, documentation was updated much more recently (about 3 months ago). So my final verdict it to avoid it for now, with option to check in future if this project is live again.
  • Jack (http://boss.bekk.no/display/BOSS/Jack)
    The latest version is 0.0.2-alpha and not updated for over a  year. Probably an abandoned project.
  • JJ (http://github.com/nakajima/jj)
    It is still a work in progress. Right now it is practically only a basic stubbing framework, missing fundamental features like mocks / spies. Not really usable yet.

The Final Verdict

To make it short: my favorite framework is Jasmine’s built-in one. It is as good and polished as the rest of this top-notch framework. If you are just selecting JS testing framework (or are able to switch) Jasmine is your best choice.

If you are using either jqUnit or JSpec and are not willing to switch, use jqMock or JSpec’s built-in one, respectively, as there is no strong reason to substitute them with an external solution.

If you’re using any other xUnit or BDD framework (especially JsTestDriver, YUITest or ScrewUnit) your top choice is, hands down, jsMockito.

Comments (3)

Java Script BDD style frameworks comparison

!!! THIS BLOG IS NO LONGER ACTIVE – CHECK OUT MY CURRENT BLOG AT VELESIN.IO !!!

In the previous installment of the series I’ve looked at xUnit style frameworks. In this one, I’ll compare BDD style frameworks.

JSpec
http://visionmedia.github.com/jspec

JSpec, influenced strongly by Ruby’s RSpec, takes a very interesting route: test cases are not written in plain Java Script, but rather in a custom DSL, closely resembling RSPec syntax (e.g. parentheses are optional).

SYNTAX

With a custom grammar:

describe 'ShoppingCart'
  before_each
    cart = new ShoppingCart
  end

  describe 'addProducts'
    it 'should add several products'
      cart.addProduct('cookie')
      cart.addProduct('icecream')
      cart.should.have 2, 'products'
    end
  end

  describe 'checkout'
    it 'should throw an error when checking out with no products'
      -{ cart.clear().checkout() }.should.throw_error EmptyCart
    end
  end
end

Alternatively in plain Java Script:

JSpec.describe('ShoppingCart', function(){
  before_each(function{
    cart = new ShoppingCart
  })

  describe('addProducts', function(){
    it ('should add several products', function(){
      cart.addProducts('cookie')
      cart.addProducts('icecream')
      expect(cart).to(have, 2, 'products')
    })
  })

  describe('checkout', function(){
    it ('should throw an error when checking out with no products', function(){
      expect(function(){ cart.clear().checkout() }).to(throw_error, EmptyCart)
    })
  })
})

PROS

  • RSPec like DSL.
    It gives you a very clean and concise syntax, more readable than normally possible in JS (you’ll love it, especially if you also code in Ruby).
  • Support for shared behaviors.
  • Pretty HTML runner.
  • Very robust server-based runner.
    Allows aggregating test results from several browsers at once. Support also non-browser platforms like Rhino or Env.js. Very nice for Continuous Integration.
  • Built-in mock framework.
  • Support for mocking AJAX requests.
  • Support for mocking timers.
  • HTML fixtures.
  • Nested describes.
  • Very robust set of matchers (including matchers for jQuery).
  • Ruby / Rails integration (with auto-test like functionality: automatic tests invocation when source file changes).
  • Extensible via modules and hooks.

CONS

  • RSPec like DSL.
    Unfortunately, pretty custom syntax comes at a cost. Test code must first go through a custom parser, which tends to break sometimes (e.g. embedding JS function declaration in test cases is tricky). It is no major problem – it happens only in some edge cases – but be prepared for an extensive debugging from time to time. Also, custom grammar doesn’t play well with syntax coloring in many IDEs, which may make test using custom grammar actually less readable, not more.
    It is possible to alternatively use regular Java Script syntax instead (what makes it IDE coloring friendly), but still the code goes through the parser, so all the problems with mis-interpreted syntax remain.
  • Shared behaviors implementation.
    JSpec provides its own mechanism (similar to the one used in RSpec) for sharing behaviors among many specifications.  However, its implementation is a bit awkward – shared behavior is not only used for sharing, but it is also automatically invoked as a regular, standalone behavior, what makes using it as a pure virtual set of tests rather awkward and error prone.
  • A little overcomplicated extensibility.
    Instead of using plain Java Script code for implementing helpers, you must use special module syntax. This again is the result of using custom grammar, which doesn’t allow you to intermix plain JS code with JSpec code as freely as you want.

CONCLUSION

JSpec is a very solid platform. It is probably the most feature packed framework on the market. However, the custom Ruby-like grammar is a mixed blessing – it gives you a very nice syntax, but unfortunately not without the cost. Anyway, this is a really good framework, one of the best available – and definitely recommended.

Jasmine
http://github.com/pivotal/jasmine

Jasmine is another RSpec like framework, created by authors of jsUnit and Screw.Unit. It takes a bit different approach than JSpec though, utilizing Java Script syntax to the maximum instead of trying to mimic Ruby.

SYNTAX

describe('some suite', function () {
  var suiteWideFoo;

  beforeEach(function () {
    suiteWideFoo = 0;
  });

  afterEach(function () {
    //...
  });

  describe('some nested suite', function() {
    var nestedSuiteBar;

    beforeEach(function() {
      nestedSuiteBar=1;
    });

    afterEach(function() {
      //...
    });

    it('nested expectation', function () {
      expect(suiteWideFoo).toEqual(0);
      expect(nestedSuiteBar).toEqual(1);
    });
  });

  it('top-level describe', function () {
    expect(suiteWideFoo).toEqual(0);
    expect(nestedSuiteBar).toEqual(undefined);
  });
});

PROS

  • Very clean syntax with well thought scope handling.
    And the side effect for it is that many interesting features just work, in plain Java Script, without need to use special API etc. E.g. you can create shared behaviors just by encapsulating any part of your spec (tests only, or the whole describe clause, with tests, beforeEach filters etc.) in a JS function, and then calling that function from anywhere inside your describe block, without need to use any fancy syntax for shared behavior declaration. The same goes for helper methods etc.
  • Built-in top-notch mocking framework.
  • Support for asynchronous tests.
  • Nested describes.
  • Very nice HTML runner.
  • Good support for Continuous Integration.
    Either via jasmine-ruby (http://github.com/pivotal/jasmine-ruby) or via integration with jsTestDriver (http://code.google.com/p/js-test-driver/wiki/XUnitCompatibility)

CONS

  • From the version number (current one is 0.10.3) it seems it may be a bit young. However, it is feature packed and very stable, so this con is probably over-defensive.

CONCLUSION

This is my favorite JS testing framework. I don’t see practically any cons. Definitely recommended.

JSSpec

http://jania.pe.kr/aw/moin.cgi/JSSpec

This is one of the first BDD style frameworks for Java Script. Therefore, it gained some traction, being used by such projects like e.g. MooTools, but nowadays it is starting to show its age.

SYNTAX

describe('Plus operation', {
    before_each : function() {
        // ...
    },

    'should concatenate two strings': function() {
        value_of("Hello " + "World").should_be("Hello World");
    },

    'should add two numbers': function() {
        value_of(1 + 2).should_be(3);
    },

    after_each : function() {
        // ...
    }
});

PROS

  • HTML runner.
  • Conditional execution for different browsers.
    There’s an interesting feature that lets you embed special condition clauses in your test names that tell the framework to run given test only in specified browsers (although I have some doubts about using it, as I think my JS code should be rather portable, and the exact same set of test should pass cross-browser).

CONS

  • A bit awkward syntax.
    The framework uses “value_of” instead of commonly established “expect” and tests names are specified directly as function names instead of an established “it” method with a string parameter.
  • No nested describes.
  • No support for Continuous Integration.
  • Not updated for some time – it doesn’t seem to be actively maintained any more.

CONCLUSION

It is decent and usable framework, but aged, lacking in features when compared to its younger counterparts. Its very basic capabilities, together with a bit awkward syntax, not matching established modern standards, doesn’t make it a compelling choice. Not recommended.

Other Frameworks

  • Screw.Unit (http://github.com/nkallen/screw-unit)
    A quite decent, mature framework. Maybe there are no exceptional features that distinguish it from the crowd, but anyway it is a solid contender. However, it seems to be abandoned in favor of Jasmine framework, so most probably you cannot expect active development of Screw.Unit any more.
  • Inspec (http://github.com/aq1018/inspec)
    This framework seems to be at an early stage. Documentation is practically nonexistent. Even its own test suite (to which author sends back for more examples) is fairly incomplete. Not usable in production for now.
  • jShoulda (http://jshoulda.scriptia.net)
    It is officially abandoned by the author in favor of SugarTest.
  • SugarTest (http://sugartest.scriptia.net)
    It is immature still (current version is 0.2, feature set is relatively basic) and it doesn’t seem to be very actively developed (most recent commit is about a year old).

The Final Verdict

It is quite easy to choose top two contenders, as both JSpec and Jasmine clearly stand out from the crowd. It is much harder though to select between these two. Both are mature, actively supported and robust, and choke-full of very cool features.

JSpec seems more feature complete, however its custom grammar becomes problematic at times. On the other hand, Jasmine’s clear syntax and scoping, allow to easily re-create many of missing features (e.g. shared behaviors or helpers can be implemented just as plain JS functions), enabling at the same time rock-solid test code that doesn’t need to rely on the whims of custom grammar parser. Also, Jasmine offers no half-baked features. If it is in, it is polished (a good example of this is its built-in mocking framework, which is probably the most robust one on the market).

My final decision? Jasmine.

It is a real joy to work with this framework.

Comments (4)

Java Script xUnit style frameworks comparison

!!! THIS BLOG IS NO LONGER ACTIVE – CHECK OUT MY CURRENT BLOG AT VELESIN.IO !!!

In this installment of the series I’ll look at xUnit frameworks. Comparison of BDD frameworks and standalone mock frameworks will follow in subsequent posts.

JsTestDriver
http://code.google.com/p/js-test-driver

This one is a very strange beast. Its main focus, as claimed by authors, is command line and continuous integration – you can think of jsTestDriver more as a robust test runner platform rather than a typical assertion framework. This is very visible in its design – its client-server architecture based runner is as robust as it may get, definitely one of (or the) best on the market, while assertion framewrok, although solid, is relatively bare bone.

SYNTAX

TestCase("MyTestCase", {
    setUp: function() {
        //...
    },

    testAssertionWithMessage: function() {
        assertTrue("this is not true", false);
    },

    testAssertionWithoutMessage: function() {
        var actual = 5;
        assertEqual(5, actual);
    },

    tearDown: function() {
        //...
    }
};

PROS

  • Robust Continuous Integration support: parallel execution in many browsers at once (both local and remote) with aggregated result reporting; designed with speed in mind (keeps everything permanently loaded in browser, reloads only files that changed – what gives a huge speed boost by avoiding frequent re-parsing of code)
  • Allows replacing its assertion layer with another framework. This way jsTestDriver acts only as a test runner while actual test suites are written using alternative syntax. This offers possibility to replace not so pretty jsTestDriver syntax with something more fancy (even changing its flavor from xUnit to BDD in case of integrating with Jasmine) and also makes porting your existing code to jsTestDriver a breeze if you are lucky to use one of supported frameworks. Currently though, only a few integrations are supported (as of time of this writing only QUnit, YUI Test and Jasmine).
  • Integrates with Eclipse and IntelliJ IDEA. If you are using one of these, it may compensate for the lack of HTML runner (bad luck for NetBeans users though).
  • Automatically resets DOM between tests. Also provides some support for HTML fixture loading (although rather very basic, based on comments: if you have HTML inside a comment block in your test, it will get automatically loaded into DOM).
  • No need to create any HTML wrappers to run tests.
  • There is a code coverage plugin for jsTestDriver available (http://code.google.com/p/js-test-driver/wiki/CodeCoverage).

CONS

  • No HTML runner. If you happen to use IDE that is not supported by jsTestDriver, you must read test results from the console, with its ugly and hard to read formatting of error messages.
  • A bit over-complicated to run on a developer machine. For CI this is no problem, but having to keep open two consoles (one to control a server, second to display test results) plus one or more browsers on a local workstation is a little overkill for me.

CONCLUSION

This is a really solid framework, with simple, but nice and clean syntax and a couple of unique features. And although its CI oriented client-server architecture may feel too heavy for some usage scenarios, it is a real deal for bigger, more complex projects. Among xUnit style frameworks this is definitely one of the best choices.

QUnit
http://docs.jquery.com/QUnit

This is an “unofficial official” testing framework for jQuery. It’s mature and quite established on the market (also partly due to the popularity of jQuery). It is rather simplistic, containing by design only most core set of features (with two notable exceptions: support for HTML fixtures and testing of asynchronous calls).

SYNTAX

test("a basic test without module", function() {
    var value = "hello";
    equals("hello", value, "We expect value to be hello");
});

module("Some Module", {
    setup: function() {
        //...
    },
    teardown: function() {
        //...
    }
});

test("first test within module", function() {
    ok(true, "all pass");
});

test("second test within module", function() {
    ok( true, "all pass" );
});

PROS

  • HTML runner with nice output formatting.
  • Automatic clean-up of DOM between tests. Some support for HTML fixtures – content of predefined DIV, if present inside runner’s page HTML, is re-created between tests (although this is not fully clean when you run multiple suites at once – all fixtures for all suites must be cluttered in HTML in such case, instead of being stored separately and loaded on demand).
  • Support for testing asynchronous code.
  • Integrates with jsTestDriver.

CONS

  • Non standard (and non intuitive) syntax. Test methods are placed outside, instead of inside of module. Assertion have very custom naming (e.g. “ok” instead of “assertTrue”). Surprising “expect” method (setting expectations at the beginning of a test on how many calls to other assertions will be made in this test).
  • Absolutely barebone set of assertions (only 3 assertions available: “ok”, “equals” and “same”).
  • No command line runner (and no support for CI).
  • Tests pollute global namespace.

CONCLUSION

Being very simple and being a standard platform for jQuery core tests may seem tempting at first, but this framework is so simplistic, and has such a strange, custom syntax, that I’ll not recommend it. Built-in support for HTML fixtures is a plus, but the way it is implemented makes it useful only for a small test suites, so this doesn’t change the final verdict.

jqUnit
http://code.google.com/p/jqunit

jqUnit is just a wrapper around QUnit framework. It doesn’t add any new features, only wraps the QUnit with a different, JSUnit compatible syntax.

PROS

  • The same as for QUnit (although I’m not sure about jsTestDriver integration).
  • Fixes a bit “weird” QUnit syntax. Allows easy porting of your old JsUnit code.

CONS

  • The same as for QUnit.

CONCLUSION

If you really need to port your huge code base from JsUnit, this may be an interesting choice. Otherwise it is not recommended.

YUI Test
http://developer.yahoo.com/yui/yuitest

YUI Test is a part of Yahoo YUI library. It is mature and very actively maintained (as well as YUI library itself). Although it uses YUI library namespace, it can be used also standalone.

SYNTAX

var oTestCase = new YAHOO.tool.TestCase({
    name: "TestCase Name",

    // Special instructions
    _should: {
        ignore: {
            testSomethingUnimportant: true //ignore this test
        },

        error: {
            testSomethingExceptional: true //this should throw an error
        }
    },

    setUp : function () {
        //...
    },

    tearDown : function () {
        //...
    },

    testSomething: function () {
        YAHOO.util.Assert.areEqual("Something", this.someValue, "Value should be 'Something'");
    },

    testSomethingUnimportant: function () {
        //this test is ignored
    },

    testSomethingExceptional: function () {
        methodThatThrowsException();
    }
});

PROS

  • Robust set of assertions.
  • Built-in mocking of mouse and keyboard events.
  • Supports asynchronous tests.
  • Not only test cases but also test suites can have setUp and tearDown methods – this allows almost BDD like nesting (however with not so nice syntax as in case of BDD frameworks).
  • Integrates with jsTestDriver.

CONS

  • Very ugly HTML runner. It tries to emulate the look of a console – with all its readability problems. (However, YUI Test runner provides very robust support for test result serialization, with option to pass serialized results to a given URL, so it should be possible to create your own pretty printer).
  • No command line runner. (Although, again – robust support for test result serialization should allow you to code your own support for CI / command line invocation).
  • YUI related namespacing. Having to write “YAHOO.util.Assert.areEqual” instead of just “assertEqual” is awkward. It may be mitigated a little though, by doing something like:
    var Assert = YAHOO.util.Assert;
    Assert.areEqual(...);
    
  • Strange, non standard “_should” syntax for “special” instructions (like ignoring tests or expecting exceptions to be thrown).
  • No support for HTML fixtures.

CONCLUSION

YUI Test is a very solid framework. If the lack of robust Continuous Integration support is not a show stopper for you, it is definitely worth considering. A couple of minor syntax annoyances are a small crack on the surface of otherwise polished product, however they are far from being serious enough to discourage you from using it.

JsUnit
http://www.jsunit.net

One of the first, most “traditional” xUnit type frameworks. It is still actively maintained, but it’s starting to show its age.

SYNTAX

function setUp() {
    //...
}

function testAddition() {
    assertEquals("2 plus 3 is 5", 5, add(2, 3));
}

function tearDown() {
    //...
}

PROS

  • Based on JUnit for Java, so it has “standard” syntax.
  • HTML runner.
  • There is some support for server-based testing in several browsers at once (via JUnit or Ant).

CONS

  • Tests methods not encapsulated in classes (pollute global namespace).
  • Very simplistic regarding features (no fixtures, no async testing etc.)

CONCLUSION

Nothing interesting here. This framework might be popular at a time, but now there are other, younger and much better contenders. Not recommended.

Other frameworks

  • jsUnity (http://jsunity.com)
    Seems quite promising, but for now it is a bit too young (as the time of this writing current version is 0.6). It has a clean, readable syntax with a nice set of assertions, but apart from this it is still in a very basic shape: no advanced runners, no CI integration etc. – just a minimal set of  functionality you can expect from a testing framework. On the one hand, there were several updates to this framework, on the other, the newest one is about a year old – so it is difficult to tell if this project is still actively maintained. If so, it may be worth looking at in future.
  • JsUnit (berlios) (http://jsunit.berlios.de)
    Not updated for ages… Probably no more actively maintained.
  • JsUnitTest (http://jsunittest.com)
    Never reached full maturity (latest version is 0.7.3, documentation is sparse) while at the same time not updated for a long time – seems an abandoned project.
  • J3Unit (http://j3unit.sourceforge.net)
    Declared to be in beta stage, but last update was 4 years ago… Seems abandoned.

The Final Verdict

There are two xUnit frameworks for Java Script that stand out from the crowd: jsTestDriver and YUI Test. YUI Test has a more feature packed assertion framework (with user input events mocking, async tests etc.) while jsTestDriver distinguishes itself with its awesome support for multi-browser server-based runner.

The final verdict?

JsTestDriver.

While YUI Test features may seem attractive, most of them is not so hard to mimic by hand, especially when using jQuery or a similar JS framework (e.g. simulating mouse and keyboard events in jQuery is dumb simple). On the other hand, hand-coding jsTestDriver’s support for distributed, parallel testing is a completely different pair of boots. And it’s a must have on any reasonable complex project, especially when using CI.

QUnit also stands out from the crowd in one aspect – its very neat and readable HTML test runner, which may be quite tempting. However, lack in other areas make it suitable only for small, simple projects.

It is worth noting that both QUnit and YUI Test integrations are supported by jsTestDriver – so you can have best of both words if you want – e.g. jsTestDriver’s runner with YUI Test’s robust set of assertions. However, none of xUnit frameworks provides enough advantages over standard jsTestDriver’s assertion framework to justify additional hassle – so, my final verdict is a plain jsTestDriver (at least as far as xUnit style frameworks go – BDD style frameworks are a completely different story).

Also, for a complete solution you need a mocking framework (not built-in any of xUnit frameworks).

Standalone mocking frameworks, as well as BDD flavor frameworks, will be compared in detail in next installments of this series.

Comments (6)

Choosing a Java Script testing framework

!!! THIS BLOG IS NO LONGER ACTIVE – CHECK OUT MY CURRENT BLOG AT VELESIN.IO !!!

When it comes to unit testing, Java Script is still often treated as a retarded young brother of more “serious” languages. But the times of tiny snippets doing basic DHTML tricks are gone long and for good. Looking at a couple of sites I was working with recently, it’s hard to not realize how JS heavy they are. Front end code is often more complex than its server side counterpart (which is sometimes just a very thin wrapper over ORM layer). It is also no longer just “chrome” – nowadays JS powers many of app’s vital features.

It’s obvious you need to test it as thoroughly as the rest of your code base. But which framework should you choose? As there is a real abundance of Unit Testing tools for JS, I’ve decided to make my choice educated and carried out a research. This post presents a list of JS testing tools I’ve found and an overview of some important factors you should keep in mind when selecting one of them. In next installments of this short series I’ll continue with more detailed comparison of frameworks listed below.

xUnit style frameworks

BDD style frameworks

Standalone mock frameworks

Important considerations

Mature, well documented and actively developed
I’m OK with using “edge” libraries in my app, as long as I keep it well covered with tests. But the testing platform itself must be rock solid. Staring for long minutes at a red bar, wondering if this is a bug in your class, your test code, in the framework, or just a bad API usage, although intellectually challenging, isn’t really anything funny. Good support and quick bug fixes for a tool you base the quality of your whole app upon won’t hurt either…

Support for different types of test runners
There are different scenarios for running Unit Tests: on a developer machine, from Continuous Integration server, inside an IDE.  If you want to work really comfortably under all circumstances, your framework should support at least in-browser HTML runner, command line runner and an ability to aggregate results from running your tests automatically on several web browsers at once.

Built in mock framework
Java Script is a very open language. It’s not a problem to manually stub or mock a method in JS. However, restoring old method state after each test and verifying expectations by hand can quickly become burdensome. A robust mock framework, simplifying and automating such tasks, can spare you a lot of pain.

Although there are several standalone JS mock frameworks available, it is a nice bonus if such a framework is integrated into the testing framework, as it usually provides cleaner and more consistent syntax, thus enhancing readability of your tests. Another nice bonus, relieving you from a lot of ugly plumbing, is support for mocking AJAX requests and timers.

Built-in support for DOM manipulation testing
Typically, you interact with the DOM in your tests simply via framework of your choice (jQuery, Prototype etc.). But there are some features that make DOM manipulation testing a bit easier, e.g. built-in API for loading HTML fixtures or ability to automatically clean up DOM between tests.

BDD style syntax
Although the difference between two flavors of syntax – BDD and “classic” xUnit – may seem mostly a matter of taste, I think BDD style offers a couple of tangible benefits, especially nested contexts that allow for much cleaner organization of test code, resulting in improved maintainability. Together with natural language like matcher syntax and a more flexible test case naming (free text strings instead of method names) this has a positive impact on readability – which for tests is even more important than for a regular code.

I hope this brief overview will already help you get started with selecting your favorite framework. Look forward to further posts in this series for more detailed comparisons.

Comments (8)