Archive for April, 2010

Java Script BDD style frameworks comparison

!!! THIS BLOG IS NO LONGER ACTIVE – CHECK OUT MY CURRENT BLOG AT VELESIN.IO !!!

In the previous installment of the series I’ve looked at xUnit style frameworks. In this one, I’ll compare BDD style frameworks.

JSpec
http://visionmedia.github.com/jspec

JSpec, influenced strongly by Ruby’s RSpec, takes a very interesting route: test cases are not written in plain Java Script, but rather in a custom DSL, closely resembling RSPec syntax (e.g. parentheses are optional).

SYNTAX

With a custom grammar:

describe 'ShoppingCart'
  before_each
    cart = new ShoppingCart
  end

  describe 'addProducts'
    it 'should add several products'
      cart.addProduct('cookie')
      cart.addProduct('icecream')
      cart.should.have 2, 'products'
    end
  end

  describe 'checkout'
    it 'should throw an error when checking out with no products'
      -{ cart.clear().checkout() }.should.throw_error EmptyCart
    end
  end
end

Alternatively in plain Java Script:

JSpec.describe('ShoppingCart', function(){
  before_each(function{
    cart = new ShoppingCart
  })

  describe('addProducts', function(){
    it ('should add several products', function(){
      cart.addProducts('cookie')
      cart.addProducts('icecream')
      expect(cart).to(have, 2, 'products')
    })
  })

  describe('checkout', function(){
    it ('should throw an error when checking out with no products', function(){
      expect(function(){ cart.clear().checkout() }).to(throw_error, EmptyCart)
    })
  })
})

PROS

  • RSPec like DSL.
    It gives you a very clean and concise syntax, more readable than normally possible in JS (you’ll love it, especially if you also code in Ruby).
  • Support for shared behaviors.
  • Pretty HTML runner.
  • Very robust server-based runner.
    Allows aggregating test results from several browsers at once. Support also non-browser platforms like Rhino or Env.js. Very nice for Continuous Integration.
  • Built-in mock framework.
  • Support for mocking AJAX requests.
  • Support for mocking timers.
  • HTML fixtures.
  • Nested describes.
  • Very robust set of matchers (including matchers for jQuery).
  • Ruby / Rails integration (with auto-test like functionality: automatic tests invocation when source file changes).
  • Extensible via modules and hooks.

CONS

  • RSPec like DSL.
    Unfortunately, pretty custom syntax comes at a cost. Test code must first go through a custom parser, which tends to break sometimes (e.g. embedding JS function declaration in test cases is tricky). It is no major problem – it happens only in some edge cases – but be prepared for an extensive debugging from time to time. Also, custom grammar doesn’t play well with syntax coloring in many IDEs, which may make test using custom grammar actually less readable, not more.
    It is possible to alternatively use regular Java Script syntax instead (what makes it IDE coloring friendly), but still the code goes through the parser, so all the problems with mis-interpreted syntax remain.
  • Shared behaviors implementation.
    JSpec provides its own mechanism (similar to the one used in RSpec) for sharing behaviors among many specifications.  However, its implementation is a bit awkward – shared behavior is not only used for sharing, but it is also automatically invoked as a regular, standalone behavior, what makes using it as a pure virtual set of tests rather awkward and error prone.
  • A little overcomplicated extensibility.
    Instead of using plain Java Script code for implementing helpers, you must use special module syntax. This again is the result of using custom grammar, which doesn’t allow you to intermix plain JS code with JSpec code as freely as you want.

CONCLUSION

JSpec is a very solid platform. It is probably the most feature packed framework on the market. However, the custom Ruby-like grammar is a mixed blessing – it gives you a very nice syntax, but unfortunately not without the cost. Anyway, this is a really good framework, one of the best available – and definitely recommended.

Jasmine
http://github.com/pivotal/jasmine

Jasmine is another RSpec like framework, created by authors of jsUnit and Screw.Unit. It takes a bit different approach than JSpec though, utilizing Java Script syntax to the maximum instead of trying to mimic Ruby.

SYNTAX

describe('some suite', function () {
  var suiteWideFoo;

  beforeEach(function () {
    suiteWideFoo = 0;
  });

  afterEach(function () {
    //...
  });

  describe('some nested suite', function() {
    var nestedSuiteBar;

    beforeEach(function() {
      nestedSuiteBar=1;
    });

    afterEach(function() {
      //...
    });

    it('nested expectation', function () {
      expect(suiteWideFoo).toEqual(0);
      expect(nestedSuiteBar).toEqual(1);
    });
  });

  it('top-level describe', function () {
    expect(suiteWideFoo).toEqual(0);
    expect(nestedSuiteBar).toEqual(undefined);
  });
});

PROS

  • Very clean syntax with well thought scope handling.
    And the side effect for it is that many interesting features just work, in plain Java Script, without need to use special API etc. E.g. you can create shared behaviors just by encapsulating any part of your spec (tests only, or the whole describe clause, with tests, beforeEach filters etc.) in a JS function, and then calling that function from anywhere inside your describe block, without need to use any fancy syntax for shared behavior declaration. The same goes for helper methods etc.
  • Built-in top-notch mocking framework.
  • Support for asynchronous tests.
  • Nested describes.
  • Very nice HTML runner.
  • Good support for Continuous Integration.
    Either via jasmine-ruby (http://github.com/pivotal/jasmine-ruby) or via integration with jsTestDriver (http://code.google.com/p/js-test-driver/wiki/XUnitCompatibility)

CONS

  • From the version number (current one is 0.10.3) it seems it may be a bit young. However, it is feature packed and very stable, so this con is probably over-defensive.

CONCLUSION

This is my favorite JS testing framework. I don’t see practically any cons. Definitely recommended.

JSSpec

http://jania.pe.kr/aw/moin.cgi/JSSpec

This is one of the first BDD style frameworks for Java Script. Therefore, it gained some traction, being used by such projects like e.g. MooTools, but nowadays it is starting to show its age.

SYNTAX

describe('Plus operation', {
    before_each : function() {
        // ...
    },

    'should concatenate two strings': function() {
        value_of("Hello " + "World").should_be("Hello World");
    },

    'should add two numbers': function() {
        value_of(1 + 2).should_be(3);
    },

    after_each : function() {
        // ...
    }
});

PROS

  • HTML runner.
  • Conditional execution for different browsers.
    There’s an interesting feature that lets you embed special condition clauses in your test names that tell the framework to run given test only in specified browsers (although I have some doubts about using it, as I think my JS code should be rather portable, and the exact same set of test should pass cross-browser).

CONS

  • A bit awkward syntax.
    The framework uses “value_of” instead of commonly established “expect” and tests names are specified directly as function names instead of an established “it” method with a string parameter.
  • No nested describes.
  • No support for Continuous Integration.
  • Not updated for some time – it doesn’t seem to be actively maintained any more.

CONCLUSION

It is decent and usable framework, but aged, lacking in features when compared to its younger counterparts. Its very basic capabilities, together with a bit awkward syntax, not matching established modern standards, doesn’t make it a compelling choice. Not recommended.

Other Frameworks

  • Screw.Unit (http://github.com/nkallen/screw-unit)
    A quite decent, mature framework. Maybe there are no exceptional features that distinguish it from the crowd, but anyway it is a solid contender. However, it seems to be abandoned in favor of Jasmine framework, so most probably you cannot expect active development of Screw.Unit any more.
  • Inspec (http://github.com/aq1018/inspec)
    This framework seems to be at an early stage. Documentation is practically nonexistent. Even its own test suite (to which author sends back for more examples) is fairly incomplete. Not usable in production for now.
  • jShoulda (http://jshoulda.scriptia.net)
    It is officially abandoned by the author in favor of SugarTest.
  • SugarTest (http://sugartest.scriptia.net)
    It is immature still (current version is 0.2, feature set is relatively basic) and it doesn’t seem to be very actively developed (most recent commit is about a year old).

The Final Verdict

It is quite easy to choose top two contenders, as both JSpec and Jasmine clearly stand out from the crowd. It is much harder though to select between these two. Both are mature, actively supported and robust, and choke-full of very cool features.

JSpec seems more feature complete, however its custom grammar becomes problematic at times. On the other hand, Jasmine’s clear syntax and scoping, allow to easily re-create many of missing features (e.g. shared behaviors or helpers can be implemented just as plain JS functions), enabling at the same time rock-solid test code that doesn’t need to rely on the whims of custom grammar parser. Also, Jasmine offers no half-baked features. If it is in, it is polished (a good example of this is its built-in mocking framework, which is probably the most robust one on the market).

My final decision? Jasmine.

It is a real joy to work with this framework.

Comments (4)

Java Script xUnit style frameworks comparison

!!! THIS BLOG IS NO LONGER ACTIVE – CHECK OUT MY CURRENT BLOG AT VELESIN.IO !!!

In this installment of the series I’ll look at xUnit frameworks. Comparison of BDD frameworks and standalone mock frameworks will follow in subsequent posts.

JsTestDriver
http://code.google.com/p/js-test-driver

This one is a very strange beast. Its main focus, as claimed by authors, is command line and continuous integration – you can think of jsTestDriver more as a robust test runner platform rather than a typical assertion framework. This is very visible in its design – its client-server architecture based runner is as robust as it may get, definitely one of (or the) best on the market, while assertion framewrok, although solid, is relatively bare bone.

SYNTAX

TestCase("MyTestCase", {
    setUp: function() {
        //...
    },

    testAssertionWithMessage: function() {
        assertTrue("this is not true", false);
    },

    testAssertionWithoutMessage: function() {
        var actual = 5;
        assertEqual(5, actual);
    },

    tearDown: function() {
        //...
    }
};

PROS

  • Robust Continuous Integration support: parallel execution in many browsers at once (both local and remote) with aggregated result reporting; designed with speed in mind (keeps everything permanently loaded in browser, reloads only files that changed – what gives a huge speed boost by avoiding frequent re-parsing of code)
  • Allows replacing its assertion layer with another framework. This way jsTestDriver acts only as a test runner while actual test suites are written using alternative syntax. This offers possibility to replace not so pretty jsTestDriver syntax with something more fancy (even changing its flavor from xUnit to BDD in case of integrating with Jasmine) and also makes porting your existing code to jsTestDriver a breeze if you are lucky to use one of supported frameworks. Currently though, only a few integrations are supported (as of time of this writing only QUnit, YUI Test and Jasmine).
  • Integrates with Eclipse and IntelliJ IDEA. If you are using one of these, it may compensate for the lack of HTML runner (bad luck for NetBeans users though).
  • Automatically resets DOM between tests. Also provides some support for HTML fixture loading (although rather very basic, based on comments: if you have HTML inside a comment block in your test, it will get automatically loaded into DOM).
  • No need to create any HTML wrappers to run tests.
  • There is a code coverage plugin for jsTestDriver available (http://code.google.com/p/js-test-driver/wiki/CodeCoverage).

CONS

  • No HTML runner. If you happen to use IDE that is not supported by jsTestDriver, you must read test results from the console, with its ugly and hard to read formatting of error messages.
  • A bit over-complicated to run on a developer machine. For CI this is no problem, but having to keep open two consoles (one to control a server, second to display test results) plus one or more browsers on a local workstation is a little overkill for me.

CONCLUSION

This is a really solid framework, with simple, but nice and clean syntax and a couple of unique features. And although its CI oriented client-server architecture may feel too heavy for some usage scenarios, it is a real deal for bigger, more complex projects. Among xUnit style frameworks this is definitely one of the best choices.

QUnit
http://docs.jquery.com/QUnit

This is an “unofficial official” testing framework for jQuery. It’s mature and quite established on the market (also partly due to the popularity of jQuery). It is rather simplistic, containing by design only most core set of features (with two notable exceptions: support for HTML fixtures and testing of asynchronous calls).

SYNTAX

test("a basic test without module", function() {
    var value = "hello";
    equals("hello", value, "We expect value to be hello");
});

module("Some Module", {
    setup: function() {
        //...
    },
    teardown: function() {
        //...
    }
});

test("first test within module", function() {
    ok(true, "all pass");
});

test("second test within module", function() {
    ok( true, "all pass" );
});

PROS

  • HTML runner with nice output formatting.
  • Automatic clean-up of DOM between tests. Some support for HTML fixtures – content of predefined DIV, if present inside runner’s page HTML, is re-created between tests (although this is not fully clean when you run multiple suites at once – all fixtures for all suites must be cluttered in HTML in such case, instead of being stored separately and loaded on demand).
  • Support for testing asynchronous code.
  • Integrates with jsTestDriver.

CONS

  • Non standard (and non intuitive) syntax. Test methods are placed outside, instead of inside of module. Assertion have very custom naming (e.g. “ok” instead of “assertTrue”). Surprising “expect” method (setting expectations at the beginning of a test on how many calls to other assertions will be made in this test).
  • Absolutely barebone set of assertions (only 3 assertions available: “ok”, “equals” and “same”).
  • No command line runner (and no support for CI).
  • Tests pollute global namespace.

CONCLUSION

Being very simple and being a standard platform for jQuery core tests may seem tempting at first, but this framework is so simplistic, and has such a strange, custom syntax, that I’ll not recommend it. Built-in support for HTML fixtures is a plus, but the way it is implemented makes it useful only for a small test suites, so this doesn’t change the final verdict.

jqUnit
http://code.google.com/p/jqunit

jqUnit is just a wrapper around QUnit framework. It doesn’t add any new features, only wraps the QUnit with a different, JSUnit compatible syntax.

PROS

  • The same as for QUnit (although I’m not sure about jsTestDriver integration).
  • Fixes a bit “weird” QUnit syntax. Allows easy porting of your old JsUnit code.

CONS

  • The same as for QUnit.

CONCLUSION

If you really need to port your huge code base from JsUnit, this may be an interesting choice. Otherwise it is not recommended.

YUI Test
http://developer.yahoo.com/yui/yuitest

YUI Test is a part of Yahoo YUI library. It is mature and very actively maintained (as well as YUI library itself). Although it uses YUI library namespace, it can be used also standalone.

SYNTAX

var oTestCase = new YAHOO.tool.TestCase({
    name: "TestCase Name",

    // Special instructions
    _should: {
        ignore: {
            testSomethingUnimportant: true //ignore this test
        },

        error: {
            testSomethingExceptional: true //this should throw an error
        }
    },

    setUp : function () {
        //...
    },

    tearDown : function () {
        //...
    },

    testSomething: function () {
        YAHOO.util.Assert.areEqual("Something", this.someValue, "Value should be 'Something'");
    },

    testSomethingUnimportant: function () {
        //this test is ignored
    },

    testSomethingExceptional: function () {
        methodThatThrowsException();
    }
});

PROS

  • Robust set of assertions.
  • Built-in mocking of mouse and keyboard events.
  • Supports asynchronous tests.
  • Not only test cases but also test suites can have setUp and tearDown methods – this allows almost BDD like nesting (however with not so nice syntax as in case of BDD frameworks).
  • Integrates with jsTestDriver.

CONS

  • Very ugly HTML runner. It tries to emulate the look of a console – with all its readability problems. (However, YUI Test runner provides very robust support for test result serialization, with option to pass serialized results to a given URL, so it should be possible to create your own pretty printer).
  • No command line runner. (Although, again – robust support for test result serialization should allow you to code your own support for CI / command line invocation).
  • YUI related namespacing. Having to write “YAHOO.util.Assert.areEqual” instead of just “assertEqual” is awkward. It may be mitigated a little though, by doing something like:
    var Assert = YAHOO.util.Assert;
    Assert.areEqual(...);
    
  • Strange, non standard “_should” syntax for “special” instructions (like ignoring tests or expecting exceptions to be thrown).
  • No support for HTML fixtures.

CONCLUSION

YUI Test is a very solid framework. If the lack of robust Continuous Integration support is not a show stopper for you, it is definitely worth considering. A couple of minor syntax annoyances are a small crack on the surface of otherwise polished product, however they are far from being serious enough to discourage you from using it.

JsUnit
http://www.jsunit.net

One of the first, most “traditional” xUnit type frameworks. It is still actively maintained, but it’s starting to show its age.

SYNTAX

function setUp() {
    //...
}

function testAddition() {
    assertEquals("2 plus 3 is 5", 5, add(2, 3));
}

function tearDown() {
    //...
}

PROS

  • Based on JUnit for Java, so it has “standard” syntax.
  • HTML runner.
  • There is some support for server-based testing in several browsers at once (via JUnit or Ant).

CONS

  • Tests methods not encapsulated in classes (pollute global namespace).
  • Very simplistic regarding features (no fixtures, no async testing etc.)

CONCLUSION

Nothing interesting here. This framework might be popular at a time, but now there are other, younger and much better contenders. Not recommended.

Other frameworks

  • jsUnity (http://jsunity.com)
    Seems quite promising, but for now it is a bit too young (as the time of this writing current version is 0.6). It has a clean, readable syntax with a nice set of assertions, but apart from this it is still in a very basic shape: no advanced runners, no CI integration etc. – just a minimal set of  functionality you can expect from a testing framework. On the one hand, there were several updates to this framework, on the other, the newest one is about a year old – so it is difficult to tell if this project is still actively maintained. If so, it may be worth looking at in future.
  • JsUnit (berlios) (http://jsunit.berlios.de)
    Not updated for ages… Probably no more actively maintained.
  • JsUnitTest (http://jsunittest.com)
    Never reached full maturity (latest version is 0.7.3, documentation is sparse) while at the same time not updated for a long time – seems an abandoned project.
  • J3Unit (http://j3unit.sourceforge.net)
    Declared to be in beta stage, but last update was 4 years ago… Seems abandoned.

The Final Verdict

There are two xUnit frameworks for Java Script that stand out from the crowd: jsTestDriver and YUI Test. YUI Test has a more feature packed assertion framework (with user input events mocking, async tests etc.) while jsTestDriver distinguishes itself with its awesome support for multi-browser server-based runner.

The final verdict?

JsTestDriver.

While YUI Test features may seem attractive, most of them is not so hard to mimic by hand, especially when using jQuery or a similar JS framework (e.g. simulating mouse and keyboard events in jQuery is dumb simple). On the other hand, hand-coding jsTestDriver’s support for distributed, parallel testing is a completely different pair of boots. And it’s a must have on any reasonable complex project, especially when using CI.

QUnit also stands out from the crowd in one aspect – its very neat and readable HTML test runner, which may be quite tempting. However, lack in other areas make it suitable only for small, simple projects.

It is worth noting that both QUnit and YUI Test integrations are supported by jsTestDriver – so you can have best of both words if you want – e.g. jsTestDriver’s runner with YUI Test’s robust set of assertions. However, none of xUnit frameworks provides enough advantages over standard jsTestDriver’s assertion framework to justify additional hassle – so, my final verdict is a plain jsTestDriver (at least as far as xUnit style frameworks go – BDD style frameworks are a completely different story).

Also, for a complete solution you need a mocking framework (not built-in any of xUnit frameworks).

Standalone mocking frameworks, as well as BDD flavor frameworks, will be compared in detail in next installments of this series.

Comments (6)

Choosing a Java Script testing framework

!!! THIS BLOG IS NO LONGER ACTIVE – CHECK OUT MY CURRENT BLOG AT VELESIN.IO !!!

When it comes to unit testing, Java Script is still often treated as a retarded young brother of more “serious” languages. But the times of tiny snippets doing basic DHTML tricks are gone long and for good. Looking at a couple of sites I was working with recently, it’s hard to not realize how JS heavy they are. Front end code is often more complex than its server side counterpart (which is sometimes just a very thin wrapper over ORM layer). It is also no longer just “chrome” – nowadays JS powers many of app’s vital features.

It’s obvious you need to test it as thoroughly as the rest of your code base. But which framework should you choose? As there is a real abundance of Unit Testing tools for JS, I’ve decided to make my choice educated and carried out a research. This post presents a list of JS testing tools I’ve found and an overview of some important factors you should keep in mind when selecting one of them. In next installments of this short series I’ll continue with more detailed comparison of frameworks listed below.

xUnit style frameworks

BDD style frameworks

Standalone mock frameworks

Important considerations

Mature, well documented and actively developed
I’m OK with using “edge” libraries in my app, as long as I keep it well covered with tests. But the testing platform itself must be rock solid. Staring for long minutes at a red bar, wondering if this is a bug in your class, your test code, in the framework, or just a bad API usage, although intellectually challenging, isn’t really anything funny. Good support and quick bug fixes for a tool you base the quality of your whole app upon won’t hurt either…

Support for different types of test runners
There are different scenarios for running Unit Tests: on a developer machine, from Continuous Integration server, inside an IDE.  If you want to work really comfortably under all circumstances, your framework should support at least in-browser HTML runner, command line runner and an ability to aggregate results from running your tests automatically on several web browsers at once.

Built in mock framework
Java Script is a very open language. It’s not a problem to manually stub or mock a method in JS. However, restoring old method state after each test and verifying expectations by hand can quickly become burdensome. A robust mock framework, simplifying and automating such tasks, can spare you a lot of pain.

Although there are several standalone JS mock frameworks available, it is a nice bonus if such a framework is integrated into the testing framework, as it usually provides cleaner and more consistent syntax, thus enhancing readability of your tests. Another nice bonus, relieving you from a lot of ugly plumbing, is support for mocking AJAX requests and timers.

Built-in support for DOM manipulation testing
Typically, you interact with the DOM in your tests simply via framework of your choice (jQuery, Prototype etc.). But there are some features that make DOM manipulation testing a bit easier, e.g. built-in API for loading HTML fixtures or ability to automatically clean up DOM between tests.

BDD style syntax
Although the difference between two flavors of syntax – BDD and “classic” xUnit – may seem mostly a matter of taste, I think BDD style offers a couple of tangible benefits, especially nested contexts that allow for much cleaner organization of test code, resulting in improved maintainability. Together with natural language like matcher syntax and a more flexible test case naming (free text strings instead of method names) this has a positive impact on readability – which for tests is even more important than for a regular code.

I hope this brief overview will already help you get started with selecting your favorite framework. Look forward to further posts in this series for more detailed comparisons.

Comments (8)