Java Script BDD style frameworks comparison

In the previous installment of the series I’ve looked at xUnit style frameworks. In this one, I’ll compare BDD style frameworks.

JSpec
http://visionmedia.github.com/jspec

JSpec, influenced strongly by Ruby’s RSpec, takes a very interesting route: test cases are not written in plain Java Script, but rather in a custom DSL, closely resembling RSPec syntax (e.g. parentheses are optional).

SYNTAX

With a custom grammar:

describe 'ShoppingCart'
  before_each
    cart = new ShoppingCart
  end

  describe 'addProducts'
    it 'should add several products'
      cart.addProduct('cookie')
      cart.addProduct('icecream')
      cart.should.have 2, 'products'
    end
  end

  describe 'checkout'
    it 'should throw an error when checking out with no products'
      -{ cart.clear().checkout() }.should.throw_error EmptyCart
    end
  end
end

Alternatively in plain Java Script:

JSpec.describe('ShoppingCart', function(){
  before_each(function{
    cart = new ShoppingCart
  })

  describe('addProducts', function(){
    it ('should add several products', function(){
      cart.addProducts('cookie')
      cart.addProducts('icecream')
      expect(cart).to(have, 2, 'products')
    })
  })

  describe('checkout', function(){
    it ('should throw an error when checking out with no products', function(){
      expect(function(){ cart.clear().checkout() }).to(throw_error, EmptyCart)
    })
  })
})

PROS

  • RSPec like DSL.
    It gives you a very clean and concise syntax, more readable than normally possible in JS (you’ll love it, especially if you also code in Ruby).
  • Support for shared behaviors.
  • Pretty HTML runner.
  • Very robust server-based runner.
    Allows aggregating test results from several browsers at once. Support also non-browser platforms like Rhino or Env.js. Very nice for Continuous Integration.
  • Built-in mock framework.
  • Support for mocking AJAX requests.
  • Support for mocking timers.
  • HTML fixtures.
  • Nested describes.
  • Very robust set of matchers (including matchers for jQuery).
  • Ruby / Rails integration (with auto-test like functionality: automatic tests invocation when source file changes).
  • Extensible via modules and hooks.

CONS

  • RSPec like DSL.
    Unfortunately, pretty custom syntax comes at a cost. Test code must first go through a custom parser, which tends to break sometimes (e.g. embedding JS function declaration in test cases is tricky). It is no major problem – it happens only in some edge cases – but be prepared for an extensive debugging from time to time. Also, custom grammar doesn’t play well with syntax coloring in many IDEs, which may make test using custom grammar actually less readable, not more.
    It is possible to alternatively use regular Java Script syntax instead (what makes it IDE coloring friendly), but still the code goes through the parser, so all the problems with mis-interpreted syntax remain.
  • Shared behaviors implementation.
    JSpec provides its own mechanism (similar to the one used in RSpec) for sharing behaviors among many specifications.  However, its implementation is a bit awkward – shared behavior is not only used for sharing, but it is also automatically invoked as a regular, standalone behavior, what makes using it as a pure virtual set of tests rather awkward and error prone.
  • A little overcomplicated extensibility.
    Instead of using plain Java Script code for implementing helpers, you must use special module syntax. This again is the result of using custom grammar, which doesn’t allow you to intermix plain JS code with JSpec code as freely as you want.

CONCLUSION

JSpec is a very solid platform. It is probably the most feature packed framework on the market. However, the custom Ruby-like grammar is a mixed blessing – it gives you a very nice syntax, but unfortunately not without the cost. Anyway, this is a really good framework, one of the best available – and definitely recommended.

Jasmine
http://github.com/pivotal/jasmine

Jasmine is another RSpec like framework, created by authors of jsUnit and Screw.Unit. It takes a bit different approach than JSpec though, utilizing Java Script syntax to the maximum instead of trying to mimic Ruby.

SYNTAX

describe('some suite', function () {
  var suiteWideFoo;

  beforeEach(function () {
    suiteWideFoo = 0;
  });

  afterEach(function () {
    //...
  });

  describe('some nested suite', function() {
    var nestedSuiteBar;

    beforeEach(function() {
      nestedSuiteBar=1;
    });

    afterEach(function() {
      //...
    });

    it('nested expectation', function () {
      expect(suiteWideFoo).toEqual(0);
      expect(nestedSuiteBar).toEqual(1);
    });
  });

  it('top-level describe', function () {
    expect(suiteWideFoo).toEqual(0);
    expect(nestedSuiteBar).toEqual(undefined);
  });
});

PROS

  • Very clean syntax with well thought scope handling.
    And the side effect for it is that many interesting features just work, in plain Java Script, without need to use special API etc. E.g. you can create shared behaviors just by encapsulating any part of your spec (tests only, or the whole describe clause, with tests, beforeEach filters etc.) in a JS function, and then calling that function from anywhere inside your describe block, without need to use any fancy syntax for shared behavior declaration. The same goes for helper methods etc.
  • Built-in top-notch mocking framework.
  • Support for asynchronous tests.
  • Nested describes.
  • Very nice HTML runner.
  • Good support for Continuous Integration.
    Either via jasmine-ruby (http://github.com/pivotal/jasmine-ruby) or via integration with jsTestDriver (http://code.google.com/p/js-test-driver/wiki/XUnitCompatibility)

CONS

  • From the version number (current one is 0.10.3) it seems it may be a bit young. However, it is feature packed and very stable, so this con is probably over-defensive.

CONCLUSION

This is my favorite JS testing framework. I don’t see practically any cons. Definitely recommended.

JSSpec

http://jania.pe.kr/aw/moin.cgi/JSSpec

This is one of the first BDD style frameworks for Java Script. Therefore, it gained some traction, being used by such projects like e.g. MooTools, but nowadays it is starting to show its age.

SYNTAX

describe('Plus operation', {
    before_each : function() {
        // ...
    },

    'should concatenate two strings': function() {
        value_of("Hello " + "World").should_be("Hello World");
    },

    'should add two numbers': function() {
        value_of(1 + 2).should_be(3);
    },

    after_each : function() {
        // ...
    }
});

PROS

  • HTML runner.
  • Conditional execution for different browsers.
    There’s an interesting feature that lets you embed special condition clauses in your test names that tell the framework to run given test only in specified browsers (although I have some doubts about using it, as I think my JS code should be rather portable, and the exact same set of test should pass cross-browser).

CONS

  • A bit awkward syntax.
    The framework uses “value_of” instead of commonly established “expect” and tests names are specified directly as function names instead of an established “it” method with a string parameter.
  • No nested describes.
  • No support for Continuous Integration.
  • Not updated for some time – it doesn’t seem to be actively maintained any more.

CONCLUSION

It is decent and usable framework, but aged, lacking in features when compared to its younger counterparts. Its very basic capabilities, together with a bit awkward syntax, not matching established modern standards, doesn’t make it a compelling choice. Not recommended.

Other Frameworks

  • Screw.Unit (http://github.com/nkallen/screw-unit)
    A quite decent, mature framework. Maybe there are no exceptional features that distinguish it from the crowd, but anyway it is a solid contender. However, it seems to be abandoned in favor of Jasmine framework, so most probably you cannot expect active development of Screw.Unit any more.
  • Inspec (http://github.com/aq1018/inspec)
    This framework seems to be at an early stage. Documentation is practically nonexistent. Even its own test suite (to which author sends back for more examples) is fairly incomplete. Not usable in production for now.
  • jShoulda (http://jshoulda.scriptia.net)
    It is officially abandoned by the author in favor of SugarTest.
  • SugarTest (http://sugartest.scriptia.net)
    It is immature still (current version is 0.2, feature set is relatively basic) and it doesn’t seem to be very actively developed (most recent commit is about a year old).

The Final Verdict

It is quite easy to choose top two contenders, as both JSpec and Jasmine clearly stand out from the crowd. It is much harder though to select between these two. Both are mature, actively supported and robust, and choke-full of very cool features.

JSpec seems more feature complete, however its custom grammar becomes problematic at times. On the other hand, Jasmine’s clear syntax and scoping, allow to easily re-create many of missing features (e.g. shared behaviors or helpers can be implemented just as plain JS functions), enabling at the same time rock-solid test code that doesn’t need to rely on the whims of custom grammar parser. Also, Jasmine offers no half-baked features. If it is in, it is polished (a good example of this is its built-in mocking framework, which is probably the most robust one on the market).

My final decision? Jasmine.

It is a real joy to work with this framework.

About these ads

4 Comments

  1. […] was googling around a bit to find a tool that fits my needs and with the help of  this article I was able to identify Jasmine as the best suiting solution. You can also find some nice Jasmine […]

  2. […] Comparisons of Assertion Frameworks: BDD style vs xUnit […]

  3. Steve said

    You might also consider Yadda (https://github.com/acuminous/yadda) and cucumber.js (https://github.com/cucumber/cucumber-js)

  4. Thanks! +1 for cucumber-js consideration.

RSS feed for comments on this post

Comments are closed.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: