I have already shared my comparison of two JavaScript testing solutions where I admitted that I favor Jest over Mocha. Back then, I listed all major differences between those tools summarized with advantages and disadvantages of migrating to Jest:
Pros:
- Simpler API, less boilerplate code.
- Flexible and easy configuration.
- Test files executed in isolation.
- Advanced watch mode.
- Snapshots support = easier start with testing.
- Code coverage.
Cons:
- Another migration.
- Mocha has still a bit better performance (according to my quick tests).
My analysis got very positive feedback, with only a few little concerns, so I got encouraged to take action and verify the assumptions stated. I picked two different projects to play with to ensure both of them will uniformly benefit from using Jest.
Gutenberg
There are three main development focuses for WordPress core this year. The new editor is one of them and it is being actively developed on GitHub as a WordPress plugin under the codename Gutenberg. The goal here is to completely redefine content editing experience that makes creating rich posts effortless. This is also an excellent opportunity to modernize technology stack behind WordPress. If you want to find out more how Gutenberg is changing radically the latter, I recommend the blog post from my teammate Riad Benguella.
When I first looked at tests configuration in Gutenberg, it turned out it has a more complicated testing setup with Mocha than I expected. Fortunately, I was able to provide a working proof of concept using Jest in a day or two. It turned out that it is not that hard to make the first test suites pass because both tools use a very similar API. It was sufficient to add the following list of aliases for Mocha methods in the setup script referenced with the setupTestFrameworkScriptFile config option:
global.after = global.afterAll; global.before = global.beforeAll; global.context = global.describe;
It required much more gymnastics to make all the remaining tests pass. It was mostly caused by the Webpack build step included in the flow. Fortunately, it was possible to achieve the same result using Jest configuration options exclusively and thus speed up the boot time by a couple of seconds. The excerpt from the Jest config presented below illustrates how Webpack features can be mapped:
{
...
"moduleNameMapper": {
"\\.(scss|css)$": "/test/style-mock.js"
},
"transform": {
"^.+\\.jsx?$": "babel-jest",
"\\.pegjs$": "/test/pgejs-transform.js"
},
...
}
If you are keen to learn more about this setup, I recommend diving into the aforementioned pull request. It was satisfactory to see it working, but it was not enough because it is even better to make it right, make it fast. At that stage, all test suites were still using Mocha specific global methods, Chai assertions and Sinon test doubles. It was very convenient in the process of migration, but it seemed like a much better idea to take advantage of API that Jest provides natively. Fortunately, there is an existing tool that turns all that tedious refactoring work into a pleasant experience. The jest-codemods project, maintained by Kenneth Skovhus, offers the interactive CLI mode which covers most of the necessary transformations from Mocha and Chai specific APIs to Jest alternatives. Here is a simplified example of the common changes applied:
# Before
- import { expect } from 'chai';
import { registerBlockType, unregisterBlockType } from 'blocks';
import { editor } from '../state';
describe( 'editor()', () => {
- before( () => {
registerBlockType( 'core/test-block', {} );
} );
- after( () => {
unregisterBlockType( 'core/test-block' );
} );
- it( 'should return empty blocksByUid, blockOrder, history by default', () => {
const state = editor( undefined, {} );
- expect( state.blocksByUid ).to.eql( {} );
- expect( state.blockOrder ).to.eql( [] );
- expect( state ).to.have.keys( 'history' );
} );
} );
# After
import { registerBlockType, unregisterBlockType } from 'blocks';
import { editor } from '../state';
describe( 'editor()', () => {
+ beforeAll( () => {
registerBlockType( 'core/test-block', {} );
} );
+ afterAll( () => {
unregisterBlockType( 'core/test-block' );
} );
+ test( 'should return empty blocksByUid, blockOrder, history by default', () => {
const state = editor( undefined, {} );
+ expect( state.blocksByUid ).toEqual( {} );
+ expect( state.blockOrder ).toEqual( [] );
+ expect( state ).toHaveProperty( 'history' );
} );
} );
The only remaining work left was to manually port Sinon spies, stubs and their custom Chai expectations. All of this allowed to get rid of a pretty good number of npm dependencies from the repository: chai, dirty-chai, sinon, sinon-chai and sinon-test. At the same time, those changes made tests execution a bit faster because it was no longer necessary to initialize Chai configuration for every test suite. Finally, it simplified developer experience as the full documentation can now be found in one place – on Jest’s website. To get more in-depth knowledge about changes introduced check out the related pull request. The end result was very beneficial with regard to the existing 30+ test suites. It confirmed that testing code with Jest is definitely an improvement compared to what Mocha and friends have to offer.
Calypso
Let’s get back to Calypso – a single interface to manage all WordPress.com and Jetpack-enabled sites. I have already shared in the previous post that the initial exploration has started with this pull request. The gist of the research can be concluded with the following quotation:
It was quite easy to get to the point where all unit tests that verify code executed on the server work properly with Jest. I also rewrote test files to use the Jest API to show how writing tests compares to Mocha. I was able to get even further and integrate Jest with Circle CI or PhpStorm. It didn’t take much time to setup code coverage and watch mode, neither.
As a final result, this diff got merged not only with the new configuration for server-side testing but also with the revamped setup for the integration tests. Frankly speaking, it was only a very small subset of all existing tests – 8 files in total.
It was about the time to finally tackle over 1 000 test suites living in Calypso repository which ensures the quality of codebase powering the client-side. It needs to be clarified that those tests also verify code that is shared with the server-side part. All tests are grouped under one configuration even though some of them can be identified as unit tests and other as component tests. The first step was similar to the previously described cases, adding the same list of aliases for Mocha specific global API methods. This allowed close to 800 test suites to pass all checks with no effort. However, it took lots of time to make the rest of tests pass. It isn’t a big surprise that the size of the final pull request was monstrous.
Let me list the most common refactorings performed:
- Mockery served well for quite some time, but its setup was always fragile. To avoid very rare but obviously unexpected timeouts all tests were updated to use file mocks offered by Jest.
- Jest checks if matching file contains at least one test. All non-test files like fixtures or mocks had to be moved to subfolders to make sure they don’t get listed as failures.
- Mocha allows sharing an instance property between all methods for a given test suite. Jest doesn’t enable the same option which prevents using
thiskeyword. It is possible to declare a local variable instead to achieve the same goal.
All above helped to get rid of a few in-house developed test helpers:
nockControl– it was no longer in use since the integration tests were introduced. All tests triggering network requests are now grouped under this new type.useFakeDom– it is possible now to set browser-like test environment per test suite using a special comment:/** @jest-environment jsdom */.useFilesystemMocksanduseMockery– Jest offers its own file mocks handling, which is a way more flexible, reliable and faster than what was built on top of Mockery.
Finally, the code that powered Mocha custom test runner got removed altogether with mocha npm dependency. jest-codemods helped to replace Mocha global methods (after, before, context and it) with Jest’s equivalent API. It made all temporarily added aliases obsolete. At the same time, Chai, Sinon and Enzyme remained integrated without any modifications to make the transition faster and easier. To better illustrate what has changed I prepared a slightly adjusted example of the actual code:
# Before
/**
* External dependencies
*/
import { expect } from 'chai';
import sinon from 'sinon';
- import useMockery from 'test/helpers/use-mockery';
- import useFakeDom from 'test/helpers/use-fake-dom';
describe( 'Search', function() {
- let EMPTY_COMPONENT, React, TestUtils;
- useFakeDom();
- useMockery( mockery => {
- React = require( 'react' );
- TestUtils = require( 'react-addons-test-utils' );
-
- EMPTY_COMPONENT = require( 'test/helpers/react/empty-component' );
-
- mockery.registerMock( 'lib/analytics', {} );
- mockery.registerMock( 'gridicons', EMPTY_COMPONENT );
- } );
- before( function() {
- this.searchClass = require( '../' );
- } );
beforeEach( function() {
- this.onSearch = sinon.stub();
} );
describe( 'with initialValue', function() {
beforeEach( function() {
- this.initialValue = 'hello';
- this.searchElement = React.createElement( this.searchClass, {
- initialValue: this.initialValue,
- onSearch: this.onSearch
- } );
- this.rendered = TestUtils.renderIntoDocument( this.searchElement );
} );
- it( 'should set state.keyword with the initialValue after mount', function() {
- expect( this.rendered.state.keyword ).to.equal( this.initialValue );
} );
} );
} );
# After
+ /** @jest-environment jsdom */
+ jest.mock( 'lib/analytics', () => ( {} ) );
+ jest.mock( 'gridicons', () => require( 'components/empty-component' ) );
/**
* External dependencies
*/
+ import { createElement } from 'react';
import { expect } from 'chai';
+ import { renderIntoDocument } from 'react-addons-test-utils';
import { stub } from 'sinon';
/**
* Internal dependencies
*/
+ import SearchClass from '../';
describe( 'Search', () => {
+ let onSearch, rendered;
beforeEach( () => {
+ onSearch = stub();
} );
describe( 'with initialValue', () => {
+ const initialValue = 'hello';
beforeEach( () => {
+ const searchElement = createElement( SearchClass, {
+ initialValue,
+ onSearch
+ } );
+ rendered = renderIntoDocument( searchElement );
} );
+ test( 'should set state.keyword with the initialValue after mount', () => {
+ expect( rendered.state.keyword ).to.equal( initialValue );
} );
} );
} );
Ideally, all the remaining migration steps done for Gutenberg project should be also repeated. However, I think it is reasonable to be more flexible in this case given the scale. Personally, I would keep Chai assertions untouched for the next couple of weeks or months to make the transition smoother for all the existing contributors. If having two different assertion styles turns out to be too confusing, there is always a ready solution to take advantage of – Jest codemods. I’m torn between keeping Sinon or removing it completely. It might be more reasonable to keep the tests using it untouched because rewriting them manually to use Jest mocks would require lots of work with little benefits.
My initial exploration gave the impression that Mocha will perform better than Jest with the set of tests involved. It turned out that it was a completely wrong assumption. It was possible with Jest to speed up execution of all tests from ~90 to ~50 seconds on MacBook Pro 2015. It was achieved by using lazy initialization technique for Chai integration with both Enzyme and Sinon libraries. I learned about this approach from the article shared by Gary Borton from Airbnb. For comparison with Mocha, it took before the transition well over 60 seconds to finish a single run in an ideal scenario where no test randomly timed out.
All the changes discussed above made the Calypso documentation a bit outdated. You can find all updates related to the migration in this pull request. It was also a perfect opportunity to finally document testing approach for WordPress.com.
Final thoughts
First of all, I want to make it clear that I still think that Mocha is an excellent tool for testing your JavaScript code. It gives you lots of freedom by allowing to use any assertion library you want. The same rule applies to the interfaces (BDD, TDD, etc.) and reporters (spec, dot, etc.). It integrates well with mocking libraries and a variety of other test helpers. However, such flexibility comes at a price, you are left on your own to discover which independent pieces need to be put together. You also have to make sure they play nicely with each other. It might be especially challenging to keep with the latest versions of all picked dependencies given that they release new versions on their own pace.
Luckily, Jest addresses those concerns by providing a reasonable default configuration. It combines all the necessary ingredients to allow writing the vast majority of tests. What I like the most about Jest is that it ensures a proper isolation between individual test files. In addition, it parallelizes test runs across workers to speed up things.
If you consider migration to Jest, you rather start with a simple proof of concept. It will help you evaluate if the effort invested will bring expected benefits. There is no guarantee that it will perform better than exceptionally well fine-tuned Mocha configuration. I would also double check if it can be automated with jest-codemods. Manual refactoring can be very long and tedious for a larger codebase. Personally, I’m very excited seeing how rapidly Jest is evolving. It makes me confident that the number of reasons to migrate from any other existing testing solution to Jest will only grow in the near future.
Acknowledgments
This post was reviewed by Kenneth Skovhus.