Calypso data kung fu – WordPress.com use case

This post is going to be used as a presentation during my upcoming talk at ReactJS Wrocław meetup. The event is planned for Thursday, September 29, 2016 at 19.00. You can find more details here.

Calypso

Calypso is the codename for a WordPress.com admin interface. This is what I wrote about it in one of my previous posts:

This is a universal (aka isomorphic) JavaScript single page app written in ES6 using webpack, express, React, Flux Redux, WordPress Rest API and many other front-end libraries.

Background

• Automattic – the company behind WordPress.com.
• The WordPress CMS codebase – the result of many thousands of people collaborating over 13 years.
• Company-wide development skill-set: PHP, jQuery and Backbone.
• WordPress.com REST API – common interface for web and mobile.
• Automattic acquired Cloudup in 2013, an API-powered file-sharing tool built with JavaScript.
• The initial commit was made in early 2014.
• Calypso became an open source in November 2015!
• Electron based desktop apps were released at the same time.

Stats

• About 96 million sites created by more than 110 million users.
• Over 400 million people view more than 20 billion pages each month.
• Users produce about 65 million new posts and 44 million new comments each month.
• Over 200 contributors on GitHub.

Kung Fu

The name of this talk was inspired by the song “Kung Fu” from the Polish hip hop band Kaliber 44. In particular this part:

To idzie tak:
Raz – świadomy wybór kierunku
I z punktu doskonalenie swego kunsztu
Dwa – osiągnięcie w nim poziomu mistrza
Godnego szacunku – to właśnie Kung Fu

The easiest way to express this quote in English is to use help of Wikipedia:

Kung fu is a Chinese term referring to any study, learning, or practice that requires patience, energy, and time to complete. In its original meaning, kung fu can refer to any discipline or skill achieved through hard work and practice, not necessarily martial arts.

Our Approach to Data

There have been three major eras of data management throughout the lifetime of development. Handling data has evolved to allow us to better adapt to the scale at which the application has grown.

This talk is mostly based on the documentation file created by my colleague Andrew Duthie.

Emitter Objects (June 2014 – April 2015)

Our original approach to managing data took an object-oriented approach:

• Instance of the store inherits from the EventEmitter mixin.
• Store breaks the Single Responsibility Principle – it allows to get current state, but also fetch data from the server or update data on the server.
• React component re-renders its content with the help of data-observe mixin.
• Store object is injected as a prop into React component.

Facebook Flux (April 2015 – December 2015)

We also tried to architecture our application using Flux that Facebook promoted at that time for building client-side web applications:

• Action creator dispatches view or server actions on the global Dispatcher object.
• Flux store subscribes to the Dispatcher, manipulating data in response to action types it is concerned with.
• Store exports a number of helper getter functions for accessing data stored in a private variable.
• Flux store inherits from the EventEmitter interface and emits a change event.
• Higher-order data component communicates with the stores and passes data down to the wrapped component using another HOC – StoreConnection.

Redux Global State Tree (December 2015 – Present)

The factors why we decided to back off from Flux are listed below:

• Strong need for offline support in desktop apps.
• The struggle with server-side rendering implementation.
• Growing number of Flux stores became hard to manage and scale.
• Unit testing was very complex and error prone.
• Redux became production ready and highly promoted in the React community.

Our Redux architecture is based on the recommendations shared by Dan Abramov in his video tutorials published on Egghead:

• Getting Started with Redux
• Building React Applications with Idiomatic Redux

Global state

• The root module of the state directory exports a single reducer function.
• combineReducers is used to separate data concerns into their own piece of the overall state.
• The files mirror the structure of the global tree.

Actions

• An object describing an intended state mutation.
• All action types are defined in state/action-types.js.
• Action types are considered global – any reducer can react to any type.

Action creators

• A function that returns an action object.
• Bounded action creators help to deal with async code using redux-thunk middleware.

Reducers

• A function that, given the current state and an action, returns a new state.
• It’s quite common that the subject reducer is itself a combined reducer.
• createReducer abstraction helps to remove boilerplate code (example).

Selectors

• A helper function for retrieving data from the state tree.
• The entire state object is passed to the selector.
• Data normalization – minimize redundancy and to avoid synchronization concerns.
• Selectors should always be pure functions.
• Caching concerns on selectors can be overcome by using memoization techniques – similar to reselect library (example).

Data components

• Ensuring that data is available, is one that we’d wish to eliminate (Relay?).

• Presentational and Container Components.

• Query components:
  • Responsible for dispatching the actions that fetch the desired data.
  • They neither accept nor render children.
• App components:
  • It wraps a visual component, connecting it to the global application state.
  • react-redux library assists in creating bindings.
  • We create purely presentational components whenever possible.

Offline Support

Implemented only in a limited range due to existence of legacy code.

Data persistence

We persist the Redux state to the browser’s storage using LocalForage (IndexedDB, WebSQL or localStorage) to avoid rebuilding the Redux tree from scratch on each page load.

• On page load Redux store get created with that persisted initial state.
• Every time state changes the storage is updated with new data.
• SERIALIZE and DESERIALIZE actions help to handle data that can’t be easily saved into storage (e.g. Immutable.js).
• These special actions are not dispatched.
• DESERIALIZE action should use a JSON schema file (example) to detect data shape changes using is-my-json-valid library.
• Data is stored only up to 7 days.
• Some subtrees may choose to never persist data (default setting).

REST API caching

We also added a sync-handler wrapper for REST API client. It works like a middleware that intercepts every GET request as follows:

• If data is already cached then return it immediately.
• Send original request to the REST API server.
• Update cache with fresh data.

This approach allows to serve data even when the user is offline.

Server-side Rendering

It is great for SEO and also as a progressive enhancement. React is able to detect HTML code generated on the server when the client code is executed!

• There is a special set of constraints to follow when building components and libraries (/** @ssr-ready **/ pragma).
• Initial render needs to be synchronous.
• This technique works only for logged out users for the time being.
• Data is exposed from the server to the client with the JavaScript variable initialReduxState set in the HTML source.
• Data passed from the server gets merged into the client’s Redux initial state.
• Special action  SERVER_DESERIALIZE allows to validate data and format it properly.
• Example page: https://wordpress.com/theme/wayfarer.

To find out more check this document.

Going Forward

On our roadmap:

• Move everything to Redux (related issue).
• Replace redux-thunk middleware with our own data layer middleware (related PR).
• Implement full offline experience.

Wrapping Up

I hope you enjoyed it! I encourage everyone to improve their Kung Fu and contribute to Calypso. I’m always happy to help and answer any questions.

References

1. Automattic/wp-calypso – Calypso public repository.
2. The story behind the new WordPress.com – Automattic’s Engineering blog.
3. Calypso: Our Approach to Data – documentation authored by Andrew Duthie.
4. Calypso: Server-side Rendering – documentation.
5. Contributing to Calypso – guidelines.