squbs: A New, Reactive Way to Build Applications.

A Mini-Introduction to squbs

A new stack called “squbs”(spelled in all lower case with the pronunciation rhyming with “cubes”) makes use of the loose coupling already provided by actors. It creates a modular layer (for your nano-services) called “cubes” that are symmetric to other cubes. Unlike libraries with concrete dependencies at the API layers, cubes ride on the actor system and only expose the messaging interface already provided in Akka. The interdependency between cubes are loose and symmetric. It is not hard to see the roots of the name “squbs” from these concepts and properties.

There are only a few principles coming together for designing squbs:

1. It must be extremely lightweight with no measurable performance deficit over a similar Akka application built from scratch.
2. New APIs over the Akka APIs are based on absolute necessity. Developers should not need to learn any squbs API or message protocol to build services or applications on squbs. The knowledge base needed to build squbs applications should be the Akka knowledge base which developers can acquire from training, documentation, and forums available on the Internet.
3. It should be open source from the core up, with hooks for plugging in PayPal operationalization that cannot be open sourced.

With this, squbs became the standard for building Akka-based reactive applications @PayPal.

Culture & Language

Programming to the reactive, functional landscape is very different from the traditional Java programming we have done for the last 20 years. It requires immutability, leaving behind Java Beans and nulls and most mutable APIs in the Java ecosystem, adopting error containers like the Scala Try (a Java version is available in open source), and an ultimate awareness of dangerous function closing-overs and any blocking behavior.

While squbs supports both Scala and Java use cases, Akka APIs are clearly more suitable to their native Scala ecosystem. We also debate whether it is easier for an engineering team with Java background to stay with Java and adopt a culture very different from how we have programmed Java for the last 20 years (empirical programming with pervasive mutability) and try to set our own culture, standards, and guidelines against what Java programmers are used to do, or have them learn Scala and adopt the pre-existing Scala culture well suited for this programming model, with a lot of support libraries that are readily immutable and functional.

The culture adoption is hard to describe. Before teams get their hands dirty, the ultimate answer was almost always Java. It should not be to any surprise as it is their bread and butter language. There is also a resistance from management and architects that do not get their hands dirty. It is the resistance to any change, a resistance to do what you know and not go beyond. Any such change is perceived to introduce a tremendous risk to their projects – in many cases not even knowing they are already trying to create a sub-culture just under the Java brand name – a set of people who speak Java in a very different dialect and will have to maintain this functional, reactive dialect of Java in each of their teams.

For teams and members that actually do get this far and get their sleeves rolled up, they tend to see very quickly their work is much easier when using Akka’s native tongue: Scala. The libraries, the culture, and all they need is readily available. It feels natural, with no tweaks. Also, learning Scala to the point of building reactive services is barely scratching the surface of the ocean depths of Scala.

Programming Patterns

How do you take a programmer who only knows how to write linear code, and make them build high performance, actor-based systems? Since we do not have the luxury to hire the top 5% ubercoders, we have to make sure our developers can be trained to do the job.

Luckily, a vast majority of services do similar things. They receive requests or messages, make database calls to read/write the database, make other service calls, call a rule engine, fetch data from cache, write to cache, all in combination. Here goes our micro-service that does one thing. Some others have one or other forms of stream processing, sometimes ETL.

Because it is useful to create some common patterns that teams can readily adopt, we defined and built our set of programming patterns, which over time, will manifest as application templates. It allows for developers to see the problem and marry it to a well-defined and well-studied pattern ensuring short-term success to their projects.

A common pattern is the “Orchestrator Pattern” which orchestrates requests or messages by talking to a multitude of resources, all asynchronously. These resources may be in the same system, with a manifestation as an actor, another cube which just happens to be on the same address space, or a remote resource altogether.

Another trait we have is graciously called the “Perpetual Stream”. It is no different from just Akka Streams, except that it encourages very long-running streams that will start with the service and stop when the service instance gets stopped. Providing this pattern and utility in squbs allows for streams to hook into the system state and ensure no messages are dropped at shutdown. It provides an additional benefit of modeling the whole flow of the service in a central place, providing a clear oversight and understanding of the service’s functionality.

Results

The adoption of Akka and squbs have already provided very high-scale results. With as little as 8 VMs and 2 vCPU each, applications were able to serve over a billion hits a day. Our systems stay responsive even at 90% CPU, very uncharacteristic for our older architectures. This provides for transaction densities never seen before. Batches or micro-batches do their jobs in one-tenth of the time it took before. With wider adoption, we will see this kind of technology being able to reduce cost and allow for much better organizational growth without growing the compute infrastructure accordingly.

Needless to say, Akka and squbs are still players at the “infrastructure” level. squbs is an open source project by eBay and PayPal. It was designed to be open sourced from the very beginning and is free from pollution and deep library dependencies. We believe the customization hooks allow squbs to fit into any operational environment and would benefit any organization who wants to adopt Akka-based technologies in a larger scale.

Visit us at https://github.com/paypal/squbs and leave your comments and questions on the Gitter channel you find on this github site.