How do three people build a scalable backend in a language only one has ever used before in just three months? By getting something else to do the heavy lifting.
(Spoilers: Paul learned Scala, you can too; Ryan convinced Paul to switch to Scala, you can convince your boss too; Akka is a powerful tool for doing distributed-anything in Java/Scala environments)
When Conspire graduated from TechStars, we started planning for our revamped product and we knew that meant a lot of new code. The old codebase was pretty hairy and we weren’t proud of it and definitely weren’t sad to see it go. We had a few requirements for our new backend:
- Scalable (elastic)
- Simple to understand
We’d had some great success with Akka building parts of our backend in the prototype stage and we were excited about some of its coming features—especially clustering—and decided to use it as the basis for our backend. Akka had already helped solve some of our scalability issues and was rock-solid for us during prototyping (we’d built a precaching service with Akka to solve some serious performance problems).
(Shameless plug: To join Conspire and get a personalized weekly update about your email relationships, sign up at goconspire.com.)
We were less certain about Scala. I’d been using it for about two years and loved it but the rest of the team was less sure. Akka can be used from Java quite well and indeed, Java was our plan. We already used Scala in a few minor spots of our protoype’s code base but neither Paul nor Alex had any experience with Scala so we “agreed” to use Java. I, err, reneged on that once I got fed up with Java’s verbosity. (This turned out to be a good decision)
Our backend is completely decoupled from our frontend. Not only does this design simplify the number of code paths through our backend, we can take it down (or watch it fail) and our users won’t notice. When you’re building your startup on a stack that’s only a month old and that none of your team has production experience with, this is a big plus.
We divided our backend into three siloed sections: IMAP processing, analytics and mailing. These are overseen by a supervisor. Each section is designed to be independent of the others, communicating only via the supervisor. Reducing the cohesion reduces concerns about scalability and resilience—worker nodes don’t need to know anything about other worker nodes and therefore don’t care if another fails.
We take a KISS approach to persistence: convert it to JSON and throw it on S3. Given our use case, we don’t have any need to query individual pieces of data for users or across multiple users. We load all of a user’s data, process it and write the results back out. For a given user we easily end up with 50-100MB of data and S3 gives us a cheap, fast and reliable storage service without worrying about scalability.
Given this architecture, what does Akka bring to the table?
- Fault-tolerance and resilience: any given node can fail without taking down the entire backend
- Scaling out is as simple as spinning up a new EC2 instance and having it join the cluster—Akka will start using it as a worker node for whatever role it’s registered with
- Actors: a concurrency model that’s easy to reason about (more on this in the next post)
We built our backend over the last three months. Originally the bulk of the code was to be in Java but I slowly started writing more and more in Scala. Paul and Alex were reluctant but Paul gave Scala a shot. His first time writing Scala was in building out the first pass of our new analytics service. In one sitting, he went from no prior Scala experience to an analytics service that worked correctly the first time it compiled.
I didn’t just shanghai the rest of the team into using Scala. I used a two-pronged attack. I began by implementing small things in Scala to demonstrate its big benefits, putting Scala in perspective by showing our it made our code better. That’s very important with a langauge as “scary and complex” as Scala. While doing that I piqued their curiousity with links and articles on the benefits of Scala, such as Jonas Bonér’s excellent talk on "Going Reactive" and Twitter’s own Scala School. The simplicity of writing typesafe, asynchronous code in Scala is a huge benefit when building a backend like ours.
In the end, we’re very happy with where our backend wound up. Akka takes care of the heavy lifting of our dual requirements for fault tolerance and elastic clustering. We had some frantic moments where we were afraid Akka wasn’t going to live up to its promises but we worked through those.
We’ll be going into more detail in future posts—this is the first in a series of five. Overall, we’ve been very pleased with Akka despite some heated moments with it in the last few weeks.
The rest of our series on Akka at Conspire: