Using Scala and being fond of functional programming techniques at The Guardian, we were really looking forward to the Code Mesh lineup this year. We were especially excited about bringing more functional languages to client side development, so we were looking closely at the Elm and PureScript talks, as well as the more familiar Scala talks.
What follows is a quick round up of the talks we saw and things that interested us.
Conflict Resolution for Eventual Consistency
As progressive web apps, service workers and an offline-first approach become the latest and greatest thing, we need systems that can avoid the performance and connection issues of network dependence.
When multiple users can access and edit a document concurrently, conflicts will arise. At the Guardian, our editorial tools include a fairly extensive content management system, therefore the insights from Martin Kleppmann - as he discussed the different approaches and algorithms for conflict resolution - were incredibly helpful.
Eventual consistency requires delivery and convergence; a message should be relayed within a finite period, and if data diverges, it needs to converge again.
A system of operational transformations with unique IDs is one solution to concurrent operations; the unique ID can be based on the node ID and and an incrementing transaction ID.
How do we design APIs for concurrently editable data, that are not horrendously confusing?
However, you will still need to decide which operation ‘wins’ in some scenarios (according to business requirements), and deleting operations remain tricky! Martin explained an approach where a character is marked as ‘deleted’, yet the ID is kept around as a tombstone, so that it can still be referenced.
Martin also gave an excellent talk at Code Mesh 2015 on Transactions: Myths, Surprises and Opportunities.
Load balancing is Impossible
Why do websites use load balancers? Abstraction and failover are the two main reasons, then finally, balancing load, which is more often about load distribution than load balancing. Tyler McMullen, CTO at Fastly, has been modelling load balancing and request latency. This talk was of particular interest, since Fastly is the Guardian CDN.
Efficient load balancing is an exceptionally hard problem. Most load balancers talk to origin servers using a ‘round-robin’ or ‘randomize’ approach. However, random load balancing is detrimental to latency due to the high variance in random allocation. This talk was an exploration of other load-balancing methods.
If you are not seeing latency issues, you are probably over-provisioned
At a scale where multiple load balancers are required, load balancing becomes even more challenging! Distributed load balancers are hard, and distributed ‘join-shortest-queue’ is a nightmare. What does work quite well in modelling is ‘power of two choices’, where the load balancer picks two nodes at random, then applies the ‘join-shortest-queue’ choice. The final option covered was ‘join-idle-queue’; where available servers ask for a task from the load balancer.
Quaich: A “Serverless” Microframework for Event-driven Scala Programming on AWS Lambda
This talk was looking at the Quaich (pronounced ‘quake’) AWS Lambda library by Brendan McAdams.
This library simplifies the marshalling of AWS Lambda JSON events for various event types. This reduces a lot of the boilerplate code when working with HTTP requests for API gateway, and implements a scalatra-like interface for defining routes etc. We use Lambda a lot at The Guardian, so I’m looking forward to seeing support for other events types (S3, DynamoDb).
As an aside, our own Simon Hildrew is working on Plaλ (pronounced ‘plambda’) which attempts to wedge the Play! framework into Lambda. We also have an exciting list of Lambda tools and examples.
Distributed Jamming with Sonic Pi and Erlang (Joe Armstrong and Sam Aaron)
Has there ever been a better keynote than Joe Armstrong, co-inventor of Erlang, talking about Black MIDI, musical theory, and experiments with Erlang and Sonic Pi? We experienced the joy of live coding 19821 parallel processes (each representing one note) and discovered the pitfalls of process timing. As the after party kicked off, Sam Aaron continued the live coding with Sonic Pi.
Streaming, Database & Distributed Systems: Bridging the Divide
An exciting and informative whistle stop tour of stream processing approaches and problems. Ben Stopford showed us the analogs between streams and traditional table structures in terms of architecture, and then went further into the pitfalls of stateful stream processing. This was a lot to take it at 9am in the morning, but we do a large amount of stream processing as part of our analytics platform using AWS Kinesis and Lambdas. The ideas presented here should hopefully help us out in future stateful processing needs!
Web Programming without Errors, and Coding without Typing
This was an impressive, high energy presentation about Elm! Jessica Kerr did a fantastic job live coding her way through a Elm app for adding diagram annotations and gave a great introduction to Elm, it’s processing lifecycle, and how its static typing reduces runtime errors.
Jessica also did most of the presentation using atomist, a tool that assists project creation and evolution. It does this by using predefined macros for automating the management of code that may be boilerplate, or need to be duplicated across multiple locations.
You can also see Jessica in action at Elm-conf.
Building a Graphical IDE in Elm/Purescript for an Embedded Language that Compiles to the Erlang VM
More Elm in this talk; Claudia Doppioslash took us a through an experience report on her IDE for developing software on embedded devices. She did a very thorough comparison between the different Elm versions, a great comparison between the various languages that compile to JavaScript, a good overview of PureScript and the various libraries and frameworks that she has tried out.
Flexible Paxos: Reaching Agreement without Majorities
Heidi Howard gave a brilliant talk about constructing new fault-tolerant systems, with sufficient server personification to make the talk highly memorable.
Majorities are not necessary to safely reach distributed consensus.
Heidi already has a detailed write-up of the key points on her blog, which I recommend reading.
Final keynote: Joe Armstrong Interviews Alan Kay
I feel like this section could just be a selection of aphorisms from Alan Kay:
Computing is like a pop culture, it is not interested in the past
By the time a language gets in good shape, the language has become obsolete
You can only be proud of the fact that you did it, you can never be proud of the thing
Science doesn’t give you science, something like the Royal Society gives you science; you have to have other people to debug your brain
Alan Kay confessed he used to be more of a page and paragraph person, until he worked at Xerox and needed to catch the attention and imagination of managers for stuff to get done.
Besides some amazing quotes, we also got a crash course in industry history as Alan talked through the evolution of programming. Conversation then ranged from music and philosophy, to biology.
Elsewhere
If the monads, databases and lambdas became too much, Code Mesh also had a great track on the history and the philosophy of computer science. After a packed room last year for his talk on the Apollo guidance computer, Brian Troutwine took the main stage to give us ‘A History of Space Stations’. Another fantastic talk was ‘Before Unix: An Early History of Timesharing Systems’ by Mark Allen.
Conclusion
All in all, it was an enjoyable conference, with a passionate group of attendees and speakers. We will be back next year!
Videos of all the 2016 talks should be online soon. The Code Mesh 2015 talks can be found here: https://www.youtube.com/user/ErlangSolutions.