Want to send a request to a server and get a response back? Use HTTP! Simple as that. But when it comes to managing communication via persistent, bi-directional connections, e.g. WebSockets or TCP you’re looking at one heck of a choice.
This post hopes to shed some light onto the world of MQTT, XMPP, STOMP, AMQP, WAMP and all sorts of proprietary alternatives (plus paves the ground for the next post, explaining why we choose to build our own). In this spirit, here’s to one of the most often referenced XKCD comics: (I can’t believe we’re at this point yet again :-)
Hold on, there’s no such thing as a “realtime protocol”!
That’s true; but I’m using the term as a collective noun for protocols that are designed to deliver messages, synchronize data and facilitate request/response workflows over persistent, bidirectional, connections — e.g. WebSockets or TCP.
To gain an overview, let’s structure them by their purpose:
Low(er) level protocols such as TCP are designed to deliver one message from one sender to one receiver. They have no opinion on how this message should be structured, requested, retrieved, secured or stored. They are just the mailman that brings the mail.
Protocols like WebSockets sit on top of TCP and add additional functionality, such as headers to transport meta-data, splitting larger messages across multiple packets, basic authentication mechanisms or routing/redirection information — but at the end of the day they, are still just a way for two points to exchange data with each other.
When it comes to building larger, more complex systems you want some method to your madness; you want a higher level paradigm:
Publish-Subscribe (also called an observer pattern at times) is one of the most common ways to facilitate stateless(ish) messaging and many-to-many communication within larger systems. “Subscribers” can subscribe to “topics”, (sometimes also called “channels”, “events” or “namespaces”), “publishers” can publish messages to these topics that will be routed to all interested subscribers.
This paradigm is enormously flexible, efficient and scalable. It elegantly decouples senders from receivers and allows individual clients to join and leave topics as their interests change. As an analogy, imagine a newspaper:
readers (subscribers) open the politics, sports or weather sections (topics), editors publish articles to them. Whilst publishing, the editor doesn’t write for a specific reader — they write for general consumption. Likewise, the reader doesn’t expect a specific article — they open the section due to general interest in the topic and consume whatever has been published to it.
There’s a number of protocols that facilitate pub-sub messaging: MQTT, STOMP, WAMP or Pushers and PubNub’s protocols to name just a few. So which one should you choose?
The Message Queue Telemetry Transport is a binary protocol that’s predominantly used to facilitate communication between servers and low-powered IoT devices.
It sits on top of TCP and provides publish/subscribe functionality as well as some additional bits, e.g. delivery guarantees (at least once, at most once etc.), storing the last “good” message or registering hooks that ought to be called upon connection loss.
It is very lightweight and explicitly designed to work in unstable and unpredictable network environments.
So when would you use it?Definitely for IoT, but there’s no reason not to use it everywhere. Clients are available for pretty much every language and with the advent of binary support even browsers can send and receive MQTT messages via WebSocket.
The Streaming Text Oriented Messaging Protocol is a text-based standard used predominantly for WebSocket communication. On top of the usual pub/sub semantics it offers guaranteed message blocks via a begin/publish/commit sequence as well as positive and negative acknowledgements.
It is explicitly designed to be easy to implement — and as a result there’s a wide array of clients for almost every programming language. On the flipside it is a bit heavier in terms of message sizes and interpretation speed.
So when would you use it?Stomp is a good choice when it comes to pub/sub messaging for browsers. When used in conjunction with a multi-protocol broker such as Apache Apollo it allows for interesting combinations, e.g. displaying updates from IoT devices via MQTT in a browser chart.
Binary or Text Based?
So far we’ve looked at two protocols: A binary one and a text based one. Time to take a quick excursion to understand the difference:
Information between computers is exchanged by turning the light or electricity in a cable on or off or by switching between peaks and centers of a WiFi signal.
It is — by any means — binary. As a result, all protocols are binary protocols.
Once information is transmitted, the recipient has two choices — it can derive information directly from the streams of 0 and 1 it has received by grouping them into bytes— or it can perform an extra step, cast it into text and then interpret this text.
The former approach is called binary. It saves a number of expensive operations and is the standard when it comes to transmitting any non-textual information, e.g. images, sounds, videos or complex files. It can also be used to transmit text-information, e.g. by prepending a few bytes to every message to convey meta-information, e.g. message length or type and only cast the actual message data to text.
However, as a lot of information exchanged in a pub-sub architecture is text based anyways, a lot of protocols choose to simply cast the whole information to text — thus reducing complexity and increasing readability for humans at the expense of performing an extra computational step upon message receipt.
The Web Application Messaging Protocol (not to be confused with the Windows/Apache/MySQL/PHP Stack) is an attempt at developing an open, text based standard that combines Publish/Subscribe with Request/Response patterns as well as complex routing and delivery strategies. It is most commonly used in conjunction with the crossbar.io router and autobahn client libraries.
Whilst architecturally sound, it never reached the popularity of the other, more purpose build protocols.
Pusher / PubNub & Co
Realtime platform-as-a-service offerings such as pusher or pubnub use their own, proprietary protocols. Pusher released detailed specifications around their JSON based protocol, encouraging third parties to build clients. Pubnub is a bit more closed, but supports a range of open protocols such as MQTT to allow for interaction.
Sometimes, simple Pub/Sub isn’t enough. That’s where queues come in — they provide a wide variety of messaging and storage patters. In their simplest incarnation queues are just a way to send messages from a “publisher” to a “consumer “— with the difference that the consumer can receive messages at its own pace.
This is usually achieved using a retrieve/acknowledge strategy. A consumer takes a number of messages of the queue, acknowledges their receipt, processes them and then takes the next batch of the queue. This is an incredibly powerful approach for scalable backends. Imagine you’re building an Instagram clone that allows users to upload pictures and apply filters to them. Applying filters is a computationally expensive operation — so rather than doing it directly upon upload, the server receiving the image just makes a note of its location within the file system and what kind of filter needs to be applied and pushes this note onto a queue.
Another computer that’s good at applying filters eventually takes this note of the queue, loads the image, processes it and acknowledges that the task has been completed. To scale your image processing capabilities all you have to do now is to add more consumers to your queue.
This pattern can be extended to allow for complex routing, task allocation and even request/response workflows. But because it is so powerful, queuing protocols tend to be heavier than their pure pub/sub rivals:
The Advanced Message Queuing Protocol is the juggernaut of the queuing specs. It is a binary messaging spec implemented by a wide variety of brokers such as RabbitMQ or HornetQ and allows for numerous routing and distribution strategies.
So when would you use it?Whenever you have requirements that go beyond simple publish-subscribe messaging between components. AMQP is reliable, proven and feature-rich. What it is not though is particularly fast or lightweight. If you are looking for a slim alternative you might be better off with:
ZeroMQ is both a protocol and a combination of components implementing it. It aims to be a faster, decentralized alternative to AMQP.
So when would you use it?When you are looking to achieve complex workflows with higher speed and no central point of failure — and are prepared for a surprising amount of learning and complexity.
The Java Message Service is a combination of a protocol and the components implementing it. As part of the Java Enterprise Edition (JEE) it is used to abstract inter-service communication behind a Java-y interface.
So when would you use it?When you are using Java and are bought into the Java Enterprise Platform already. If not, you might have an easier time just using the Java client for one of the alternatives mentioned above.
Even with streaming data sometimes all you want to do is ask a single question and receive a single response. This can be done using an HTTP request — but then again you do already have a connection with your server open — so why not just use that?
The process of request/response communication over such a persistent connection is usually referred to as RPC (Remote Procedure Call) or RMI (Remote Method Invocation) — it is supported by protocols such as AMQP or ZeroMQ via response-queues, JMS via Java RMI, WAMP or dedicated RPC frameworks/protocols, such as gRPC or Apache Thrift.
DataSync is the newest addition to this list of realtime communication concepts and associated protocols (although some would argue that approaches such as CORBA paved the way for it).
DataSync synchronizes the actual data in a data-store with clients. Any change that’s made by a given client is stored and shared with all interested subscribers. DataSync is extremely powerful as it hides all the complexities of maintaining state in realtime apps away, reduces complexity and speeds up development by magnitudes, but it is also a fairly new approach with no open protocol standard.
DataSync, Pub/Sub and Request/Response are incredibly potent patterns, but while there are plenty of establish standards for pure pub/sub or queuing, open protocols for request/response or most importantly data-sync are still largely unavailable.
With deepstream’s DRP (distributed realtime protocol) we are confident to have found a solid approach that combines these concepts into a single protocol whilst ensuring the maximal efficiency in terms of message size, extensibility and interoperability. I can’t wait to share more details in the following post.