Okay, let's face it. For most of deepstream's existence, a lot of you have been clueless about this feature we call 'listening'. Well, let's break it down in this post.
Listening as a feature, helps you to cut down unnecessary flow of data, which leads to increased efficiency and lower costs.
Let's understand listening by considering the following three scenarios:
Traditional data providers
Consider a scenario where you as a client need some realtime data, say weather data of a particular country. Conventionally, it would work as follows:
As seen above,
- The data provider continously fetches data from the realtime data store.
- The data provider then continously broadcasts this data to the server.
- As soon as the client requests for the data, the server starts sending it since the provider is already providing it.
Quite apparently, the server is under a lot of unnecessary load. One way to solve this is to implement a separate active data provider that sends only the required data to the server which will then be sent to the client. Let's see this next!
Implementing your own active data providers
As mentioned above, we can can build a mechanism that allows the data providers to send only the required data when any client requires it. The following may be one of the ways to implement this in practice:
In this implementation, all the logic lies in the controlling data provider. This data provider is responsible for sending only the required data to the server, thus not increasing the load on the server.
Whenever a client shows interest in certain data, the server asks the controlling data provider for it, which in turn fetches it from the cache or asks the other data providers to send this data.
This is just so complicated to implement and some of us would rather stick to some inefficiency and unnecessary data transformation in the backend rather than try implementing this whole thing.
Listening for data providers
Here's where listening comes in handy. Using listening in deepstream, you can skip this whole big step of implementing your own active data provider and instead start using it in your application out of the box.
With listening, you can make your data providers 'listen' to a particular pattern like 'weather/*'. These data providers will then start providing data only when atleast one client requests for it until there's no client who requires it.
Listening also makes it easy to implicitly add load balancing to your application by having multiple data providers listen to mutually exclusive subsets of data.
Since these data providers listen to a particular pattern, it is possible to control these data providers to make them send custom data which differs from client to client based on some criteria. Let's have a look at an example to understand this better!
Consider a data provider who is providing prices for products on Amazon. Using listening, you could make these providers send discounted prices to Amazon's favourite customers.
With listening, it's not just the data that's being optimally used, it effectively cuts down the cost as well if you are using paid third party data sources.
Feel free to go through this complete listening tutorial to learn more about how to implement it in your applications.