A case study about delaying and buffering data transmission to fix performance issues
This is about an issue that was encountered a few weeks ago in a side project and how it was fixed.
This is about an issue that was encountered a few weeks ago in a side project and how it was fixed.
The issue
The issue is about a software being unable to handle real-time monitoring events fast enough and ending up being overwhelmed by this heavy load
Photo deLuis VillasmilsurUnsplash
The context of the issue
The platform project helps to execute tasks organized in a Directed Acyclic Graph like you would with Apache Airflow.
Without saying too much about it, there is:
- a back-end, written mostly in Python at the moment,
- a front-end in ReactJS,
- they communicate with each other using both REST API and WebSocket,
- in some cases, heavy processing is delegated to worker instances using Celery+RabbitMQ.
To monitor the progress of tasks, events are sent to the front-end using WS. Those can be the running status of a task, the value of a variable, the overall progress indicator, … It isn’t about knowing all what happened, but about having a view of the current state of the processing.
When tasks are running on a worker, events are first sent from the worker to the back-end and there they are relayed to the front-end (so no need to have direct access to workers).
I hope you followed so far.
So, where is the catch?
The catch is that when the worker has far more processing power than the the back-end, it can easily overload it by sending events faster than they are processed and it lead to a self-induced deny of service.
It occurred in a use case where a very fast task is used a lot of time in a process.
The resolution of the issue
As for any issue solving, we need first to determine what happens, when it occurs and how to fix it or provide a way to prevent getting in the situation where it occurs.
The analysis
The main reason we relay events from the worker to the coordinator is to be able to monitor the progression of the task remotely. So as a matter of fact do we really need to be notified of all events that occurs in real time?
The answer is no, we only need to have latest values.
ie: I don’t need to know that the progression was 0.80 and 50ms after 0.81, only that the latest progression value was 0.81.
It isn’t an issue skipping some event as long as the latest sent event was nearly in real time.
First part of the solution — sending only updated values
Well, the first part of the solution was already in place before the issue was encountered. It is to prefer sending delta values instead of full payload whenever possible. But we should keep the possibility to send directly the full payload when a new client connect to the software/back-end.
What I mean is when the internal state was x=1, y=1 and then an event occurs and now y=2, there is no need to send both x and y to a client that already know that the previous values were 1 and 1. (But we would send both x and y to a new client that would connect afterward.)
It implies that the part that receive the data should be conceived to work with both full and partial data.
Second part of the solution — delaying/skipping transmission
For the second part of the solution, it is based on the assumption that for each event that can occurs another one might occurs very shortly after and change the value of the same variable.
If the first event set a progression of 0.80 and 10ms later the second one set a value of 0.81, it wouldn’t be a problem not having sent the first 0.80 value, would it?
So instead of transmitting data each time an event occurs:
- we aggregate the content of all events in two buffer, one that is never cleared, to keep track of the full payload to send to new subscribers — one that is used for delta transmission and is cleared after each transmission.
- we launch another thread (if one wasn’t already running) that will wait for 300ms before sending the content of the delta buffer and then empty it.
To conclude
This solution did fixed the issue on this project.
No source-code was provided here because it isn’t something linked to a language in particular, and the article is more about proposing one way to fix a potential issue than about giving an absolute answer to a problem. (There might be other and even better ways)
Thats’s all folks!