Friday, August 21, 2015

"Do a Something!"

I have been spending a great deal of time worrying about possible overruns in an ElixirFBP flow. Overruns are possible when the producer of Information Packets (IPs) sends them to a component that can not process them quickly enough and, as a result, all kinds of terrible things can happen. Even though Erlang/Elixir is designed from the ground up to be a message passing system, there is the very real possibility of a process's mailbox of messages becoming full.

This possibility lead me to read (and blog) about Reactive Streams and I have attempted to implement the design outlined in the Reactive Stream specification. What has bothered me about this approach is that it adds an extra processing step between any two flow components. This is the Subscription that I've described earlier. Adding a step is. of course, going to slow things down - it would be much nicer not to have this extra processing time added on. But, it doesn't appear that this is possible; something has to be monitoring the size of component's mailbox.

As part of my research into solutions to this problem, I spent the last couple of days watching a series of presentations given at the 2013 Erlang conference. The presentations were part of a track entitled: Load Regulation and Back Pressure. Ulf Wiger's 'Jobs' Load Regulation Framework was very interesting. Also, Bryan Fink's Riak Pipe.

What now? Well, I know that there are potential problems, especially if ElixirFBP ever is used in an environment such as real-time monitoring. And, I also know that there are solutions out there. But, I haven't been making much progress in the last few weeks. I think I might be suffering from Analysis Paralysis. As an old boss of mine once told me: "Do a Something!". So I will.