Monday, May 25, 2015

The point of my work is not to re-invent Flow-Based Programming nor to re-invent visual programming (for a really good list of visual programming languages, visit this website). Rather, I want to explore how one can take advantage of the inherent concurrency of Elixir/Erlang processes. Impedance mismatch, when used in software development, is how hard it is to make something behave like something else. There seems to be less of an impedance mismatch with the idea of implementing flow-based or streaming systems in Elixir.

ElixirFBP is showing some life! Last week I was able to create a small graph consisting of an adder and an output component. This is what the code looks like:
defmodule Examples.Add1 do
  alias ElixirFBP.Graph
  alias ElixirFBP.Network

  @graph_1      "graph_n1"
  @node_1       "node_1"
  @node_2       "node_2"

  def start do
    {:ok, fbp_graph_reg_name} = Graph.start_link(@graph_1)
    # Add the components to the graph
    Graph.add_node(fbp_graph_reg_name, @node_1, "Math.Add")
    Graph.add_node(fbp_graph_reg_name, @node_2, "Core.Output")
    # Connect the components
    Graph.add_edge(fbp_graph_reg_name, @node_1, :sum, @node_2, :in_port)
    # Set the initial values
    Graph.add_initial(fbp_graph_reg_name, 42, @node_1, :addend)
    Graph.add_initial(fbp_graph_reg_name, 24, @node_1, :augend)
    # Start the flow
    {:ok, _fbp_network_pid} =
        Network.start_link(fbp_graph_reg_name)
    Network.start()
  end
end
The add component is initialized with the two values: 42 and 24. The output of this component is fed into a component that simply prints its message. Granted, this is not the way one would do a little bit of arithmetic but it does test many of the ElixirFBP design elements. For a more interesting demo, please read on.

I recently gave a presentation of my ideas at a Meetup of the Portland, ME Elixir group. I got some great feedback and pointers to three systems that were worth looking at:
  1. Apache Storm
  2. Apache Spark
  3. Streamtools.
Streamtools is being developed by the R&D group at the New York Times. They have designed a flow-based system that features a graphical front-end - similar to NoFlo's - and a backend that executes components written in the Go language. They have some interesting demos - more than the addition of two numbers! One, a poller, periodically checks the status of a bike share program in New York City. I decided to try and implement this demo using ElixirFBP. Here's what it looks like:
defmodule Examples.Citibike do
  alias ElixirFBP.Graph
  alias ElixirFBP.Network

  @graph_1      "citibike"
  @node_1       "ticker"
  @node_2       "map"
  @node_3       "getHTTP"
  @node_4       "unpack"
  @node_5       "filter"
  @node_6       "output"

  def start do
    {:ok, fbp_graph_reg_name} = Graph.start_link(@graph_1)
    # Add the components
    Graph.add_node(fbp_graph_reg_name, @node_1, "Streamtools.Ticker")
    Graph.add_node(fbp_graph_reg_name, @node_2, "Streamtools.Map")
    Graph.add_node(fbp_graph_reg_name, @node_3, "Streamtools.GetHTTPJSON")
    Graph.add_node(fbp_graph_reg_name, @node_4, "Streamtools.Unpack")
    Graph.add_node(fbp_graph_reg_name, @node_5, "Streamtools.Filter")
    Graph.add_node(fbp_graph_reg_name, @node_6, "Core.Output")
    # Connect the components
    Graph.add_edge(fbp_graph_reg_name, @node_1, :out, @node_2, :in_port)
    Graph.add_edge(fbp_graph_reg_name, @node_2, :out, @node_3, :path)
    Graph.add_edge(fbp_graph_reg_name, @node_3, :out, @node_4, :in_port)
    Graph.add_edge(fbp_graph_reg_name, @node_4, :out, @node_5, :in_port)
    Graph.add_edge(fbp_graph_reg_name, @node_5, :out, @node_6, :in_port)
    # Set initial values
    Graph.add_initial(fbp_graph_reg_name, 10_000, @node_1, :interval)
    Graph.add_initial(fbp_graph_reg_name,
                      "http://www.citibikenyc.com/stations/json",
                      @node_2, :map)
    Graph.add_initial(fbp_graph_reg_name, "stationBeanList", @node_4, :part)
    Graph.add_initial(fbp_graph_reg_name, "stationName", @node_5, :filter)
    Graph.add_initial(fbp_graph_reg_name, "W 41 St & 8 Ave", @node_5, :filter_value)
    # Start the flow
    {:ok, _fbp_network_pid} =
        Network.start_link(fbp_graph_reg_name)
    Network.start()
  end
end
To run this you need to have Erlang and Elixir installed - see here. Then:
  1. Download the ElixirFBP code from github.
  2. Open a command window at the ElixirFBP directory level.
  3. You may have to perform a mix deps.get first in order to load some libraries that I use.
  4. Run iex -S mix 
  5. Type the following at the iex prompt: Examples.Citibike.start
  6. Wait about 10 seconds. You should start seeing messages appearing roughly every 10 seconds; the messages will likely be the same, unless someone has rented a bike in the last 10 seconds!
  7. Stop the execution by pressing Ctrl+c, twice. 
  8. You can run the Add example by entering  Examples.Add1.start
The code is still a little rough. Note that the Streamtools flow components are JSON-oriented. In my implementation, once a JSON response is received, it is converted to Elixir data - maps and lists. They also feed the final result into a mask component that then connects via web sockets to a browser. I did not implement this step in the flow.

Soon I'll start connecting NoFlo's graphical front end to ElixirFBP. Then you'll be able to create and run these demos using their GUI.

2 comments:

  1. In regards to systems worth of checking out, the FBP system I've been playing around with the most (a lot of fun!) is GoFlow: https://github.com/trustmaster/goflow .

    ReplyDelete
  2. ... but now my next system to play with might be ElixirFBP :)

    ReplyDelete