Simple Pub/Sub with Faye

The Scenario

One app hosts data which it provides to consumer via a simple REST api. This data contains references to resources outside of the database (read: image urls). We’ll call this app the provider. A consumer app can pull this data as necessary to render it’s own pages. Those pages may include direct links to these external resources (again: image urls). We’ll call this app the consumer. Given the simplest solution, the consumer calls the provider whenever it needs more data. This introduces a bunch of latency. No problem, right? Simply cache that data on the consumer side and only call periodically. Knowing the rough rate of data updates on the provider, we can expire the cache at some reasonable interval that things pretty much work. But… only pretty much. As soon as one of the image urls changes on the provider, and the previous version is no longer available, we find the client app trying to render images that are no longer accessible because it’s using its cached (now stale) version.

The Solution

In order to keep the apps somewhat disconnected, I wanted a simple way to tell the consumer that the provider has made an update and let the consumer update/clear it’s cache appropriately. This would solve 2 problems. First, the data will always be properly synchronized. But in this case, the data on the provider is updated in frequently. By requiring a cache update only when updates happen, the consumer app can run longer on cached data making fewer calls to the provider on average.

Enter Faye. A dirt simple publish/subscribe system for Node.js and Ruby. With the Faye library, the server can be as little as 4 lines of Javascript. In almost no time, I had a Faye server up on Heroku and a client side Sinatra app which was publishing and subscribing to different channels and passing messages around with ease. For the first implementation, I setup the subscription clients in the browser (as per the Faye docs) like this (counting on a little jQuery help):

<script src="<faye_server_url>/client.js"></script>
<script>
var client = new Faye.Client(faye_server_uri);
/* heroku doesn't like websocket */
client.disable('websocket');
client.subscribe('/mychannel,  function(msg) {
  $('#messages').append($('<li>').html(safemsg));
});
</script>

But what I really needed was for the consumer server (not just an in-browser page) to receive messages. In ruby, instead of the async callback mechanism, we need something else. Threads + EventMachine. Using Ruby 1.9.3 to power my consumer Sinatra app, I setup an EventMachine in a thread like so:

require 'rubygems'
require 'eventmachine'
require 'faye'
require 'sinatra/base'

class ConsumerApp < Sinatra::Base
  get '/'
    ...
  end
end

Thread.new {
  EM.run {
    faye_client = Faye::Client.new( <faye server url> )
    client.subscribe('/mychannel') do |message|
       puts "Received message on /mychannel [#{message}]"
       # do action based on message contents
    end
  }
}

That with a simple config.ru to run the app and we’re all ready to roll. The thread allows me to run the EventMachine and the web serving application in the same process. Now the consumer app, when it starts up, will subscribe to the ‘/mychannel’ and any messages posted there will get to the server. In the provider, we can publish messages as follows:

def notify_mychannel
  faye_client = Faye::Client.new( <faye server url> )
  faye_client.publish('/mychannel', { :message => 'something happened' })
end

You can specify any number of channels for publishing and subscribing, and the Faye architecture allows wildcards in your subscription channels. For the specific system mentioned above, you might imagine channels that look like

/<modelname>/</action>

This way, the consumer could subscribe to

/model_i_care_about/*

and then based on the action in the message, take the appropriate action on the server.

One other little trick that was required to get this on Heroku was to specify ruby 1.9.3 (for the threading). To do this, you need to use bundler > 1.2.0pre. This adds the keyword ruby to your Gemfile reader, allowing you to specify, in your gemfile, the ruby version you want to use. Check out these pages for more about that setup:

If you want to play with the whole system, I’ve put both the Faye server and the client up on github (see links below). The client is set up to listen both in the server code (as mentioned above) and in a browser (on the index page) with some very simple Javascript as laid out by the Faye docs.

I kind of which I’d thought of this kind of solution when we were working on the Fauxtaux Booth project. It might have been an easier way to communicate state between apps. I guess we’re always learning…

Security Concerns – update as of 01.11.2013

James Coglan recently found a security issue with Faye. Please refer to this post for upgrade/fix instructions.

References

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s