Quick and Dirty Caching lets MySQL rest

Ever seen this from engineyard?

The site we were working on had recently had a jump in traffic on the order of 15x. The site held up just fine. Things seemed a bit slower, but generally functional.

One of the services the site offers to users uses Delayed::Job to message/email folks about things. And to keep everyone abreast of the status of the site, we have a status page which reports, among other things, if the jobs are getting backed up. This page does a seemingly simple query of the Delayed::Jobs table and finds jobs that haven’t completed in a reasonable amount of time – it finds the delayed Delayed::Jobs. As it turns out, that query is fairly expensive. It’s a full table scan and with more users using the service, that table has gotten big. With a few services checking the system for ‘up’ status (EngineYard, 100Pulse etc), that query was getting run pretty often and ended up loading up our db server.

Using EngineYards performance graphs (to confirm that the load was consistently bad) and New Relic’s server code level monitoring we narrowed the load down to this Delayed::Job#find which was called by a #get_pending_jobs method in our status page.

How do we solve it? Caching, of course.

I’m a big fan of memcache. I’ve seen it work wonders on huge systems with piles of traffic. But we wanted a quicker fix. Setting up a new memcached machine was a little more work (though not much) than we were willing to do for this first cut. Rails caching (we’re still on 2.3.5), by default, uses ActiveSupport::Cache::MemoryStore – a super simple memcache like system (cached in memory with no persistence) but running inside the app. Perfect.

We wrapped our get_pending_jobs call like so:

def fetch_delayed_job_status
  jobs_cache = Rails.cache.read(JOB_CACHE_KEY)
  if !jobs_cache || !jobs_cache[:expires_at] || (jobs_cache[:expires_at].to_i  get_pending_jobs
      :expires_at => (Time.now + JOB_CACHE_EXPIRY).to_i
    }
    Rails.cache.write(JOB_CACHE_KEY, job_cache)
  else
    # we got it from the cache
  end
  
  jobs_cache
end

and we’re off and caching. Because we do want to get notified when something is wrong, we don’t want the cache time to be too long. Since our most frequent test service checks in every 5 minutes, we went with a 5 minute cache. So, worst case, things could be screwy for 10 minutes before we get notified. But we can live with that.

This quick and dirty solution worked great. We got immediate feedback. The email stopped coming. The load graphs on the server took a drastic dive. And New Relic showed us that the amount of time spent doing Delayed::Job#find was reduced by about 30%. Not too bad for 10ish lines of code.

One of the annoying things about this is that the cache expiry had to be managed here in this little loop. In Rails 2, only ActiveSupport::Cache::MemCacheStore accepts and uses the :expires_in option to manage cache entry expiration. In Rails 3, all the included Cache::Store implementations allow and honor the :expires_in param.

Advertisements

Fauxtaux Booth – our Mashup-o-rama

We’ve been noticing that writing software for the web is becoming more and more like playing with lego blocks. We are currently working on projects using Ruby + Rails and Node.js + Express. For both of these technology stacks we have a lot of conversations like these:

“You need a library that does <this special thing>?”
“Oh yeah, it’s on github.”

“You need a javascript plugin that does <whatever>?”
“Sure, start from that thing on jQuery plugins and you’re 95% there.”

Such is the nature of participating in and taking advantage of the open source community.

We wanted to share our latest lego-block project. We called it

Fauxtaux Booth

A photographer friend was setting up a photobooth at a conference. She wanted to have the pictures sent directly to the subjects as they were being taken.

The blocks we used for this system were:

The basic setup was as follows:

The photographer was on the net at the conference, her camera tethered to the laptop. As photos were taken, they were written to a folder that sat under a shared Dropbox folder.

On a separate server, also sharing that same Dropbox folder, we setup a watch on that folder using Guard. When we saw a change, we sent an http request (basically a ping) to the Rails app. When the app caught that http request, it would look at the directory and figure out what had changed. If there were new images, it would run several image processing steps and finally email the current customer by way of Sendgrid.

The app also served as a customer signup form. So at the conference, the photographer’s laptop also had a browser window open to the app. As customers stepped up to the booth, they filled out a simple form on the laptop with name and email which we used to send the photos.

For each sitting, the final emailed image was a photostrip with the conference’s branding/data on it.

The majority of the special coding in the app was custom image processing. The rest was basically wiring a bunch of moving parts together.

At the event, our photographer took about 80 photos and we sent out about 400 emails.

This was the setup at the conference.