Upgrading Ruby for Rails3 – Installment 3

If you’re following along, I’ve been trying to get an old up from Rails 2.3 up to Rails 3.x (previous installments 1, 2).

After a bunch of refactoring and clean up, I was able to get my coverage report to an acceptable level. The app was running successfully on Rails3.0. Now to try to upgrade ruby.

  • Update my Gemfile & .ruby-version files to use ruby 1.9.3
  • Dump rcov from the Gemfile (eventually I’ll replace that with simplecov
  • bundle and run tests
  • clean up the mess
  • undefined method `each' for String

    No problem. I dug in to find a test trying to parse an http response body by newlines. Apparently ruby-1.8.7 String would handle this.

    The fix was simple

    
    -    resp.body.each do |row|
    +    resp.body.split("\n").each do |row|
    

    Scopes with arguments

    I’m not proud to admit it but I had a scope defined which both took an argument (not such a bad thing) but then wrote custom SQL based on that parameter. A combo of changes related to how lambda arguments are handled by scopes and the way the arguments were passed into the scope caused troubles. The fix was something like this (note: the model names have been changed to protect the innocent):

    
    -  scope :participants, lambda { |*key|
    -    q = key.blank? ? Conf.participant_key : key
    -    joins(:extra_info).where("extra_infos.participation like '%%{q}%%'")
    +  scope :participants, lambda { |key|
    +    q = (key.blank? ? Conf.participant : key).to_s
    +    joins(:extra_info).where("extra_infos.participation like '%#{q}%'")
    

    If you forget about the fact that I was pushing a string unprotected into a SQL query, the two things that seemed to cause issues were the splat on the lambda’s input args and the double ‘%’ in the query string. This change at least made the code behave as it used to.

    This code is actually doing a bunch of work that could be handled with serialize :participation, Array. I’ve added a chore for myself to make that change before I go further. More about serialize can be found here.

    Even after fixing that, the scope worked fine with arguments, but was very unhappy in cases where I had nil args which lead to more failing tests. In the end, I refactored the scope into a method and it made the whole thing much simpler.

    
    def self.participants(key = nil) 
      q = key || Conf.participant
      joins(:extra_info).where("extra_infos.participation like '%#{q}%'")
    end
    
    

    Symbols to Strings

    The final straw was updating a few tests that counted on public_instance_methods. The test tried to grab all controller methods and run simple checks that methods existed. public_instance_methods returned strings in Ruby 1.8.7 and symbols in the new.

    
    1.8.7>  CatalogController.public_instance_methods(false)
    => ["index"]
    
    
    1.9.3>  CatalogController.public_instance_methods(false)
    => [:index]
    

    A quick fix (map the result to strings) got everything back in line.

    Well that was easy, how about ruby2.0.0?

    Once I had that rolling, what’s the harm in running all the way to ruby2.0.0?

    Following the same steps, I updated the Gemfile and .ruby-version. Rebundled, and started fixing issues.

    String#start_with?

    It turns out that ruby2.0.0, String#start_with?does not allow a nil input. This causes stylesheet_link_tag to fail because in Rails 3.0.20, it does the following:


    # actionpack-3.0.20/lib/action_view/helpers/asset_tag_helper.rb:749
    if has_request && include_host && !source.start_with?(controller.config.relative_url_root)

    and it seems that Rails 3.0.20 does not set relative_url_root by default. (more discussion here on stackoverflow)

    After adding a quick patch to my application.rb

    ActionController::Base.config.relative_url_root = ''

    everything was back in line.

    insert_record not available on Array

    This was the last big error that came up during tests. It seems that to get around this, I needed to move to Rails 3.1. Feeling like I was on a roll, I happily started more migrating.

    • Updated Rails to 3.1.12
    • Updated mysql2 gem and removed the activerecord-mysql2-adapter
    • added simplecov
    • removed tmail gem
    • remove config.action_view.debug_rjs from Rails config files
    • moved restful_authentication files into lib (from vendor/plugins)
    • upgraded restful_authentication to use AASM directly (instead of acts_as_state_machine method
    • Update references to Fixtures to use ActiveRecord::Fixtures

    And with that, I was on Ruby-2.0 and Rails-3.1. Not too shabby.

    I’m still not using the asset pipeline, but feel like I’m in a good place to make that jump to Rails3.2 and only have to worry about getting the pipeline in place.

rbenv and rvm plugins for Jenkins

For any of you folks still managing your own Jenkins CI box like I am, you may be interested to know that the rbenv and RVM plugins are great no-nonsense plugins that work.

My current Jenkins instance has been running for several years. Jenkins itself is getting upgraded with standard Ubuntu package upgrades, but it’s been a long time since I looked into plugin upgrades.

Older projects had some ugly setup script to get the rvm environment correct for the build. Stuff like:


. "$HOME/.rvm/scripts/rvm"
rvm use --create 1.9.3-p194@selectorschoice
gem install bundler
bundle
bundle exec rake

Much of this was came from old rvm docs or elsewhere on the web.

Well, I took the time to update my plugins recently and tried both the RVM and rbenv plugins. They both work smoothly. Simply add them to your Jenkins system. Then configure the build to run in either RVM or rbenv environment. Now my test scripts have been trimmed down to


bundle
bundle exec rake

And the log outputs are much cleaner so it’s easier to dig in when there are issues.

Check them out:

RVM Plugin version 0.4
rbenv Plugin version 0.0.15

Upgrading Rails2 -> Rails3 – installment 2

Installment 2 of my upgrade from Rails2 -> Rails3


Ack – I depend on all these gems!

I spoke with a co-worker about this. He’d been through the upgrade and mentioned that Gem upgrades was a major part of the move from Rails2 -> Rails3.

As I mentioned before, I’d taken steps to get my app as close to Rails3 ready as possible (read installment 1). Now it was time to pull the trigger.

I updated my Gemfile to require
gem "rails", "3.0.20"
and
bundle update rails

I immediately ran into issues. Out of date gems!

Since I was moving forward, I decided to do a full bundle update. This would force everything (within dependencies) to the latest. I then could go back and figure out issues. In the end, I had to pin a few gems down. I got to upgrade a few. I had to dump a few. And the rest (at least at this point), look like they’ll work out as is.

Pinned:


gem 'nokogiri', '~> 1.5.0' # At 1.6.0, this gem is no longer ready for 1.8.7. Pin it to 1.5.x.
gem 'gibbon', '~> 0.3.5' # mailchimp api
gem 'browser', "= 0.1.6" # 0.2.x does not support ruby 1.8.7

Upgraded:


gem 'dalli' # now I can use 2.x
gem 'rspec' # now I can use 2.3.x
gem 'will_paginate' # move up from version 2.3.16

Modified/replaced:

Moved mobile-fu to mobile-fu-rails3

gem 'mobile_fu-rails3'

Removed (looking for replacements)


has_many_polymorphs, ym4r_gm

Of those removed, I should add a few notes.

has_many_polymorphs

This one caused me all kinds of headaches. I did some searching and was able to find a replacement (here) that said it was Rails3 ready. I neglected to read the not so fine print from the README

This is a hacky attempt to port has_many_polymorphs to work under Rails 3. It is intended solely for my use (and use as a gem) but is offered here in case it helps anyone else develop it into a more complete port.

After sticking this into the Gemfile (and doing some other mods), my test suite started failing hard. belongs_to and has_many were failing on class load. I tore my hair out for a bit (read more here). Then some deep debugging revealed that the has_many_polymorphs update was Rails3.2 compatible, but not Rails3.0.

I’ve torn it out completely. It was relatively easily replaced by using the :polymorphic setting for Rails associations. You can read more about that here.

ym4r_gm

Strictly speaking, ym4r is not a gem, it’s a plugin. At the time I’d built this site, we wanted a map of Users. Using geokit and geokit-rails with a the Ym4r_gm plugin, I could generate a Google Map from ruby and drop it on the view. It seems that the Ym4r plugin has fallen away in favor of Google Maps For Rails. For now, I’ve decided to live without the Map (it’s only one page) until I get to Rails 3.2 and then I’ll check my options. I may just write the client side (JS) integration from scratch with Googles JS api. Less dependence on gems.

Other gem related mods

While making all these changes, and trusting my test suite, I decided to make a few more updates.

I removed mocha which I was using for mocking in tests. The new Rspec does a fine job with Double and has good support for stubs. This meant combing through my specs replacing:

before (mocha) after (rspec-*)
mock('thing') double('thing')
MyModel.any_instance.stubs(:method => val) MyModel.any_instance.stub(:method => val)
obj.expects(:meth)
.once.returns(stuff)
obj.should_receive(:meth)
.exactly(:once).and_return(stuff)

I also installed a much needed rcov to get some sense of my test coverage. I was able to stick this on the Rails2.3 branch and the Rails3.0 branch to validate that in all my mucking around, I hadn’t lost any tests. Happily, the old branch was 89% covered (not too bad, considering this was my Rails learning project). And I was able to validate that after updating tests to not rely on mocha I still had 89 percent coverage. In fact, in the process, the coverage went up, because I did some code-cruft clean up and added tests in a few spots.

As I find out more, I’ll scribble some more notes. Though these posts maybe a bit sloppily written, I hope they save a few folks out there who may still be going through this upgrade a headache or two.

Upgrading Rails2 -> Rails3

The past few months, among normal work stuff, I’ve been busy working to convert an old (5years and running) app from Rails 2.3.x to Rails 3 with the hope of finally getting to Rails 3.2 and Ruby 1.9.x.

It’s been a bit of an uphill slog, as you might imagine.

I’m going to start scribbling some of my notes here in the blog in a series of posts in hopes that others don’t rathole on the same issues that I struggled with.

Pre Rails3

After a chunk of reading online (much thanks to the airbnb post), I started by trying to update the codebase without updating Rails so that when I did move, I’d have as much of the simple stuff resolved in the Rails2 branch.

Step 1: rails_xss

I started by installing the rails_xss gem. This switches the default view generators to escape strings which is inline with what Rails3 views do.

I added to my Gemfile :
gem 'rails_xss'
gem 'erubis'

Thankfully, I had a pretty strong view test suite and was able to find and fix issues based on my test failures.

Step 2: Dump RAILS_ROOT

This was pretty much a search-replace task. Replace RAILS_ROOT with Rails.root

Step 3: include jQuery with noConflict

This app was still on PrototypeJS. It seems like it’s finally time to make the move. The first step is to simply get jQuery in the codebase with noConflict. This will allow me to convert, over time, the Prototype plugins and code to jQuery without having to move it all in one chunk.

This was pretty straight forward. Add the jQuery source from http://jquery.com/ and include it in my HAML layouts like so (at the top, before including prototype and other library js):

= javascript_include_tag "thirdparty/jquery-1.8.2.min.js"
= jQuery.noConflict();

With these mods, and getting my test suite sorted out, I was able to push out a new version still on Rails 2.3.x but much closer to being Rails3 ready that it was before these mods.

I’ll continue to add notes in subsequent posts.

References

Easy Postgres Backup/Restore from Heroku with PGBackups and Rake

Most of my projects lately have been deployed on Heroku. They’ve developed a really nice set of tools to get your Rails (and other) apps from a git repo out into the world. They do really smart things regarding database connections to make things easy to push live. If you follow the standard setup, you’ll be running on Postgres hosted at Heroku.

Often, we want to take data that may be out on a live app (staging or production level) and setup a development machine to have that data. For complex data models and complex data setups, this can be the only way to debug issues that may not have been covered by standard unit/integration tests. With PGBackups, a heroku add-on, and a couple small Rake tasks, this is a snap.

First, make sure you have pgbackups setup for your app:

% heroku addons:add pgbackups

With this enabled, you can immediately grab a snapshot of the database

heroku pgbackups:capture

You can also easily setup auto-weekly or auto-monthly snapshots

heroku addons:add pgbackups:auto-week
heroku addons:add pgbackups:auto-month

To grab the database, you ask for the url

heroku pgbackups:url

which you can then pull down with curl or by some other means. Then use pg_restore to jam it into your local database (if you want).

To make this even easier, I wrote a couple rake tasks which i bring along from project to project. To make it even easier.

Drop this file in your lib/tasks/ directory, update the namespace and development database name and you’re off to the races:

With this in place, you should be able to do:

rake my_app_namespace:db:fetch app=my_heroku_app_name
rake my_app_namespace:restore dbfile=the_dump_file_from_the_previous_command

The other task in there may not suit everyone’s purposes, but if you have sensitive User data, you can use that task to clean it up. Currently it just changes passwords to ‘monkey’. But you could do other things, like set all emails to or remove phone numbers or any other bits that you might not want to be real on your development system.

Now that I’ve done this a couple times, the next step to improve things is to get the script to read the local database.yml to grab the development database name automagically. An easy addition if you want to streamline things even more.

Custom RSpec Matchers for arrays

More than once, I’ve had to check an array for it’s order in RSpec. Usually, I just sort the array, and compare it to itself. The spec might look something like

describe MyObject
  describe "scope #ordered" do
    it "returns results in most recent order"
      expect(MyObject.ordered.all.sort(&:attr_to_order_by)).to eql my_objects
    end
  end
end

This may seem a bit opaque (as a reader) – how does the sorting relate to the test? Also, if you looking at timestamps (like sort by created_at), and you’re using fixtures or FactoryGirl.create, you may end up with duplicate timestamps. Suddenly your sort will be ill defined because the secondary sort key is undefined. A better method would be to check that the :created_at time is monotonically decreasing.

An array x is monotonically decreasing if and only if:
x_i \geq x_j \forall i < j

For these kinds of cases, I decided to write a some RSpec matchers. Following the RSpec Wiki docs here, it was a snap. I wrote matchers for monotonically increasing/decreasing and strictly increasing/decreasing tests.

The following four matchers (available from this gist (included below)) give you

   be_monotonically_increasing
   be_strictly_increasing
   be_monotonically_decreasing
   be_strictly_decreasing

So now I can rewrite my spec

describe MyObject
  describe "scope #ordered" do
    it "returns results in most recent order"
      # the most recent object should have the largest created_by timestamp
      expect(MyObject.ordered.map(&:created_by)).to be_monotonically_decreasing
    end
  end
end

These matchers follow the definitions from NIST for monotonically/strictly increasing/decreasing (see links below). The short story is:

Monotonically Increasing means:
x_i \leq x_j \forall < j

Strictly Increasing means:
x_i < x_j \forall i < j

Monotonically Decreasing means:
x_i \geq x_j \forall i < j

Strictly Decreasing means:
x_i > x_j \forall i < j

To use these, put the gist (below) custom_array_matchers.rb file in your spec/support directory and you’re off to the races. You can start using those matchers in your specs. The gist also includes specs that test the matchers. If the matcher names aren’t clear, check out the specs and hopefully that’ll reveal what they do. The algorithm itself is pretty simple – take a derivative of the array values and make sure that it’s always positive (or negative) in slope for increasing (or decreasing) checks.

The code is built for RSpec 2.something (at a minimum). They also assume your sort attribute is an number. I haven’t tried it with strings, but I think it wouldn’t take much to modify things to handle more general cases. Perhaps I’ll run into that problem and have an update. Until then, they are what they are.

References

Definitions
Gist

Google Event Tracking with jQuery

As with lots of little bits of code, if they are useful, you may write and re-write them a several times in several projects before you take the time to package them up into a reusable chunk. That finally happened to me around Google Analytics Event tracking.

Almost every project that I’ve been on in the past year has added Google Analytics (simple page tracking) right up front, and then near the end of the work, realized that they also want event/click tracking. On this last project, I decided to make it right, and package it up.

I’ve built a jQuery plugin (https://github.com/rcode5/ga_event_tracker) that makes it easy to setup click tracking all over your site. The event tracker takes 3 parameters: category, action, and label. The plugin will either find these values as attributes on the tracked DOM element (using data-category, data-action and data-label), or use values passed in when the plugin is initialized, or evaluate functions passed in on initialization. This allows for a lot of flexibility. We were able to setup tracking for most pages on a full site with a couple lines of Javascript.

Assuming you add the class ‘trackit’ to every element you want tracked, you can have something like this:

  $('.trackit').gaEventTracker();

And clicking all those elements will get tracked using the values found in the element attributes “data-category”, “data-action”, and “data-label”.

The source is available on Github with a few more examples. A test suite is included, in case you want to add your own extras and submit a pull request.

I hope this saves you a little time down the road, as I’m sure it will for me.