Easy Postgres Backup/Restore from Heroku with PGBackups and Rake

Most of my projects lately have been deployed on Heroku. They’ve developed a really nice set of tools to get your Rails (and other) apps from a git repo out into the world. They do really smart things regarding database connections to make things easy to push live. If you follow the standard setup, you’ll be running on Postgres hosted at Heroku.

Often, we want to take data that may be out on a live app (staging or production level) and setup a development machine to have that data. For complex data models and complex data setups, this can be the only way to debug issues that may not have been covered by standard unit/integration tests. With PGBackups, a heroku add-on, and a couple small Rake tasks, this is a snap.

First, make sure you have pgbackups setup for your app:

% heroku addons:add pgbackups

With this enabled, you can immediately grab a snapshot of the database

heroku pgbackups:capture

You can also easily setup auto-weekly or auto-monthly snapshots

heroku addons:add pgbackups:auto-week
heroku addons:add pgbackups:auto-month

To grab the database, you ask for the url

heroku pgbackups:url

which you can then pull down with curl or by some other means. Then use pg_restore to jam it into your local database (if you want).

To make this even easier, I wrote a couple rake tasks which i bring along from project to project. To make it even easier.

Drop this file in your lib/tasks/ directory, update the namespace and development database name and you’re off to the races:

With this in place, you should be able to do:

rake my_app_namespace:db:fetch app=my_heroku_app_name
rake my_app_namespace:restore dbfile=the_dump_file_from_the_previous_command

The other task in there may not suit everyone’s purposes, but if you have sensitive User data, you can use that task to clean it up. Currently it just changes passwords to ‘monkey’. But you could do other things, like set all emails to or remove phone numbers or any other bits that you might not want to be real on your development system.

Now that I’ve done this a couple times, the next step to improve things is to get the script to read the local database.yml to grab the development database name automagically. An easy addition if you want to streamline things even more.

Custom RSpec Matchers for arrays

More than once, I’ve had to check an array for it’s order in RSpec. Usually, I just sort the array, and compare it to itself. The spec might look something like

describe MyObject
  describe "scope #ordered" do
    it "returns results in most recent order"
      expect(MyObject.ordered.all.sort(&:attr_to_order_by)).to eql my_objects
    end
  end
end

This may seem a bit opaque (as a reader) – how does the sorting relate to the test? Also, if you looking at timestamps (like sort by created_at), and you’re using fixtures or FactoryGirl.create, you may end up with duplicate timestamps. Suddenly your sort will be ill defined because the secondary sort key is undefined. A better method would be to check that the :created_at time is monotonically decreasing.

An array x is monotonically decreasing if and only if:
x_i \geq x_j \forall i < j

For these kinds of cases, I decided to write a some RSpec matchers. Following the RSpec Wiki docs here, it was a snap. I wrote matchers for monotonically increasing/decreasing and strictly increasing/decreasing tests.

The following four matchers (available from this gist (included below)) give you

   be_monotonically_increasing
   be_strictly_increasing
   be_monotonically_decreasing
   be_strictly_decreasing

So now I can rewrite my spec

describe MyObject
  describe "scope #ordered" do
    it "returns results in most recent order"
      # the most recent object should have the largest created_by timestamp
      expect(MyObject.ordered.map(&:created_by)).to be_monotonically_decreasing
    end
  end
end

These matchers follow the definitions from NIST for monotonically/strictly increasing/decreasing (see links below). The short story is:

Monotonically Increasing means:
x_i \leq x_j \forall < j

Strictly Increasing means:
x_i < x_j \forall i < j

Monotonically Decreasing means:
x_i \geq x_j \forall i < j

Strictly Decreasing means:
x_i > x_j \forall i < j

To use these, put the gist (below) custom_array_matchers.rb file in your spec/support directory and you’re off to the races. You can start using those matchers in your specs. The gist also includes specs that test the matchers. If the matcher names aren’t clear, check out the specs and hopefully that’ll reveal what they do. The algorithm itself is pretty simple – take a derivative of the array values and make sure that it’s always positive (or negative) in slope for increasing (or decreasing) checks.

The code is built for RSpec 2.something (at a minimum). They also assume your sort attribute is an number. I haven’t tried it with strings, but I think it wouldn’t take much to modify things to handle more general cases. Perhaps I’ll run into that problem and have an update. Until then, they are what they are.

References

Definitions
Gist