Fast tests in Ruby on Rails

This is a republished guest blog post. The original article is written by Manfred Stienstra. You can find it on the fingertips blog.

Developers need to be able to run tests quickly or they will stop running them.

Slow test suites are often partially caused by slow startup times. Once you've eliminated this problem, you might want to take a look at individual tests.

Note that test suites stress your code in a totally different way from the production environment. A slow test suite doesn't mean your app will be slow in production and the other way around. Never optimize your code for the test suite.

Sometimes slow tests are an indication of slow code, always measure to be sure.

Always start by measuring the actual runtime of your test suite or individual tests when trying to speed them up. Always do a few test runs to get a feeling for the variance in time between test runs.

If your improvements are in the order of the runtime variance they might not be worth it.

We usually create a branch for our improvements in Git so we can easily compare to our mainline branch and abandon a bad idea.

On Ruby 1.8.7 inserting records through ActiveRecord was really slow. Especially in loops, repeatedly called setup methods, or factories. In Rails 4.0 with Ruby 2.0 this has gotten less of an issue.

Unnecessary round-tripping to the database server and models with lots of callbacks, uniqueness checks, and other slow code can still slow down your test suite. A lot of methods and validations on a model can be tested without saving it.

You don't want your tests posting video to YouTube or sending email over SMTP.

The easiest way to test if you're connecting to external services from your test suite is to temporarily turn off your network.

We usually stub out external services with a mock implementation. Alternatively you might want to use Webmock or override Net::HTTP#start for a more general solution.

Connecting to local services can slow down your test suite as well, but in some cases it's pragmatic to do so. Connecting to a local database seems alright, but requiring memcached might not be necessary. It's up to you to make the call (pun intended).

Some people go so far as to implement their business logic in separate modules which are then included in their models. This allows them to run tests without booting Rails. As a general rule I don't like to change my code just to speed up tests.

Kernel#exec, %x, system, and '' all spawn a shell to execute the command. Spawning a shell is relatively slow and can potentially mess up data on your disk.

You can consider wrapping code that spawns a shell in a separate library and mock it out in unit and functional tests. You can test interaction with the shell in a test specific to this library or test through the entire stack in a regression test.

Some tests need to wait for a process to complete some task before examining the result.

Usually these tests wait way too long and slow down the whole suite. It might help to tweak the wait time in these tests.

An even better solution is to use pessimistic locking on the database row or table you're trying to inspect.

it "automatically updates the author name when syncing remotely" do
  # Assume this method opens an exclusive lock on the entire authors table
  BookDb.sync!
  # We try to get our own lock to wait for the sync to finish
  author = authors(:jerry)
  author.with_lock do
    author.name.should == "Jerry Verdervoort"
  end
end

Use exclusive locks to minimize waiting for asynchronous events in tests.

You can use comparable solutions for EventMachine, Mutex, Monitor, IO#select, and others.

Sometimes one or two tests hold up the entire suite, usually these are either regression tests or tests which should be a regression test. One example could be a class which unpacks a zip file and imports its contents into the application. Even when you use a smaller zip bundle for the test it will probably take at least a few seconds.

As a final solution you could consider only running these tests before you push your code instead of before very commit. You could also consider running these tests only on your CI server, assuming you have one.

One way to separate these tests from the rest of the suite is to place them in a separate file. This makes it easy to not run them by adding another test task (i.e. rake test:fast) or configuring your test runner to skip them.

When the tests are part of a class or context with an elaborate setup and you don't want to copy to another file you can consider using an environment variable.

module Kernel
  private
  def run_slow_tests?
    ENV['FAST'].to_s.strip == ''
  end
end
describe Owner do
  it "has a name" do
    Owner.new.name.blank?.should == false
  end if run_slow_tests?
end

Use environment variables to turn off slow tests.

Tests are code and like any other code and will need maintenance. Besides correctness you might also want to keep an eye on speed. We do this by using a test runner which reports slow tests on our CI server.

Finally I would like to mention that there are quite a few tools out there to run your test suite faster. Running tests in parallel might speed up an either suite run. Test servers can save time on boot time and source loading.

A word of caution though, most of theses tools come at an expense. Added dependencies will increase setup time for new developers. Complexity of the solution might make your suite less robust. Extra configuration (i.e. of additional databases) make it less easy to get up an running.

In the long run it will pay off to be able to run the suite without these additional tools so you can easily abandon them if necessary.

We want to thank Manfred for making this article available. As mentioned above you should definitely check out the fingertips blog – they have some great stuff on there!

How do you make your tests fast? Are there any other things you found work really well for you? Let us know in the comments. We would love to get your opinion!