Ruby Method Cache - A Benchmark

I write this on a Sunday morning from the Wicked Good Ruby conference in Boston. I got here in time for coffee today. Good morning!

Yesterday’s first tech talk which really grabbed me was Charlie Somerville (GitHub) and his “MRI Magic Tricks” presentation, during which he rides roughshod through Ruby internals, rewrites class hierarchy, and demonstrates segmentation re-violation through forcible hijacking of the crash dump process. He had me laughing more than a tech talk usually would.

The other fantastic tech talk I caught was from Rachel Myers (GitHub) and Sheena McCoy (ModCloth), explaining the MRI Ruby method cache in its various implementations over the years to today, and showing the conditions which cause cache invalidations.

I have been working on my self-inflating hypermedia API client HyperResource a lot lately — watch this space, I am this close to releasing 0.2 — and one of HyperResource’s main features is performing runtime method definition when possible, with a fallback to method_missing-based dispatch in other cases.

Well, defining methods at runtime busts the entire method resolution cache. And the talk was doing a good job of convincing me that method cache invalidation was worth avoiding. But compared to method_missing? That’s been drilled into Rubyists’ heads as a gross inefficiency for years now.

I asked the two presenters whether they knew the relative performance hit from method cache busting, as compared to method_missing dispatch. They didn’t know offhand. But back to Charlie Somerville, whose illuminating article listing all operations which invalidate the method cache which Myers and McCoy reference made it easy to check.

I pecked out some code, available as a gist and shown here:

(Update 1: reworked to address Josh Jordan‘s rightful concern about Class.new‘s overhead. Thanks Josh! and Update 2: added warm-up code for JVM benchmarking.)

require 'benchmark'

module A; end

class DefinedMethodStyle
  def bust_cache_class(*);  Class.new;            nil end
  def bust_cache_extend(*); Object.new.extend(A); nil end
  def dont_bust_cache(*);   Object.new;           nil end
end

class MethodMissingStyle
  def _bust_cache_class(*);  Class.new;            nil end
  def _bust_cache_extend(*); Object.new.extend(A); nil end
  def _dont_bust_cache(*);   Object.new;           nil end

  def method_missing(method, *args)
    case method
    when :bust_cache_class  then _bust_cache_class(*args)
    when :bust_cache_extend then _bust_cache_extend(*args)
    when :dont_bust_cache   then _dont_bust_cache(*args)
    end
  end
end

dms = DefinedMethodStyle.new
mms = MethodMissingStyle.new
n = 1_000_000

if ENV['WARM_UP'] # this helps with JRuby benchmarking
  puts "Warming up.\n"
  for i in 1..n
    dms.bust_cache_extend(i); dms.bust_cache_class(i); dms.dont_bust_cache(i)
    mms.bust_cache_extend(i); mms.bust_cache_class(i); mms.dont_bust_cache(i)
  end
end

Benchmark.bm do |bm|
  puts "\n defined methods, not busting cache:"
  bm.report { for i in 1..n; dms.dont_bust_cache(i)   end }

  puts "\n method_missing dispatch, not busting cache:"
  bm.report { for i in 1..n; mms.dont_bust_cache(i)   end }

  puts "\n defined methods, busting cache with trivial Object#extend:"
  bm.report { for i in 1..n; dms.bust_cache_extend(i) end }

  puts "\n defined methods, busting cache with Class.new:"
  bm.report { for i in 1..n; dms.bust_cache_class(i)  end }

  puts "\n method_missing dispatch, busting cache with trivial Object#extend:"
  bm.report { for i in 1..n; mms.bust_cache_extend(i) end }

  puts "\n method_missing dispatch, busting cache with Class.new:"
  bm.report { for i in 1..n; mms.bust_cache_class(i)  end }
end

So how bad is method_missing, really, in the grand scheme of things?

Not so bad, relatively speaking.

$ ruby --version
ruby 2.0.0p195 (2013-05-14 revision 40734) [x86_64-darwin12.3.0]

$ ruby method-cache-test.rb
       user     system      total        real

 defined methods, not busting cache:
   0.270000   0.000000   0.270000 (  0.273828)

 method_missing dispatch, not busting cache:
   0.410000   0.000000   0.410000 (  0.408970)

 defined methods, busting cache with trivial Object#extend:
   1.390000   0.030000   1.420000 (  1.414536)

 defined methods, busting cache with Class.new:
   1.550000   0.090000   1.640000 (  1.638677)

 method_missing dispatch, busting cache with trivial Object#extend:
   1.610000   0.020000   1.630000 (  1.640575)

 method_missing dispatch, busting cache with Class.new:
   1.840000   0.060000   1.900000 (  1.889407)

When we’re not making MRI invalidate its method resolution cache, method_missing comes in not too far behind regular method resolution — it’s only about twice as time consuming, if your method_missing is relatively lightweight. 100% penalty versus straight dispatch. Hmm.

But when we flush the method cache? Ouch! That’s a 600% penalty for normal method resolution, and about 300% for method_missing dispatch.

The latest Ruby-HEAD, in the 2.1.0dev series, shows similar results, though there seems to be greater penalty for a trivial Object#extend than for a trivial Class.new:

$ ruby --version
ruby 2.1.0dev (2013-10-13 trunk 43273) [x86_64-darwin13.0.0]

$ ruby method-cache-test.rb
       user     system      total        real

 defined methods, not busting cache:
   0.310000   0.010000   0.320000 (  0.311989)

 method_missing dispatch, not busting cache:
   0.400000   0.000000   0.400000 (  0.401800)

 defined methods, busting cache with trivial Object#extend:
   1.970000   0.010000   1.980000 (  1.983993)

 defined methods, busting cache with Class.new:
   1.860000   0.010000   1.870000 (  1.870652)

 method_missing dispatch, busting cache with trivial Object#extend:
   2.180000   0.000000   2.180000 (  2.187509)

 method_missing dispatch, busting cache with Class.new:
   2.010000   0.010000   2.020000 (  2.009935)

Update 2: And seeing as I use JRuby pretty often, I figured I’d test method resolution cache-busting there too. I got some weird results until I figured out that my code was probably being optimized on the fly at an unpredictable point, so I added a “warm up” phase to the test. Behold: things are not much better on the other side.

$ ruby --version
jruby 1.7.5 (1.9.3p392) 2013-10-07 74e9291 on Java HotSpot(TM) 64-Bit Server VM 1.7.0_13-b20 [darwin-x86_64]

$ WARM_UP=1 ruby method-cache-test.rb
Warming up.
       user     system      total        real

 defined methods, not busting cache:
   0.210000   0.010000   0.220000 (  0.119000)

 method_missing dispatch, not busting cache:
   0.380000   0.000000   0.380000 (  0.314000)

 defined methods, busting cache with trivial Object#extend:
   4.160000   0.140000   4.300000 (  3.372000)

 defined methods, busting cache with Class.new:
   8.730000   0.170000   8.900000 (  7.982000)

 method_missing dispatch, busting cache with trivial Object#extend:
   3.700000   0.170000   3.870000 (  2.952000)

 method_missing dispatch, busting cache with Class.new:
   6.080000   0.130000   6.210000 (  5.306000)

This is admittedly a small and very contrived example, and real-world loads will have wildly different performance, but I think it’s fairly convincing of the point that avoiding method_missing is not as important as one might think, in environments which perform cache-busting operations reasonably often.

In relation to HyperResource, I am still satisfied with my choice to enable both runtime method definition (which take place generally once per server launch, per API data type), and method_missing-based syntactic sugar. Runtime method definition increases performance at the cost of requiring system restarts to fully recognize API-side updates, so I’ll be adding a way to disable method creation too.

(And something brought up during Joshua Ballanco‘s thorough tour through RubyMotion internals yesterday, is that RubyMotion can’t do define_method at runtime. I fear I would have to refactor a bit of HyperResource code itself in order to support RubyMotion, but it is an interesting thought.)

I was very excited to explore this topic in some depth yesterday. Comments on my benchmark and conclusions are extremely welcome!

Aces and thanks to @RachelMyers, @SheenaPMcCoy, @charliesome, and to the @WickedGoodRuby organizers.

Now go read some links that Myers and McCoy put together.