The Hidden Costs of Metaprogramming

Metaprogramming sounds like a very fancy word, but is it any good?

It can be useful, but many people don’t realize that using metaprogramming has some costs.

Just so we are on the same page…

What is metaprogramming exactly?

I define metaprogramming as using any method that:

  • Alters the structure of your code (like define_method)
  • Runs a string as if it was part of your actual Ruby code (like instance_eval)
  • Does something as a reaction to some event (like method_missing)

So what are the costs of metaprogramming? I classify them into 3 groups:

  • Speed
  • Readability
  • Searchability

Note: You could also say that there is a fourth group: Security. The reason for that are the eval methods, which don’t make any kind of security checks on what is being passed in. You have to do that yourself.

Let’s explore each of those in more detail!

Speed

The first cost is speed because most metaprogramming methods are slower than regular methods.

Here is some benchmarking code:

require 'benchmark/ips'

class Thing
  def method_missing(name, *args)
  end

  def normal_method
  end

  define_method(:speak) {}
end

t = Thing.new

Benchmark.ips do |x|
  x.report("normal method")  { t.normal_method }
  x.report("missing method") { t.abc }
  x.report("defined method") { t.speak }

  x.compare!
end

The results (Ruby 2.2.4):

normal method:   7344529.4 i/s
defined method:  5766584.9 i/s - 1.34x  slower
missing method:  4777911.7 i/s - 1.54x  slower

As you can see both metaprogramming methods (define_method & method_missing) are quite a bit slower than the normal method.

Here is something interesting I discovered…

The results above are from Ruby 2.2.4, but if you run these benchmarks on Ruby 2.3 or Ruby 2.4 it looks like these methods are getting slower!

Ruby 2.4 benchmark results:

normal method:   8252851.6 i/s
defined method:  6153202.9 i/s - 1.39x  slower
missing method:  4557376.3 i/s - 1.87x  slower

I ran this benchmark several times to make sure it wasn’t a fluke.

But if you pay attention & look at the iterations per second (i/s) it seems like regular methods got faster since Ruby 2.3. That’s the reason method_missing looks a lot slower 🙂

Readability

Error messages can be less than helpful when using the instance_eval / class_eval methods.

Take a look at the following code:

class Thing
  class_eval("def self.foo; raise 'something went wrong'; end")
end

Thing.foo

This will result in the following error:

(eval):1:in 'foo': 'something went wrong...' (RuntimeError)

Notice that we are missing the file name (it says eval instead) & the correct line number. The good news is that there is a fix for this, these eval methods take two extra parameters:

  • a file name
  • a line number

Using the built-in constants __FILE__ & __LINE__ as the parameters for class_eval you will get the correct information in the error message.

Example:

class Thing
  class_eval(
    "def foo; raise 'something went right'; end",
    __FILE__,
    __LINE__
  )
end

Why isn’t this the default?

I don’t know, but it’s something to keep in mind if you are going to use these methods 🙂

Searchability

Metaprogramming methods make your code less searchable, less accessible (via worse documentation) & harder to debug.

If you are looking for a method definition you won’t be able to do CTRL+F (or whatever shortcut you use) to find a method defined via metaprogramming, especially if the method’s name is built at run-time.

The following example defines 3 methods using metaprogramming:

class RubyBlog
  def create_post_tags
    types = ['computer_science', 'tools', 'advanced_ruby']

    types.each do |type|
      define_singleton_method(type + "_tag") { puts "This post is about #{type}" }
    end
  end
end

rb = RubyBlog.new

rb.create_post_tags
rb.computer_science_tag

Tools that generate documentation (like Yard or RDoc) can’t find these methods & list them.

These tools use a technique called “Static Analysis” to find classes & methods. This technique can only find methods that are defined directly (using the def syntax).

Try running yard doc with the last example, you will see that the only method found is create_post_tags.

It looks like this:

yard example

There is a way to tell yard to document extra methods, using the @method tag, but that is not always practical.

Example:

class Thing
  # @method build_report
  define_method(:build_report)
end

Also if you are going to use a tool like grep, ack, or your editor to search for method definitions, it’s going to be harder to find metaprogramming methods than regular methods.

“I don’t think Sidekiq uses any metaprogramming at all because I find it obscures the code more than it helps 95% of the time.” – Mike Perham, Creator of Sidekiq

Conclusion

Not everything is bad about metaprogramming. It can be useful in the right situations to make your code more flexible.

Just be aware of the extra costs so you can make better decisions.

Don’t forget to share this post if you found it useful 🙂

13 comments
Adis says 5 years ago

You have a typo on first benchmark. You missed Thing initialization.

    Jesus Castello says 5 years ago

    You are right! Thanks for pointing that out, just fixed it 🙂

Noah says 5 years ago

One way to think about the security cost: it’s a side effect of complexity. You point out that readability/searchability is lower (true.) But complexity is also higher – you need more understanding of the code to reason about what it’s doing.

Security cost is a side effect of complexity cost, which you’re expressing as readability/searchability cost.

Nasos Psarrakos says 5 years ago

Great post Jesus.

Keep it up 😉

    Jesus Castello says 5 years ago

    Thank you 🙂

Manish johari says 5 years ago

Great post Jesus, Thank you for sharing.

    Jesus Castello says 5 years ago

    Thanks for reading 🙂

Benoit Daloze says 5 years ago

Interestingly, with the latest TruffleRuby (http://www.oracle.com/technetwork/oracle-labs/program-languages/downloads/index.html), the speed cost is essentially non-existent.
Using the benchmark above and adding x.iterations = 2 to make it more stable I get:

…/graalvm-0.25/bin/ruby -v bench.rb

truffleruby 0.25, like ruby 2.3.3  [linux-x86_64]
Warming up --------------------------------------
       normal method   436.542k i/100ms
      missing method     2.301M i/100ms
      defined method     2.202M i/100ms
       normal method     2.421M i/100ms
      missing method     2.527M i/100ms
      defined method     2.950M i/100ms
Calculating -------------------------------------
       normal method    189.232M (± 8.2%) i/s -    900.678M in   5.006158s
      missing method    208.035M (± 2.0%) i/s -      1.041B in   5.006995s
      defined method    216.816M (± 6.2%) i/s -      1.077B in   4.994133s
       normal method    189.811M (± 5.5%) i/s -    946.680M in   5.009507s
      missing method    201.205M (± 8.5%) i/s -    995.617M in   5.001431s
      defined method    207.183M (±11.7%) i/s -      1.018B in   5.004248s

Comparison:
      defined method: 207182797.2 i/s
      missing method: 201204588.3 i/s - same-ish: difference falls within error
       normal method: 189811442.8 i/s - same-ish: difference falls within error

Also, the speed is in the order of 200 millions calls per second!

    Jesus Castello says 5 years ago

    Yeah, that’s interesting. Thanks for sharing!

    funny_falcon says 5 years ago

    200M calls/sec – it was optimized out? just couple of guards, and benchmark code overhead?

Robert Pankowecki says 5 years ago

I also decided to stop using meta code that much for similar reasons (except speed which is already good enough in Ruby) http://blog.arkency.com/2017/02/ruby-code-i-no-longer-write/

Jay says 5 years ago

I am running an older version of jruby (9.1.2.0) and the normal vs defined method is also within error. Missing method is still about 1.5x times slower. With all the JIT-ing that ruby does I think that accounts for the negligible difference in the dynamic method creation.

Warming up --------------------------------------
       normal method   202.307k i/100ms
      missing method   196.264k i/100ms
      defined method   213.748k i/100ms
Calculating -------------------------------------
       normal method     16.209M (±10.7%) i/s -     79.911M in   4.995190s
      missing method     10.914M (± 9.3%) i/s -     53.973M in   4.998669s
      defined method     14.672M (±10.5%) i/s -     71.819M in   5.001494s

Comparison:
       normal method: 16208767.3 i/s
      defined method: 14671736.4 i/s - same-ish: difference falls within error
      missing method: 10914121.6 i/s - 1.49x  slower

    Jesus Castello says 5 years ago

    Thanks for sharing your results 🙂

Comments are closed