Virtuous Code

Inspired by this talk by Jim Weirich:

Image by scottfeldstein, some rights reserved.

(For the shaving nerds, this poster features the Merkur Hefty Classic, a razor I recently acquired and am so far pretty happy with.)

In our last episode we were augmenting FMTP::Message classes to deal with messages split across multiple packets. As is often the case, fixing one problem revlealed another. What with the unstable weather patterns in Oz – you never know when a spacetime-ripping tornado will appear out of nowhere – our flying monkeys sometimes get blown off course, and arrive out of order. This results in jumbled messages and angry Wicked Witches.

We’ve submitted a revised FMTP RFC to the Flying Monkey Transport Protocol Working Group, but it’s anyone’s guess how long that will take to become an offcial recommendation. Until then, we’ve taken to embedding another bit of metadata in the message text itself. Messages now look like this:

  {{3 OF 5}}
  .... MESSAGE TEXT ...

In order to work conveniently with these enhanced messages, we need some new accessors on the FMTP::Message class:

1. A revised #end? method which uses the new header instead of the now-deprecated “ENDENDEND” token.
2. A new #seq_num accessor to tell us which message in a multi-message sequence this is.
3. A new #total_num accessor to tell us how many total messages are expected as part of the sequence.
4. A modified #data accessor which will return just the message data, minus the message ordering header.

Sounds like it’s time to inject some more methods. But not so fast. I prefer to use the inject method pattern when I only need to add one trivial method. When I need to add or modify more than one method, a few other techniques are better suited. One of the most powerful techniques is delegation.

Delegation has a long history of use in object-oriented languages. In some languages using it can be quite labor-intensive to implement. In Ruby it is so trivially easy that it’s a little surprising it isn’t used more often.

Here’s an example of a delegate class that implements the requirements above:

  require 'delegate'
  class OrderedMessage < DelegateClass(FMTP::Message)
    HEADER_PATTERN = /\{\{(\d+) OF (\d+)\}\}/
 
    def seq_num
      @seq_num ||= matches[1].to_i
    end
 
    def total_num
      @total_num ||= matches[2].to_i
    end
 
    def data
      # Implemented this way to demonstrate use of 'super'
      @data ||= super.gsub(HEADER_PATTERN,'').strip
    end
 
    def end?
      seq_num == total_num
    end
 
    private
 
    def matches
      HEADER_PATTERN.match(__getobj__.data)
    end
  end

We use this class by replacing the line that previously read:

  message = extend_message(super)

With this code:

  message = OrderedMessage.new(message)

There are a few things worth noting about this approach:

1. Using a delegate gives us a safe namespace sandbox to play in. This code defines four methods, a constant, and three instance variables. In particular, note that we define our own @data instance variable. If we were monkey patching, or even subclassing, FMTP:Message, we would run the risk of inadvertently overriding or overwriting one of the FMTP::Message constants, methods, or instance variables. With delegation, however, we can define pretty much anything we want without having to be concerned with collisions.
2. Note that #data uses super to delegate to FMTP::Message, just as if we were writing a subclass. This is a convenience of using the standard delegate library.
3. [NEW] While it’s not really demonstrated in the code above, I should point out that any FMTP::Message methods not explicitly overridden on OrderedMessage will be delegated directly to the wrapped FMTP::Message object.
4. An added benefit of using the delegate pattern is that it is very easy to test our additions to FMTP::Message in isolation. Here is the actual RSpec spec I used to develop the code above:

  describe "any ordered message", :shared => true do
     it "should have the correct sequence number" do
      @it.seq_num.should == @m
    end
 
    it "should report the correct total of messages" do
      @it.total_num.should == @n
    end
 
    it "should have the correct data" do
      @it.data.should == @payload
    end
  end
 
  describe OrderedMessage do
 
    def construct_test_data(m, n, data)
      <<-END
  {{#{m} OF #{n}}}
  #{data}
  END
    end
 
    def make_message
      @data = construct_test_data(@m, @n, @payload)
      @base_message = stub("Base", :data => @data)
      @it = OrderedMessage.new(@base_message)
    end
 
    describe "given message 1 of 3 with data FOO" do
      before :each do
        @m       = 1
        @n       = 3
        @payload = "FOO"
        make_message
      end
 
      it_should_behave_like "any ordered message"
 
      it "should not be the last message" do
        # Using '@it.should_not be_end' interacts badly with DelegateClass
        @it.end?.should_not be_true
      end
    end
 
    describe "given message 3 of 5 with data BAR" do
      before :each do
        @m       = 3
        @n       = 5
        @payload = "BAR"
        make_message
      end
 
      it_should_behave_like "any ordered message"
 
      it "should not be the last message" do
        # Using '@it.should_not be_end' interacts badly with DelegateClass
        @it.end?.should_not be_true
      end
    end
 
    describe "given message 4 of 4 with data BAZ" do
      before :each do
        @m       = 4
        @n       = 4
        @payload = "BAZ"
        make_message
    end
 
      it_should_behave_like "any ordered message"
 
      it "should be the last message" do
        # Using '@it.should_not be_end' interacts badly with DelegateClass
        @it.end?.should be_true
      end
    end
  end

Using delegation to extend FMTP::Message, we don’t need to instantiate an actual FMTP::Message object in order to test our modifications. All we have to do is stub the methods of FMTP::Message that we actually use in the delegate. This can be a real win when we are working with a third-party library which has extensive and/or poorly-documented dependencies.

Applicability

Consider using delegation when:

• Vendor code controls instantiation of the target.
• Your code is the primary client of the target.
• You need to make more than one addition or modification to the target’s interface.

Caveats

Be aware that delegates produces with the delegate library are not perfect stand-ins for their target objects. In particular, by default the delegate object will have a different id and object_id. This is easy to correct, but you should be aware of it, especially when working with ActiveRecord, which uses id to associate objects with rows in the DB.

Sure, monkey patching is great and all. That period of disbelief, followed by increasing exasperation as the victim maintenance programmer discovers that an object is behaving differently than it’s source code says it should, is satisfying. But sooner or later he or she wises up and greps through the codebase, discovers where you re-opened the class in question, and the game is up.

The fact of the matter is that monkeys simply aren’t very stealthy. They are easy to find when you know what you are looking for. When you really want to catch a coder by surprise, a monkey doesn’t cut it. What you need is a Ninja.

And so, today I’m unleashing the technique of Ninja-Patching, along with a reference implementation. Ninja-Patching is silent, untraceable, precise, unpredictable, and always deadly. And unlike monkey patching, which usually happens at startup, Ninja-Patching happens when you least expect it. Here’s an example use:

  require 'ninja'
  Ninja.hire(Enumerable) do def self.to_s "PWNED by Ninjas!!!" end

That’s all you have to do. A silent assassin has been hired, and will ruthlessly hunt down its target. In this case, the target is the first object found in ObjectSpace which is a kind of Enumerable. Which one? Hard to say. Ninjas are, like I said, unpredictable. Once the target is acquired, the Ninja will wait some random amount of time in order to throw off the trail and instill a false sense of security. Then, without warning, the Ninja will attack! The block given to Ninja.hire will be executed in the context of the target object. And then it’s all over but the crying.

Here’s the implementation:

CODE = <<'END_CODE'
class Ninja
  def self.hire(target_description, &instructions)
    self.new(target_description, instructions)
  end
 
  private
 
  # target_description can be either an object which responds to +#===+, or a
  # Proc which returns true or false.
  def initialize(target_description, instructions)
    @target_description = if target_description.kind_of?(Proc) then
                            target_description
                          else
                            lambda {|obj| target_description === obj}
                          end
    @instructions       = instructions
    @target             = acquire_target(@target_description)
    if @target.equal?(self)
      $stderr.puts "Never double-cross a Ninja!"
      exit
    elsif @target
      stalk(@target)
    else
      raise "No such object found!"
    end
  end
 
  def acquire_target(target_description)
    ObjectSpace.each_object do |object|
      if target_description.call(object)
        return object
      end
    end
    nil
  end
 
  def stalk(target)
    Thread.new do
      sleep(rand(60))
      attack!(target)
    end
  end
 
  def attack!(target)
    target.instance_eval(&@instructions)
  end
end
END_CODE
 
# Using eval conceals the Ninja in the stack trace
eval CODE

This code is released under the Ninja Public License (NPL), which releases me from all liability should ninja.rb turn on you and assasinate you in your Kernel#sleep(). Warning: ninja.rb has a known vulnerability to chosen_one.rb.

The first technique we’ll look at in this series is something so basic it may not even seem worth spelling out. But sometimes old-school techniques are overlooked in the excitement of a young language.

Let’s use as our example a hypothetical communications protocol, Flying Monkey Transport Protocol (FMTP). Flying Monkey Transport Protocol is a packet-based peer-to-peer networking protocol in which messages are transported from one peer to another by means of flying monkeys carrying satchels full of data.

As developers in the inter-kingdom IT department, it’s our job to make sure that communications between e.g. the Wicked Witch of the East and the Lollipop Guild flow unimpeded. Where wicked witches are concerned it’s important that no one get mixed messages.

The interface for the Ruby FMTP implementation looks something like this:

module FMTP
  class Connection
    def initialize(address)
      # ...
    end    
 
    def send(message)
      # ...
    end
 
    def receive
      # ...
    end
  end
 
  class Message
    # ...
    attr_reader :data
  end
end

Once a connection is initialized, we can receive messages from the opposite peer by calling receive, which returns a Message object:

  connection = FMTP::Connection.new("witch.east")
  message = connection.receive

Lately the Wicked Witch of the East has gotten rather chatty in her old age, and her messages have been exceeding maximum monkey capacity. As a result, we’ve been forced to start dividing her messages up across multiple monkeys. Unfortunately, the writers of the FMTP library did not plan for this possibility, so recipients of the Witch’s communiques have been getting truncated messages. We’ve been tasked with making the necessary changes in order to support multi-monkey messages.

As good Ruby programmers, we like to exploit the language’s dyanamic features to the max. And at first, this might seem like the perfect opportunity to use Ruby’s capacity for runtime class modification. We’ll just re-open the class and patch it to do what we need:

module FMTP
  class Connection
    alias_method :receive_without_buffer, :receive
    def receive
      buffer = ""
      begin
        message = receive_without_buffer
        buffer << message.data
      end until(message.data.include?("ENDENDEND"))
      Message.new(buffer)
    end
  end
end

But there’s another way to accomplish the same ends. A simpler, low-tech way: inheritance.

Inheritance has gone somewhat out of fashion in recent years. And not without reason. In the old days inheritance was seen as almost synonymous with object-orientation, and as a result it was frequently abused. Programs would consist of elaborate, many-leveled inheritance heirarchies that resembled an inbred royal family tree. These programs were hard to understand and hard to maintain.

Ruby programmers have, for the most part, learned their lesson well in this regard. I rarely see a Ruby application with more than two layers of inheritance. For the most part this is a good thing. But occasionally the avoidance of inheritance leads to implementing more complex solutions in places where inheritance is a perfectly legitimate technique.

This is one of those cases. Here is how the code would look using inheritance:

  class BufferedConnection < Connection
    def receive
      buffer = ""
      begin
        message = super
        buffer << message.data
      end until(message.data.include?("ENDENDEND"))
      Message.new(buffer)
    end

And here’s how it’s used:

  connection = BufferedConnection.new("witch.east")
  message = connection.receive

The difference is small, to be sure. But the inheritance version has a number of advantages. It’s slightly shorter. It’s simpler, because there is no need to alias the original method to a new name; we can just use super. The name BufferedConnection makes it obvious that we are using a buffered variant of a Connection. There’s no chance of our becoming confused by the disparity between what the original #receive method says, and how it actually behaves. And we know that since we have to explicitly ask for the buffered version, there’s no chance of our inadvertantly breaking code somewhere else in the program by changing the semantics of Connection.

It might seem like too obvious a technique to even mention. But it’s easy to forget about the prosaic solutions in a language that gives us so many possibilities. You should still put some thought into whether inheritance isappropriate in any given situation. So long as it is a legitimate IS-A relationship and the Liskov Substitution Principle is satisfied, though, there’s nothing wrong with a little good old-fashioned inheritance.

Applicability

Consider using inheritance when:

• You control object creation.

This is the beginning of a series of posts on sustainable development in Ruby. No, I’m not talking about writing code on wind-powered laptops while sipping fair-trade coffee. But the sustainable development movement has a fairly direct analog in software development. As programmers, we work within a code ecosystem. The ease with which we write new programs is impacted by the choices of other coders before us, and likewise the decisions we make while coding affect other programmers down the line.

As with industrial development, in the early years of a particular software ecosystem it’s not always obvious that the choices we are making might be detrimental to our successors.

the first generation of programmers is usually enthusiastic; any failure is a personal failure, so you can gloss over those things. It’s the second generation that’s going to be less enthused, that’s going to stare in bafflement at these classes that mysteriously spawn methods, and trying to figure out what’s going when there’s an exception in dynamically generated code.

— Ian Bicking

When a programming language is relatively young, unsustainable practices often go unnoticed, or are dismissed as easily avoidable.

A mere flesh wound, says our programming primate: I usually don’t get conflicts, so I’ll pretend they won’t happen. The thing is, as thing scale up, rare occurrences get more frequent, and the costs can be very high.

— Gilad Bracha

When a language is only a few years old, the amount of code written in it is necessarily small. Legacy code is measured in the thousands of lines, rather than hundreds of thousands. If a section of code, or a third-party library, is causing problems – well, you can always rewrite it. As codebases grow, however, the rewrite option becomes less and less viable. Coping with legacy code is an everyday fact of life for most professional programmers.

Parameters

If we take it to mean any and all techniques for making software more robust and easy to maintain, sustainable software development is a very broad subject. Indeed, one could argue that most of the major advances in the software field in the last 30+ years have been made with sustainability in mind – OO, refactoring, TDD, to name a few.

In this and the following essays I’m only going to be addressing one specific aspect of sustainability in the Ruby language. I’m going to be talking about the practice of dynamic class modification, colloquially “monkeypatching”. I’m addressing this subject because I believe injudicious use of dynamic class modification to be one of the greatest threats to long-term sustainability currently facing Ruby.

Definitions

What are we talking about when we say monkeypatching? Some divide dynamic class modification into two categories:

• Runtime addition of methods
• Runtime redefinition (overwriting) of methods

Some maintain that only the latter definition is true monkeypatching. The line is blurrier than it might first appear, however. If two separate libraries both add a #to_xml method to the Object class, then individually they are only adding a method, but when both libraries are required by the same program one will be overwriting the other – whichever one is loaded last. For this reason I will use monkeypatching to mean both the dynamic addition and redefinition of methods, albeit with an emphasis on redefinition.

Specifically, I will be referring to dynamic non-local addition and redefinition of methods. By non-local, I mean that the dynamic modification occurs outside of the original class definition. The following is not a monkeypatch:

  class Foo
    attr_accessor :bar # defines bar, bar=
 
    # redefines bar
    def bar
      # ...
    end
  end

Whereas this version demonstrates monkeypatch:

  class Foo
    attr_accessor :bar # defines bar, bar=
  end
 
  # re-open the class
  class Foo
    # redefines bar
    def bar
      # ...
    end
  end

No easy answers

this “monkey patching” thing is seriously powerful

— Chad Fowler

languages—like Ruby—that include dangerous features give the fringe a broader latitude to invent new things. Of course, they also break things and they invent stupid things and they get excited and write entire applications by patching core classes instead of writing new classes and commit all sorts of sin.

— Reginald Braithwaite

One of the things that’s really great about agile languages is they give you the power to do anything. One of the most horrible things about agile languages is they give every other idiot the same power to stab you in the back with a rusty pitchfork.

— Zed Shaw

This series will not tell you when to monkeypatch and when not to. It is not my intent to set myself up as the arbiter of when monkeypatching is justified. These posts make the assumption that you already understand that Ruby dynamism is both tremendously powerful and potentially dangerous. I want to present some alternatives to monkeypatching, so that you can make an informed decision when deciding whether to use monkeypatching to solve a particular problem.

Conventions

In order to help you make that judgement, I’ll be characterizing the techniques presented by their applicability. Some of the aspects affecting applicability are introduced below. In this context, the term vendor is used to refer to any code you don’t have the ability to change upstream – whether from a third-party library, or written by another team down the hall. The term target is used to mean a vendor-defined object whose methods you wish to modify.

• Who controls creation of the target? – does your code call the class constructor, or is the object given to you already created by vendor code?
• Are you the only client? – is the target used by third-party code, or is your code the only code that touches it after it is created?
• Can you intercept the target? – if the target is produced and consumed by vendor code, is there still a point at which your code has access to it? Or is it’s use completely internal to vendor code?

These factors and others will affect which techniques are right for any given case.