Enhancing WWW::Mechanize

Today I wanted to extract a report from a search appliance. This task involved logging in to the appliance, receiving a cookie, and following a few links. No problem with Mechanize, right?

agent = WWW::Mechanize.new
login_form = agent.get(THUNDERSTONE_URL).forms.first
login_form.set_fields(:iname => LOGIN, :ipass => PASSWORD)
page = agent.submit(login)
page = agent.click page.links.text(PROFILE_NAME)
errors = agent.get_file "#{page.uri}/tsverrors.csv"

Only it turned out that it wasn't receiving the cookie. After a bit of Firebug inspection, I realized that the cookie wasn't being sent in the HTTP headers, but instead in the document itself as a meta http-equiv. Evidently Mechanize doesn't pick those up. We can use the nifty built-in Hpricot Xpath parsing and extract the value of the cookie:

page.search("//meta[@http-equiv = 'Set-Cookie']").first.attributes['content']

But how do we tell Mechanize to use it? The cookie() method is read-only. Fortunately, set_headers() is a protected method that is available to subclasses. So, I decided to create a new class called Agent that adds a custom cookie to the headers.

class Agent < WWW::Mechanize
  attr_accessor :custom_cookie
 
  def set_headers(uri, request, cur_page)
    super(uri, request, cur_page)
    request.add_field('Cookie', custom_cookie) unless custom_cookie.nil?
    request
  end
end

Then I tried again with the new Agent class:

agent = Agent.new
login_form = agent.get(THUNDERSTONE_URL).forms.first
login_form.set_fields(:iname => LOGIN, :ipass => PASSWORD)
page = agent.submit(login)
agent.custom_cookie =
  page.search("//meta[@http-equiv = 'Set-Cookie']").first.attributes['content']
page = agent.click page.links.text(PROFILE_NAME)
errors = agent.get_file "#{page.uri}/tsverrors.csv"

It worked like a charm. I'm curious though why mechanize didn't grab the meta-cookie automatically. Bug or feature?