Mechanize to the rescue

February 10, 2009

If you’re using Ruby to download pages via http, you have a couple of standard choices.  Open-uri is the easiest method, as it abstracts the download and treats it just like a file.  If you need to do something like examine headers or do a POST instead of a GET, you can step down into net/http and be a little closer to the details.

If you need the complexity of sharing cookies across a session and handling re-directs from an MS IIS web server, then don’t spend two days trying to get these two to work like I just did.  Jump straight to Mechanize – just install the gem.  It handled the complex re-directs, the cookie sharing, and both GETs and POSTs with a magical ease.

I haven’t gone any further with it, but apparently Mechanize can do a lot of other stuff too, like find fields and fill out forms.  In combination with Hpricot, it would be a natural for some pretty complex web spidering.

Hope this helps save someone else two days.


  1. Noted, thanks for sharing. And I am sure someday it will save me or someone else some grief!

  2. I love Mechanize. For the perl folks in the room, the ruby Mechanize is based on the perl package of the same name.

    I’ve used it to automate a few things at work, such as forwarding my phone through the web interface (which requires some form work) and a few other things like that.

  3. Nice post. I learn something totally new and challenging on blogs I stumbleupon every day.
    It’s always interesting to read through content from other writers and practice a little something from other web sites.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

%d bloggers like this: