Tzetze Fly

If you’re using Ruby to download pages via http, you have a couple of standard choices.  Open-uri is the easiest method, as it abstracts the download and treats it just like a file.  If you need to do something like examine headers or do a POST instead of a GET, you can step down into net/http and be a little closer to the details.

If you need the complexity of sharing cookies across a session and handling re-directs from an MS IIS web server, then don’t spend two days trying to get these two to work like I just did.  Jump straight to Mechanize – just install the gem.  It handled the complex re-directs, the cookie sharing, and both GETs and POSTs with a magical ease.

I haven’t gone any further with it, but apparently Mechanize can do a lot of other stuff too, like find fields and fill out forms.  In combination with Hpricot, it would be a natural for some pretty complex web spidering.

Hope this helps save someone else two days.