Mechanize to the rescueFebruary 10, 2009
If you’re using Ruby to download pages via http, you have a couple of standard choices. Open-uri is the easiest method, as it abstracts the download and treats it just like a file. If you need to do something like examine headers or do a POST instead of a GET, you can step down into net/http and be a little closer to the details.
If you need the complexity of sharing cookies across a session and handling re-directs from an MS IIS web server, then don’t spend two days trying to get these two to work like I just did. Jump straight to Mechanize – just install the gem. It handled the complex re-directs, the cookie sharing, and both GETs and POSTs with a magical ease.
I haven’t gone any further with it, but apparently Mechanize can do a lot of other stuff too, like find fields and fill out forms. In combination with Hpricot, it would be a natural for some pretty complex web spidering.
Hope this helps save someone else two days.