If you’re using Ruby to download pages via http, you have a couple of standard choices. Open-uri is the easiest method, as it abstracts the download and treats it just like a file. If you need to do something like examine headers or do a POST instead of a GET, you can step down into net/http and be a little closer to the details.
If you need the complexity of sharing cookies across a session and handling re-directs from an MS IIS web server, then don’t spend two days trying to get these two to work like I just did. Jump straight to Mechanize – just install the gem. It handled the complex re-directs, the cookie sharing, and both GETs and POSTs with a magical ease.
I haven’t gone any further with it, but apparently Mechanize can do a lot of other stuff too, like find fields and fill out forms. In combination with Hpricot, it would be a natural for some pretty complex web spidering.
Hope this helps save someone else two days.
I had originally titled this “Installing Ruby on Rails on Mac OSX”, but as I typed, it gradually became about this other, more general, topic.
I’m doodling around with a new idea. Since it’s goal is to produce several standardized web feeds (Atom, KML, HTML) of a data set over the web, I’ll be using Ruby. Since it makes sense for this to be a REST web service, I’ll also be using Rails. Mike asked about Rails yesterday, and I just read a GREAT introduction by the 
