What if search was opt-in?

A woman in Colorado is suing Archive.org because they spidered her site and she has a warning up that says you can’t do that.  To humans, it says this.  To spiders and bots, it’s just more text.

Is Archive.org doing anything wrong, or are they providing a valuable service?  And should webmasters have to opt in before Archive spiders them, rather than opting out via the robots.txt file? Do we want to have no way of knowing what’s been on a site in the past, if we’re considering buying it?  Or is that a privacy right?

And what about search engines?  While this situation is different, it raises the question: is it fair that someone who knows diddly about the web might not even realize they need to opt out with robots.txt or their site will be spidered?  Is that their problem for not educating themselves more?

Lots of interesting questions.