The velocity of Google spidering and getting the info in the search results has been amazing. It used to be a couple of days 2 years ago, now it is hours.
mybloglog also mentions recently in their blog that they were being overly spidered (had to shut down) and them to knock it off. It’s great that googles only last a few hours but what about the other couple thousand spiders that crawl when they feel like it. It is an absolute disruption to service.
Tom says:
The velocity of Google spidering and getting the info in the search results has been amazing. It used to be a couple of days 2 years ago, now it is hours.
July 10, 2007 — 10:36 am
B.R. says:
mybloglog also mentions recently in their blog that they were being overly spidered (had to shut down) and them to knock it off. It’s great that googles only last a few hours but what about the other couple thousand spiders that crawl when they feel like it. It is an absolute disruption to service.
July 10, 2007 — 2:34 pm
Greg Swann says:
Note that I wasn’t complaining. We get spidered a lot and I like it. I just thought it was funny that they caught ’em with their pants down.
July 10, 2007 — 2:45 pm
Mike Elliott says:
I imagine things like this happen a lot these days.
July 11, 2007 — 11:46 am