A question regarding spiders

[cooba]

Yesterday, two spiders from MSN came and searched for JJ3 stuff. Today, there are a lot of Google spiders wandering around randomly. Weren't they disabled on JCF?

[Tubz]

What the hell is the difference between a Google Spider and a normal Google search bot?

[Bobby aka Dizzy]

I believe the MSN spider chooses to ignore the request to not cache our pages.

[Link]

I believe the MSN spider chooses to ignore the request to not cache our pages.

Actually, the robots.txt file was lost in the server crash.

[Bobby aka Dizzy]

Well I believe certain versions of the msnbot still choose to ignore it but that's true too.

[KRSplat]

What the hell is the difference between a Google Spider and a normal Google search bot?
Spiders search websites and cache every single page that there is a link to. I don't really know what I'm talking about, though.