I doubt they ignore the robots.txt file. We just don't block everything on the JCF.
I don't think this is really a problem, and it will allow more people to find this site if JCF threads are archived. If the bandwidth use becomes more of a problem, we may need to further limit the search spiders.
__________________
With our extreme gelatinous apology,
We beg to inform your Imperial Majesty,
Unto whom be dominion and power and glory,
There still remains that strange precipitate
Which has the quality to resist
Our oldest and most trusted catalyst.
It is a substance we cannot cremate
By temperatures known to our Laboratory.
~ E.J. Pratt
|