PDA

View Full Version : how to stop the search bots?

jdleung
07-10-2006, 11:14 PM
my site is ceaselessly visited by google and baidu search bots.

how can I stop it?

Thanks.
GeoS
07-10-2006, 11:29 PM
http://en.wikipedia.org/wiki/Robots.txt
blinking8s
07-10-2006, 11:36 PM
you need a file called robots.txt in your servers root, in the contents of the file


User-Agent: Googlebot
Disallow: /index.php


just google 'robots.txt' for more information or google 'prevent google spider' or something

*edit* lol GeoS and me were writing our reply at the same time...yeah, what the guy above said!
jdleung
07-10-2006, 11:41 PM
will this cause my site not being searched by them?
I mean if visitor can not search my site in google and baidu?

btw,GeoS give me a bad link.
Thanks.
blinking8s
07-11-2006, 02:43 AM
the link geos gave you works for me

yes, if you disable spiders from hitting your site, your content will nto be indexed by google aor whichever spider you block
jgrayson
07-11-2006, 04:45 AM
will this cause my site not being searched by them?
I mean if visitor can not search my site in google and baidu?

btw,GeoS give me a bad link.
Thanks.

try this link:
http://en.wikipedia.org/wiki/Robots_Exclusion_Standard

i'm guessing the reason why the other link didn't work is because your browser thinks http://en.wikipedia.org/wiki/Robots.txt is a text file because of the .txt - the other link i provided goes to the same page. whoever named that directory Robots.txt is an idiot
jdleung
07-11-2006, 06:26 AM
both the two link don't work. :-(
blinking8s
07-11-2006, 06:46 AM
just google robots.txt and you will get what you are looking for

both the links work, talk to your ISP
jdleung
07-11-2006, 06:56 AM
yes, I add robots.txt at the root. actually google rarely visit my site, but baidu.com visit my site over 1000 times by now. I had mailed them to complain this.I got to delete the visitro list manually. :-(
jdleung
07-11-2006, 06:58 AM
but still I want google and baidu to search my site.
blinking8s
07-11-2006, 07:08 AM
why does it bother you so much? pulling too much bandwidth?
GeoS
07-11-2006, 08:26 PM
but still I want google and baidu to search my site.

So you must to decide what you really want.
jdleung
07-12-2006, 02:41 AM
why does it bother you so much? pulling too much bandwidth?
500 times visiting a day. it makes my visitor number goes high. actually I should be happy ;-), and yes, I worry about bandwidth.

So you must to decide what you really want.
I don't want them visit the site so frequently. baidu is too hopping!
blinking8s
07-12-2006, 03:16 AM
i use http://www.haveamint.com for my photoblog traffic stats, and have the google bots and such disabled from the counters.
jdleung
07-12-2006, 03:23 AM
it's professional and cost money.
I just don't want search bots bother me too much.
GeoS
07-12-2006, 06:25 AM
So for stats use Google Analytics and remove poor counter implementation from blog template. If baidu is comming too frequently then disable it in robot.txt for a week or something like that.
jdleung
07-12-2006, 09:53 AM
So for stats use Google Analytics and remove poor counter implementation from blog template. If baidu is comming too frequently then disable it in robot.txt for a week or something like that.
yes, I'm going to disable baidu. I had mailed to them. waiting their reply.