Crawl.googlebot.com

yordan

Apr 26 2009, 02:47 PM

I had a look at my private forum, and I saw a guest connected. I had a look at his IP address, did a whois, and found that this guest was
crawl-66-249-73-137.googlebot.com
How did this googlebot come to my private website ?
Is there a way to prevent these bots from crawling my website ?

Comment/Reply (w/o sign-up)

hfbvm

Apr 26 2009, 05:49 PM

just make a robots.txt file.robots.txt will work as you want. You can even disable crawling for specific pages, so you don't have to move your pages into a folder. Something like:

CODE

User-agent: *
Disallow: /index.php
Disallow: /main.php
Disallow: /inc/
Disallow: /img/
Disallow: /images/
Disallow: /js/
Disallow: /css/

And so on and so forth its easy more queries ask me maybe i could help you

Comment/Reply (w/o sign-up)

yordan

Apr 26 2009, 06:08 PM

Seems to be exactly what I want. Thanks, I'll try that way.

Comment/Reply (w/o sign-up)

hfbvm

Apr 26 2009, 07:39 PM

where's my rep for this?

Comment/Reply (w/o sign-up)

yordan

Apr 26 2009, 07:51 PM

QUOTE (hfbvm @ Apr 26 2009, 09:39 PM)

where's my rep for this?

Your rep is here : http://www.astahost.com/index.php?s=&s...st&p=136357
Can't you really see it ?

Comment/Reply (w/o sign-up)

zorba4

Apr 26 2009, 08:14 PM

QUOTE (hfbvm @ Apr 26 2009, 07:39 PM)

where's my rep for this?

I also see your rep, you said :

QUOTE

User-agent: *
Disallow: /index.php
Disallow: /main.php
Disallow: /inc/
Disallow: /img/
Disallow: /images/
Disallow: /js/
Disallow: /css/

does your browser have a problem? I use IE7

Comment/Reply (w/o sign-up)

yordan

Apr 26 2009, 08:16 PM

QUOTE (zorba4 @ Apr 26 2009, 10:14 PM)

does your browser have a problem? I use IE7

I use Firefox 2, and I see hfbvm's rep. Maybe he started sleeping ?

Comment/Reply (w/o sign-up)

zorba4

Apr 26 2009, 08:38 PM

QUOTE (yordan @ Apr 26 2009, 08:16 PM)

Maybe he started sleeping ?

Or he forgot refreshing his browser window?

Comment/Reply (w/o sign-up)

hfbvm

Apr 27 2009, 01:30 PM

yes i can now see it sorry for the inconvenience or whatever.
I also use firefox.

Comment/Reply (w/o sign-up)

aloKNsh

Apr 28 2009, 08:36 AM

yeah i can also see....
and i use safari

Comment/Reply (w/o sign-up)

Latest Entries

yordan

May 3 2009, 12:16 PM

QUOTE (FirefoxRocks @ May 3 2009, 05:11 AM)

I'm using Firefox 3 on Ubuntu 9,04 at the moment. Just installed it today.

Exactly what I said. Linux+windows people use Firefox.
Mac people send you mails with .websave files, and they a surprised learning that safari is not your default browser.

Comment/Reply (w/o sign-up)

FirefoxRocks

May 3 2009, 03:11 AM

QUOTE (yordan @ Apr 28 2009, 04:44 AM)

Why safari ? Are you a Mac User ?

You do realize that there is also Safari for Windows. It is very similar to Google Chrome minus some features.

I'm using Firefox 3 on Ubuntu 9,04 at the moment. Just installed it today.

Comment/Reply (w/o sign-up)

Crawl.googlebot.com - how and why ?

Crawl.googlebot.com - how and why ?

Latest Entries