Robots.txt Introduction - bots and spiders crawling the web

When you are hosted you get access to cPanel, in cPanel there are several folders for the different options. When you upload your website, you upload it to a folder called "Public_html" this folder is also called your root html folder. So you put all the files hat belongs to this domain name "system482.astahost.com" in here, and when you get the account there is a default file (and others) called index.html in this folder. Anything you put in this folder is hotlinked to your www folder so you can get to your site by typeng

There are private folder in there that you can only see if you are logged into cPanel or by ftp.

The robots.txt file goes in that directory "public_html" and you will see your site on the web.
spiders and robots search your directory for a file called robots.txt and if you put it in a subdirectory it's ignored. so putting a robots.txt files in the video_games/ subdirectory will get it ignored.

Hope this helps.

Nils

 

 

 

Comment/Reply (w/o sign-up)

Answers everything!

Thanks a ton man! (I've yet to make my way through the astahost cpanel, I have approval already, but I have a few other things to deal with before I can go full tilt on my site =S)

Comment/Reply (w/o sign-up)

If you disallow everything

User-agent: *
Disallow: /

this does not block the web crawlers from your index.html, right? It just blocks them from indexing the directory contents, right?

Comment/Reply (w/o sign-up)