Nov 22, 2009
Pages: 1, 2

Robots.txt Introduction - bots and spiders crawling the web

free web hosting
Open Discussion & Free Web Hosting > Computers & Tech > How-To's and Tutorials > Websites and Web Designing

Robots.txt Introduction - bots and spiders crawling the web

NilsC
When you are hosted you get access to cPanel, in cPanel there are several folders for the different options. When you upload your website, you upload it to a folder called "Public_html" this folder is also called your root html folder. So you put all the files hat belongs to this domain name "system482.astahost.com" in here, and when you get the account there is a default file (and others) called index.html in this folder. Anything you put in this folder is hotlinked to your www folder so you can get to your site by typeng
CODE
http://system482.astahost.com/
or http://system482.astahost.com/index.html
or http://www.system482.astahost.com/
or http://www.system482.astahost.com//index.html

There are private folder in there that you can only see if you are logged into cPanel or by ftp.

The robots.txt file goes in that directory "public_html" and you will see your site on the web.
spiders and robots search your
CODE
http://system482.astahost.com/
directory for a file called robots.txt and if you put it in a subdirectory it's ignored. so putting a robots.txt files in the video_games/ subdirectory will get it ignored.

Hope this helps.

Nils

 

 

 


Comment/Reply (w/o sign-up)

m3ch4
QUOTE (NilsC @ May 3 2005, 12:00 AM)
Hope this helps.

Nils
*


Answers everything!

Thanks a ton man! (I've yet to make my way through the astahost cpanel, I have approval already, but I have a few other things to deal with before I can go full tilt on my site =S)

Comment/Reply (w/o sign-up)

mitchellmckain
If you disallow everything

User-agent: *
Disallow: /

this does not block the web crawlers from your index.html, right? It just blocks them from indexing the directory contents, right?

Comment/Reply (w/o sign-up)


Got an Opinion! Express your Views! (no registration):-
Add your Reply/ Opinion/ Views/ Comments/ Suggestion/ Questions/ Queries etc.
Posts with decent grammar & English will be accepted and please refrain from profanities.
For asking a Question, We recommend you to sign-up (for free) so that you can track the topic easily.

Nature of your Post*: Opinion/ Reply/ Comments
Question/Query
Feedback to us.
       
Name   Email
Title/Question*

This textarea will convert to Rich-Text automatically (IE, Firefox, Chrome)

Pages: 1, 2

See Also,

*SIMILAR VIDEOS*
Searching Video's for robots, txt, introduction, bots, spiders, crawling, web
advertisement



Robots.txt Introduction - bots and spiders crawling the web

Affordable Web Hosting, Low cost Web Hosting - ComputingHost.com