Control Panel based webhosting in South Africa with Graphical Stats, PHP and MySQL
Choose from different packages to suit your requirements
Register South African and other domains
Secure hosting with backup power generators, daily backups and 99.9% guaranteed uptime
Generous bandwidth and disk space allocations
24-hour email support and helpline
Sign up today and receive a free Search Engine Optimization and Web Marketing Guide
|
 |
Glossary of Terms
Robots
Search engines use special systems called web spiders or robots to crawl the web looking for web pages to index.
Websites can add a text file, in the root folder, called robots.txt, which contains instructions for robots on how your site may be accessed.
You may, for example, deny access to a particular robot or specify which pages may be accessed.
Example robots.txt
# /robots.txt file for http://mysite.co.za/
User-agent: webcrawler
Disallow:
User-agent: bad-robot
Disallow: /
User-agent: *
Disallow: /stay_out
Disallow: /logs
The first line, starting with '#', specifies a comment.
The first paragraph specifies that the robot called 'webcrawler' has nothing disallowed: it may go anywhere.
The second paragraph indicates that the robot called 'bad-robot' has all relative URLs starting with '/' disallowed.
Because all relative URL's on a server start with '/', this means the entire site is closed off.
The third paragraph indicates that all other robots should not visit URLs starting with /stay_out or /logs.
Related Terms |
Return to Glossary of Terms
|
