Advanced

Robots.txt Tips For Deailing With Bots

Posted by aweber on · Post a comment
BeginLinux.com/blog; By Mike Weber
Mail this story
Print this story

A few tips I put together while re-creating the robots.txt file on my Linux web server. The robots.txt is used to provide crawling instructions to web robots using the Robots Exclusion Protocol. When a web robots visits your site it will check this file, robots.txt, to discover any directories or pages you want to exclude from the web robot listing on the search engine. This is an important file which determines SEO for search engines and can help rankings.

Full Story

  Nav
» Read more about: Story Type: Tutorial; Groups:

» Get new stories by e-mail: nightly or as they are published

« Return to the newswire homepage

  Related stories
» More like this

Comments

No comments yet. Be the first.

Post a comment

A free account is needed to post. Write your comment here first: your text is kept in this browser while you sign up or log in, and is waiting for you when you come back.

Your message:
|||||
10
Markdown: **bold**, *italic*, [text](https://link), - lists, > quotes, $x^2$ for math. Formatting help

Sign up and post (it is free)  or  Log in and post