You might not want sure web pages of your internet site crawled because they might not be beneficial to end users if found in a online search engine's search results. Note that if your web site takes advantage of subdomains and you would like to own specific pages not crawled on a certain subdomain, you will have to create a different robots.In rea