Robots.txt is often a file that exists on the root directory of each website and may be used to instruct search engines like yahoo on which directories/information of the web site they could crawl and consist of inside their index. Search engine optimisation (Search engine optimization) is the whole process https://technik.forum