Robots.txt is a file that exists on the root Listing of each Web site and can be utilized to instruct search engines like google on which directories/data files of the website they will crawl and consist of inside their index. A sitemap is a simple file that lists all URLs https://news.rafeeg.ae