Specialized Website positioning: Technical Search engine optimisation refers to Web page and server optimization that assists the crawler in crawling, indexing and ranking functions to rank your internet site much better. txt file is then parsed and will instruct the robot as to which webpages are certainly not being crawled. https://www.youtube.com/watch?v=hgyQy18ZcIs