How robots.txt is helpful for your site?

We all know about robots.txt. robots.txt is a simple file, which allow search engine bots to crawl or don't crawl the pages of your website. You can guide search engine bot by your own way. You can select control access to all of your files folders etc. by your self. For more information on robots.txt please click here.