A robots.txt file tells search engine crawlers which URLs they can access on your site. This is used mainly to avoid overloading your site with requests and to prevent crawling of certain private or redundant content.
Specify rules for different search engine crawlers.
Allow or disallow access to specific URLs and directories.
Specify the location of your XML sitemap.
Set crawl-delay to manage server load.
Use the form above to select your user agent and add rules. You can add multiple rules for different user agents, specify allowed and disallowed paths, add your sitemap URL, and set crawl delays. Once you're done, copy the generated robots.txt content and save it as robots.txt in your website's root directory.