robots.txt is a text file containing website crawling rules for automated bots. On a Joomla website, it can restrict crawling of selected paths and identify the sitemap location.
What the file contains
The file belongs at the site root and is available at /robots.txt. Rule groups begin with User-agent to identify the bot. Disallow blocks matching paths from crawling, Allow provides exceptions and Sitemap gives the complete sitemap address.
Rules apply to a specific scheme, host and port. A main domain file therefore does not replace the rules for a separate subdomain. Support for additional directives varies between bots.
Limits of robots.txt
The file controls crawling rather than password protection. Visitors can read it and attempt to open any listed address. Protect confidential areas through server and application access controls.
Disallow does not guarantee that a URL disappears from search results. To process noindex, a crawler must receive the page or response header containing that directive. Blocking crawling can prevent it from seeing noindex.
Checks for Joomla
After editing robots.txt for Joomla, confirm that it loads as plain text and does not block important pages, CSS, JavaScript or images. Evaluate service paths in the context of the installed Joomla extensions.
When moving Joomla from staging to production, check for a leftover site-wide crawl restriction. Compare the rules with webmaster reports and request logs when diagnosing unexpected behaviour.
Conclusion
robots.txt helps manage crawling and resource use. Excluding pages from indexing and protecting data require separate mechanisms.