robots.txt

Overview

Some of the most useful clues in web enumeration are ones the site publishes on purpose — just not for you. robots.txt is a plain-text file websites use to tell search-engine crawlers which paths not to index. Ironically, that list of "please do not visit" paths is a gift to anyone enumerating the site, because it names directories the owner considers sensitive. This lesson explains what robots.txt is, the related sitemap.xml, and how to mine them for leads in an authorized target.

Learning objectives

• Explain the purpose of robots.txt and how crawlers use it.
• Describe why its Disallow entries are valuable during enumeration.
• Use robots.txt and sitemap.xml to find paths not linked from the UI.

Core concept

robots.txt is a file placed at the root of a site (/robots.txt) following the Robots Exclusion convention. It lists rules for automated crawlers: which paths they are asked to avoid (Disallow) and sometimes which they may crawl (Allow), often grouped by User-agent. It is purely advisory — it politely requests that well-behaved crawlers skip certain paths. It is not a security control: it does not block anyone, and the paths remain fully accessible.

For enumeration, the value is precisely in what owners choose to disallow. To keep something out of search results, they list its path — which means robots.txt frequently names admin panels, backups, staging areas, private directories, and other sensitive paths that are not linked anywhere else. You are handed a curated list of the paths the owner most wanted hidden.

How it works

• Fetch it directly. Request GET /robots.txt. If it exists, you get a text file with lines like User-agent: *, Disallow: /admin/, Disallow: /backup/, Disallow: /private/.
• Read every Disallow and Allow. Each Disallow path is a candidate to investigate. Because robots.txt does not enforce anything, you can simply request those paths and see what responds.
• Check sitemap.xml. Many sites also expose a sitemap.xml (sometimes referenced from robots.txt) that lists URLs for crawlers to index. This is another ready-made map of pages, occasionally including ones not linked in the visible navigation.
• Combine with other enumeration. Treat the disallowed paths as high-priority entries in your route map and feed them into directory discovery — a Disallow: /admin/ invites you to enumerate inside /admin/.

Practical example

You are enumerating an authorized CTF web app. You request /robots.txt and receive:

User-agent: *
Disallow: /admin/
Disallow: /backup/
Disallow: /old-site/

None of these are linked from the site. You request /old-site/ and find a leftover version of the application with weaker checks — exactly the sort of forgotten area a challenge might hinge on. The owner never intended you to see /old-site/; they only wanted it out of search results, and in doing so they pointed you straight to it. You then fetch /sitemap.xml, which lists a few additional pages to add to your map.

Cybersecurity relevance

robots.txt is a textbook example of security through obscurity failing: hiding a path from crawlers does nothing to protect it, and publishing the list actively advertises sensitive locations. In real assessments, robots.txt is one of the first files a tester reads because it so often reveals admin interfaces, backups, and internal tools. The defensive lesson is equally important: never rely on robots.txt to protect anything — enforce real authentication and authorization on those paths, and do not list sensitive directories there. Reading these files teaches you both a fast enumeration win and a common misconfiguration to flag.

Common mistakes

• Treating robots.txt as a security boundary — it blocks nothing and is readable by anyone.
• Skipping robots.txt and sitemap.xml during enumeration and missing an easy list of sensitive paths.
• Assuming a disallowed path is protected — it may be wide open; always request it (within scope) to check.
• Ignoring the paths as leads rather than feeding them into deeper directory discovery.

Key takeaways

• robots.txt is an advisory file listing paths crawlers should avoid; it enforces nothing and is public.
• Its Disallow entries often name admin panels, backups, and private areas not linked elsewhere, making it a prime enumeration source.
• Also check sitemap.xml for an additional list of URLs, and feed all discovered paths into your route map and directory discovery.
• Defensively, robots.txt is security through obscurity; sensitive paths must be protected by real access control.

Check your understanding

• Why does listing a path in robots.txt fail to protect it, and how does that help an enumerator?
• What kinds of paths do owners commonly place in Disallow entries, and why are they valuable leads?