Robots.txt is a crawl instruction
A robots.txt file tells compliant crawlers which paths they may request. It is useful for managing crawl activity, but it is not a reliable way to remove a URL from search results and it should not be used to hide private information.
Keep the file at the root of the host and check that it does not block pages, CSS, JavaScript, or images needed to understand public pages.
An XML sitemap is a discovery aid
A sitemap should contain the canonical, indexable URLs that matter to your site. It helps communicate your preferred URL set, but it does not guarantee that every URL will be indexed.
Keep redirects, duplicates, thin utility URLs, and pages you deliberately do not want indexed out of the sitemap. Update it as your public content changes.
Check the whole crawl path
A crawler needs to reach the page, receive a successful response, understand the content, and discover useful links to other pages. Check server status, redirects, page size, source, headings, and internal links as one system.
When you find an issue, fix the cause and retest the final URL. A report is only valuable when it leads to a clear action.
Put this into practice
Use the free GigaTools toolkit
Run a check, review the result, and make one useful improvement at a time.
Quick answers
Frequently asked questions
Does robots.txt block indexing?
Not reliably. It controls crawling, while indexation is managed through page signals and search engine systems. Do not use robots.txt as a privacy control.
Should every URL be in the sitemap?
Include important canonical pages that are available for indexing. Exclude duplicates, redirects, and pages with a deliberate noindex instruction.
How often should I check technical SEO?
Check after launches, migrations, hosting changes, and major template updates. A lightweight monthly review is also useful for an active website.