Robots.txt is a crawl instruction
A robots.txt file tells compliant crawlers which paths they may request. It is useful for managing crawl activity, but it is not a reliable way to remove a URL from search results and it should not be used to hide private information.
Keep the file at the root of the host and check that it does not block pages, CSS, JavaScript, or images needed to understand public pages.
An XML sitemap is a discovery aid
A sitemap should contain the canonical, indexable URLs that matter to your site. It helps communicate your preferred URL set, but it does not guarantee that every URL will be indexed.
Build the file with the XML Sitemap Generator, then run it through the XML Sitemap Validator. Keep redirects, duplicates, thin utility URLs, and pages you deliberately do not want indexed out of the sitemap. Update it as your public content changes.
Check the whole crawl path
A crawler needs to reach the page, receive a successful response, understand the content, and discover useful links to other pages. Check server status, redirects, page size, source, headings, and internal links as one system.
When you find an issue, fix the cause and retest the final URL. A report is only valuable when it leads to a clear action.
Understand status codes and canonical signals together
A URL that redirects, returns an error, is blocked, or points to another canonical page is not equivalent to a healthy indexable URL. Review the final response and the page’s canonical choice together. A sitemap should reinforce the URLs you want considered, not list every historical or temporary address.
When a page changes location, use a deliberate redirect and update internal links. When several URLs contain the same content, choose the preferred version and make the alternatives consistent. The Server Status Checker and URL Redirect Checker help you inspect the observable path before you change templates or deployment rules.
Make important links and resources crawlable
Search systems need ordinary, discoverable links to move through a site. Use descriptive anchor text, keep important destinations reachable from pages that already receive attention, and do not hide the only path behind an interaction that requires unavailable client-side state.
The same principle applies to page resources. Do not accidentally block the CSS, JavaScript, images, or API responses needed to understand the visible page. Technical cleanliness is not a score exercise; it is the work of making the actual content accessible and interpretable.
Use a release checklist for technical changes
Migrations, framework upgrades, redirects, CDN changes, and robots edits deserve a before-and-after record. Capture the old and new URL, expected status, canonical, sitemap membership, internal links, and a representative mobile rendering. This makes a technical release reviewable instead of relying on memory.
After deployment, test the production domain rather than a local or staging copy. Run the XML Sitemap Validator, inspect a few important pages with the Meta Tags Analyzer, and review crawl or indexing reports when the site has enough traffic to produce meaningful evidence.
Diagnose causes instead of collecting warnings
A technical report can contain dozens of observations, but not every observation has the same impact. Start with the user-facing or crawl-blocking cause: a failed response before a missing refinement, a redirect chain before a minor header, and an inaccessible template before an isolated page detail.
Group related warnings into one fix and retest the final production URL. This prevents teams from applying contradictory changes to individual pages while the underlying template, routing rule, or deployment setting remains broken. Technical SEO is most valuable when it shortens the path from evidence to a verified fix.
Decide what belongs in the sitemap
A sitemap is not a database export. Include the URLs that represent your best, canonical, indexable public content and that you would be comfortable sending a visitor to. Exclude redirects, duplicates, thin temporary pages, private areas, and URLs that deliberately carry a noindex instruction.
Review the file when you publish, remove, or migrate content. A small site can maintain this manually; a larger site should generate it from the same source of truth that knows which pages are public. Validate the XML, then inspect a sample of the listed URLs so a syntactically valid file does not hide an editorial or routing mistake.
Frequently asked questions
Does robots.txt block indexing?
Not reliably. It controls crawling, while indexation is managed through page signals and search engine systems. Do not use robots.txt as a privacy control.
Should every URL be in the sitemap?
Include important canonical pages that are available for indexing. Exclude duplicates, redirects, and pages with a deliberate noindex instruction.
How often should I check technical SEO?
Check after launches, migrations, hosting changes, and major template updates. A lightweight monthly review is also useful for an active website.