XML Sitemaps Explained: What to Include, What to Leave Out, and How to Read the Results

An XML sitemap is a list of the URLs you want search engines to know about, with optional hints about when each one last changed. It does not make pages rank and it does not force indexing. What it does is remove discovery as a bottleneck: on a site with thousands of pages, weak internal linking, or content that changes often, the sitemap is how Google learns a URL exists days or weeks before it would have stumbled on it. On a 20-page brochure site it barely matters. Knowing which of those you are is the first step.

What a sitemap looks like

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/tools/image-compressor/</loc>
    <lastmod>2026-09-10</lastmod>
  </url>
  <url>
    <loc>https://example.com/blog/compress-images/</loc>
    <lastmod>2026-09-14</lastmod>
  </url>
</urlset>

Two tags are worth using: <loc> (required) and <lastmod> (Google uses it if it is accurate). The other two you will see, <changefreq> and <priority>, are ignored by Google and Bing and can be left out.

Limits: 50,000 URLs and 50 MB uncompressed per file. Larger sites use a sitemap index, a file that lists other sitemaps, which is what WordPress SEO plugins generate (sitemap_index.xml pointing at post-sitemap.xml, page-sitemap.xml and so on).

Advertisement

What belongs in it, and what doesn’t

The sitemap should contain only URLs that are:

• Canonical (not a redirect, not a duplicate, not a parameter variant)
• Returning 200
• Indexable (no noindex, not blocked in robots.txt)
• Worth indexing

The last one is where most sitemaps go wrong. Auto-generated sitemaps routinely include tag archives with one post, author pages, paginated series, attachment pages, and thin category pages. Every one of those is a URL you are asking Google to spend crawl budget on and then judge. Sites with thousands of thin tag pages in the sitemap regularly show “Crawled – currently not indexed” in Search Console, which is Google declining the invitation. Exclude them at the source: in Yoast or Rank Math, turn off the sitemap for tags, media attachments and any taxonomy with mostly single-item archives.

Generating one

• WordPress: Yoast, Rank Math and All in One SEO generate and update it automatically. Core WordPress also ships a basic one at /wp-sitemap.xml; disable it if a plugin provides a better one, to avoid two sitemaps.
• Shopify, Squarespace, Wix, Webflow: generated automatically at /sitemap.xml; you can exclude pages in settings.
• Static sites, custom builds, small hand-coded sites: the XML Sitemap Generator crawls the site from the homepage, follows internal links, and outputs a valid file you upload to the root. Re-run it when you add pages.
• Next.js, Astro, Hugo: each has a sitemap plugin or a sitemap.ts convention that builds it at deploy time.

Submitting it

1. Upload to the site root: https://example.com/sitemap.xml (or the index file).
2. Add the line Sitemap: https://example.com/sitemap.xml to robots.txt. The Robots.txt Generator includes this field. Crawlers that read robots.txt will find the sitemap without further action.
3. Submit in Google Search Console → Sitemaps, and in Bing Webmaster Tools. This is a one-time action; they re-fetch on their own schedule.

Reading the Search Console report

After submission, the Sitemaps page shows “Success” with a discovered URL count, or an error. The useful report is Pages (Indexing) → filter by sitemap. Three patterns:

• Submitted and indexed — good.
• Crawled – currently not indexed — Google saw it and passed. Usually thin or duplicate content. Improve or remove from the sitemap.
• Discovered – currently not indexed — Google knows the URL but has not crawled it yet. On a new site this is normal for weeks; on an established site it suggests low crawl priority, often from too many low-value URLs competing.

Any “Excluded by noindex” or “Redirect” entries in a sitemap-filtered view are errors in what you submitted, not in Google’s judgment. Fix the sitemap.

Keeping it healthy

• lastmod must be honest. Google has said it ignores lastmod on sites where it is set to “now” on every crawl. Update it only when content changes.
• Remove URLs when you remove pages. A sitemap full of 404s tells Google the file is unreliable. The Broken Links Checker can be pointed at a sitemap to verify every URL still returns 200.
• Don’t include UTM or session parameters. Those are not canonical URLs.
• Separate sitemaps by type (pages, posts, products, images, video) on larger sites, so the Search Console filter tells you which type is under-indexed.
• Image and video sitemaps are worth adding for e-commerce and media sites; they surface assets that are loaded by JavaScript and might otherwise be missed.

Frequently asked questions

Does a sitemap help SEO?

It helps discovery, not ranking. Its value scales with site size and how well the site is internally linked. Every site should have one because it costs nothing, but a small site will not see a measurable change from it.

How often should I update the sitemap?

Whenever pages are added or removed. CMS plugins do this automatically; static sites need a regeneration in the deploy step.

Should I include noindex pages?

No. Including a page and telling Google not to index it are contradictory signals and generate errors in Search Console.

Google only indexed 40 of my 500 sitemap URLs. Why?

Most likely the other 460 are thin, duplicate, or low value in Google’s judgment, or the site is new and crawl is still ramping. Check the Pages report for the reason per URL.