How to Write a robots.txt File (With Examples for WordPress, Shopify & Next.js)

A robots.txt file is a plain-text file at the root of your domain that tells search engine crawlers which parts of your site they may request. It is one of the smallest files on any website and one of the easiest to get wrong. A single misplaced slash can stop Google from crawling every page you own, and the mistake can sit unnoticed for weeks because nothing on the site itself breaks.

This guide explains what each line does, gives you copy-paste files for the three most common platforms, and shows how to check the result with our free Robots.txt File Generator.

What robots.txt does (and what it doesn’t)

The file works on the honour system. Well-behaved crawlers such as Googlebot and Bingbot read it before requesting any other URL and respect the rules they find. It controls crawling, not indexing: a page blocked in robots.txt can still appear in search results if other sites link to it, just without a description. If you need a page kept out of the index entirely, use a noindex meta tag or an HTTP header and let crawlers reach the page so they can see that instruction.

Advertisement

It also does nothing for security. Anyone can open yourdomain.com/robots.txt and read it, so listing /secret-admin/ tells the world exactly where to look.

The five directives you actually need

Directive	What it does	Example
User-agent	Names the crawler the following rules apply to. * means all crawlers.	User-agent: Googlebot
Disallow	A path prefix the crawler should not request.	Disallow: /wp-admin/
Allow	Re-opens a path inside a disallowed folder. Google and Bing honour it; a few older bots ignore it.	Allow: /wp-admin/admin-ajax.php
Sitemap	Absolute URL of your XML sitemap. Can appear anywhere in the file, any number of times.	Sitemap: https://example.com/sitemap_index.xml
Crawl-delay	Seconds between requests. Bing and Yandex respect it; Google ignores it (use Search Console instead).	Crawl-delay: 5

Two wildcard characters are supported by the major engines: * matches any sequence of characters and $ anchors a match to the end of the URL. Disallow: /*.pdf$ blocks every URL ending in .pdf; without the $ it would also block /file.pdf?download=1.

Example 1: WordPress

WordPress serves a virtual robots.txt if no physical file exists, and plugins like Yoast or Rank Math add a Sitemap line automatically. A sensible starting point for most WordPress sites:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Disallow: /*?replytocom=

Sitemap: https://example.com/sitemap_index.xml

Notes on each choice:

• /wp-admin/ is blocked but admin-ajax.php is re-allowed because many themes and plugins load front-end content through it.
• Internal search result pages are blocked. They are infinite, thin and can waste crawl budget on larger sites.
• replytocom URLs are created by threaded comments and duplicate every post.
• Do not block /wp-content/ or /wp-includes/. Google needs your CSS and JavaScript to render pages; blocking them triggers “blocked resource” warnings and can hurt rankings.

Example 2: Shopify

Shopify generates a robots.txt for every store and, since 2021, lets you override it by creating a robots.txt.liquid template in your theme. The default file already blocks checkout, cart, account and internal search paths, so most stores only need to add or remove a few rules. A typical customisation that keeps the defaults and adds a block for filtered collection URLs looks like this inside robots.txt.liquid:

{% for group in robots.default_groups %}
  {{- group.user_agent }}
  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}
  {%- if group.user_agent.value == '*' -%}
    {{ 'Disallow: /collections/*+*' }}
    {{ 'Disallow: /collections/*?*sort_by*' }}
  {%- endif -%}
  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

Editing the Liquid file directly is safer than replacing it wholesale, because Shopify updates its default rules over time and a hand-written static file will not receive those changes.

Example 3: Next.js

With the App Router you can create app/robots.ts and Next.js will serve it at /robots.txt at build time:

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: ['/api/', '/dashboard/', '/preview'] },
    ],
    sitemap: 'https://example.com/sitemap.xml',
  }
}

For the Pages Router, drop a static robots.txt into the public/ folder. Either way, remember that a staging deployment on Vercel will serve the same file, so if you want staging kept out of Google, add an environment check that returns disallow: '/' when process.env.VERCEL_ENV !== 'production'.

Five mistakes that cost real traffic

1. Disallow: / left over from staging. This is the classic. The site launches, the file was never changed, and Google drops every page. Check the live file the day you launch.
2. Blocking CSS and JS folders. Google renders pages like a browser. If it cannot load your stylesheet it may judge the page as mobile-unfriendly.
3. Using robots.txt to hide a page you want de-indexed. Blocking prevents Google from seeing your noindex tag, so the URL can stay in results indefinitely.
4. Case and trailing-slash errors. Paths are case-sensitive. Disallow: /Blog/ does nothing for /blog/. And Disallow: /blog blocks /blog-post-title as well, because it is a prefix match.
5. A relative sitemap URL. The Sitemap line must be absolute, including the protocol.

How to generate and test your file

If you would rather not write the syntax by hand, the Robots.txt File Generator builds the file from checkboxes: choose the crawlers, add the paths you want blocked, paste your sitemap URL and copy the output. Before uploading it, run these checks:

• Open yourdomain.com/robots.txt in a browser and confirm it returns the new content with a 200 status. A 404 is fine (crawl everything); a 5xx will make Google stop crawling entirely.
• Use the robots.txt report in Google Search Console (Settings → Crawling) to see the version Google last fetched and any parse errors.
• Confirm the sitemap URL in the file matches the one you have submitted. Our XML Sitemap Generator can build one if your platform doesn’t.
• Check that important pages still return normally with the HTTP Header Checker; a robots change should never alter status codes, and if it does, something else moved at the same time.

Frequently asked questions

Do I need a robots.txt file at all?

No. If the file is missing, crawlers assume everything is allowed. Most sites benefit from one only to declare the sitemap and block a few low-value paths.

How long does Google take to pick up changes?

Google caches robots.txt for up to 24 hours. You can request a recrawl from the robots.txt report in Search Console to speed it up.

Can I block AI crawlers?

Yes. Add separate groups for GPTBot, ClaudeBot, CCBot and others with Disallow: /. These companies have publicly committed to honouring robots.txt, though it remains a request rather than an enforcement mechanism.

Does robots.txt affect page speed or ranking directly?

No. It has no ranking weight. Its only effect on rankings is indirect, through what Google is or isn’t able to crawl and render.