Magento SEO: robots.txt, canonicals, layered navigation and speed in Magento 2

Technical Magento SEO comes down to five Admin screens: Search Engine Robots for robots.txt, Catalog > Search Engine Optimization for URLs and canonicals, XML Sitemap, Base URLs (Secure) for HTTPS and Full Page Cache for speed. This guide walks through each setting in Magento Open Source and Adobe Commerce 2.4.9, the current release.

To check it on your own site: Online shops

Magento (Magento Open Source, and Adobe Commerce as the paid edition) is a PHP ecommerce platform that controls almost everything a search engine sees: URLs, canonicals, robots.txt, the sitemap and product markup. It does not control the server: the redirect to https, compression and whatever headers your Nginx or Apache adds. This guide applies to self-hosted stores and to Adobe Commerce on Adobe's cloud infrastructure; Adobe Commerce as a Cloud Service uses a different storefront with its own documentation1. What every shop has in common is covered in the guide to ecommerce SEO.

Which Magento version are you on, and where does each setting live?

The current line is 2.4.9, released on 12 May 2026, with regular support until 31 May 2029; 2.4.8 is supported until 31 May 20282. Always run the latest security patch for your line. Menu names below are exactly as Adobe documents them.

How do you configure robots.txt in Magento?

Magento generates robots.txt from the Admin. If the request for /robots.txt reaches Magento rather than a physical file, its own controller answers with whatever you saved1. The steps:

1. Go to Content > Design > Configuration.
2. Pick the scope row (Global, or a website or store view if each needs its own rules) and click Edit.
3. Expand Search Engine Robots.
4. Keep Default Robots on INDEX, FOLLOW in production. A NOINDEX left over from staging will deindex the whole shop.
5. Type your rules into «Edit Custom instruction of robots.txt file», or click Reset to Default to restore Adobe's defaults, then save.

Adobe's default instructions include Disallow: /*?, which blocks every URL with parameters1. That keeps filters, sorting and internal search out of the crawl, which is what Google recommends for faceted navigation3. The side effect is that ?p=2, ?p=3 and so on are blocked too, while Google asks for every page in a paginated series to be crawlable with its own canonical4.

Here is the robots.txt we suggest, annotated. We checked it with Protego, a parser that applies Google's rules (wildcards, $, longest rule wins): product pages and ?p=2 are allowed; ?color=49, ?color=49&p=2, sorting, /checkout/ and search are blocked.

User-agent: *
Disallow: /index.php/
# Block filters, sorting and search...
Disallow: /*?
# ...but let pagination (?p=2) be crawled: the longer rule wins
Allow: /*?p=
Disallow: /checkout/
Disallow: /app/
Disallow: /lib/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=

# Model training: blocking these is an editorial choice
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Each crawler obeys only the most specific group that names it and ignores the rest5. If you add a User-agent: OAI-SearchBot group, it stops reading the * group, so you must repeat the cart and account rules there. In the example, OAI-SearchBot, Claude-SearchBot and PerplexityBot (the ones that decide whether ChatGPT, Claude or Perplexity can cite you) follow the general group. GPTBot, ClaudeBot and Google-Extended are about model training, and Google-Extended has no effect on Google Search67.

How does Magento prevent duplicate product and category URLs?

The options sit under Stores > Settings > Configuration > Catalog > Catalog > Search Engine Optimization, at store view scope8:

• Product URL Suffix and Category URL Suffix. Enter the suffix (html, htm or nothing) without the dot, and never inside the URL key, or you will get it twice9. Decide when you build the shop: changing it later changes every catalogue URL and means reviewing all your redirects.
• Use Categories Path for Product URLs. Off by default. Turn it on and a product opens with and without the category path; Adobe warns this creates several URLs for the same page8.
• Create Permanent Redirect for URLs if URL Key Changed. Leave it on Yes so that every URL key change creates a 301 from the old one.
• Use Canonical Link Meta Tag for Categories and for Products. Adobe recommends enabling both10.

The product canonical points to domain plus URL key (/driven-backpack.html) even when the page is opened from /gear/bags/driven-backpack.html, because product URL keys are unique10. In the current 2.4 code, the category canonical keeps the ?p= page parameter and drops filters11. That is why letting pagination be crawled makes sense: each page declares itself canonical.

For redirects Magento does not create on its own (merging two categories, old URLs from another platform), use Marketing > SEO & Search > URL Rewrites with a permanent (301) Redirect Type12. Under Stores > Settings > Configuration > General > Web, enable Use Web Server Rewrites to drop index.php from URLs13.

What should you do with layered navigation?

Layered navigation is the block of filters (size, colour, price) on Magento's anchor categories14. Each filter adds a parameter to the category URL, such as ?color=49. With the default robots.txt those URLs are not crawled, and if one is, its canonical points to the unfiltered category. That is what you want for most combinations.

If a combination has real search demand («red trail running shoes»), do not index the filter: create it as a subcategory with its own URL key, copy and products. It gets a clean URL, its own canonical and a place in the sitemap. If you use an extension that turns filters into «friendly» URLs, check it does not generate thousands of indexable combinations3.

How do you generate the XML sitemap in Magento?

The sitemap has two screens15:

1. Under Stores > Settings > Configuration > Catalog > XML Sitemap you choose what goes in (categories, products, CMS pages), whether images are added, and automatic generation: Generation Settings > Enabled, start time and frequency. From 2.4.9 you can pick Generation Method Batch, a batch mode built for large catalogues16. Under Search Engine Submission Settings, Enable Submission to Robots.txt adds the Sitemap: line to robots.txt.
2. Under Marketing > SEO & Search > Site Map, click Add Site Map, give the file a name and path (for example sitemap.xml in /), choose the store view and click Save & Generate.

Automatic generation relies on Magento's cron: without it, the sitemap goes stale. Do not fuss over frequency and priority: Google ignores <priority> and <changefreq>, uses <lastmod> when it is reliable and accepts up to 50,000 URLs or 50 MB per file17.

How do you speed Magento up: production mode, Varnish and JavaScript?

Magento has three operating modes (default, developer and production), and only production is optimised to serve the shop. Switching to it compiles dependencies and deploys static content18:

bin/magento deploy:mode:show
bin/magento deploy:mode:set production

Magento does not merge, bundle or minify CSS and JavaScript by default, and those options (Advanced > Developer) only appear in developer mode19. In production you switch them on from the command line, using the configuration paths Adobe documents, and then redeploy static content:

bin/magento config:set dev/css/minify_files 1
bin/magento config:set dev/js/minify_files 1
bin/magento config:set dev/template/minify_html 1

For caching, Adobe «strongly recommends» Varnish in production because the built-in full-page cache is much slower20. You select it under Advanced > System > Full Page Cache > Caching Application and export the VCL for your Varnish version from the same screen21. Varnish does not terminate TLS, so a proxy sits in front of it, usually Nginx; its setup is covered in the guides to Nginx HTTPS, HSTS and security headers and Nginx gzip, Brotli and caching.

Measure the result against Google's thresholds at the 75th percentile: LCP of 2.5 s or less, INP of 200 ms or less and CLS of 0.1 or less22.

What structured data does Magento ship with?

Adobe documents that the product template carries schema.org markup by default10. In the 2.4 code it is microdata inside the product page23. This table compares it with what Google asks for in merchant listings24:

The product page meets the requirements but lacks what matters most to a shopper. A theme other than Luma, or an extension, can change the markup: check yours in the Rich Results Test, and if an extension adds JSON-LD, make sure it does not sit alongside theme microdata with different values.

How do you force HTTPS in Magento?

Under Stores > Settings > Configuration > General > Web > Base URLs (Secure), set the Secure Base URL to https:// and enable Use Secure URLs on Storefront and Use Secure URLs in Admin13. The same section holds Enable HTTP Strict Transport Security (HSTS) and Upgrade Insecure Requests.

Be careful with the HSTS switch: in the 2.4 code it sends max-age=31536000, one year, with no way to shorten it25. The safe path is to start with a short max-age and raise it once everything works over https, because HSTS is hard to undo26. For that gradual start, set the header on the web server and leave the switch off. The permanent redirect from http to https is also the server's job.

How to check it

• robots.txt: open https://www.example.com/robots.txt and review the robots.txt report in Search Console.
• Canonicals: Search Console's URL Inspection shows the canonical you declare and the one Google picks, for a product and for a filtered category.
• Markup: run a product page through the Rich Results Test.
• HTTPS and headers: curl -I https://www.example.com/ and Mozilla's HTTP Observatory.
• Speed: PageSpeed Insights, with field data if your shop has enough traffic.

What SmoothSeen does with this

SmoothSeen analyses a product or category page and checks what this guide configures: robots.txt, sitemap, canonical and noindex, product structured data, speed with real-user Core Web Vitals (or Lighthouse when there is no field data), compression, HTTPS and security headers. On the AI side it looks at which crawlers your robots.txt allows, separating search from training, and whether your server returns a 403 to any of them.

What to do this week

Open your robots.txt and look for Disallow: /*?: if it is there, add Allow: /*?p= and check in Search Console that page 2 of a category can be crawled. Then make sure both catalogue canonical settings are on Yes. To see the rest of the list measured, analyse your online shop with SmoothSeen.

Frequently asked questions

Should I remove the .html suffix from Magento URLs?

Adobe treats it as a matter of preference: some people believe URLs without a suffix are indexed better, and some companies require one by internal standard. What Adobe does insist on is applying the same choice to products and categories. If your shop is already indexed, changing it means migrating every catalogue URL with 301 redirects, so it is only worth doing if you are rebuilding the structure anyway.

Should product URLs include the category path?

There is no need. With the path enabled, one product opens from as many URLs as it has categories, and you depend on Google honouring the canonical, which is a hint rather than an order. Without the path, each product has one stable address that does not change when you move it to another category.

Does Magento's robots.txt block ChatGPT or Perplexity?

The default rules do not name any AI crawler: they all follow the general group, which only blocks internal folders, the cart and URLs with parameters. Product and category pages stay open to OAI-SearchBot and PerplexityBot. If you add a dedicated group for one of them, remember it will stop reading the general group, so copy the blocking rules across.