robots.txt and sitemap.xml for .UZ sites: what to configure and what to avoid
Two text files in the root decide what the engine sees and where crawl budget goes. One wrong line can remove an entire site from the index.
What robots.txt actually does
robots.txt governs crawling, not indexing. A page blocked by Disallow can remain in the index without a snippet if external links point to it — Google documents this explicitly.
Removing a page from the index requires meta name="robots" content="noindex" in the HTML or an X-Robots-Tag: noindex header. The page must not be blocked in robots.txt, or the crawler cannot read the directive — a classic logical trap.
The practical implication: use robots.txt to save crawl budget on utility sections, never as a removal tool.
Google versus Yandex directive support
Google supports User-agent, Disallow, Allow, Sitemap and the * and $ wildcards. It dropped Crawl-delay support in 2019 — the line is ignored entirely and crawl rate is managed automatically.
Yandex supports Clean-param, a unique and genuinely useful directive. The line Clean-param: utm_source&utm_medium&sort /catalog/ tells the crawler those parameters do not change content, consolidating URL variants. Google simply skips the line without error.
The Host directive, once used to declare the primary mirror for Yandex, is deprecated. Mirrors are resolved through 301 redirects and rel=canonical.
A working template for a commercial .uz site
A sensible minimum: a User-agent: * block disallowing /admin/, /cart/, /checkout/, /search/, /*?sort= and /*?page=; Allow rules for static assets such as /assets/ and /*.css; a separate User-agent: Yandex block with the same rules plus Clean-param; and a final Sitemap line with the full absolute URL.
Never block /assets/, /static/, CSS or JS from Googlebot. Without them the renderer sees an unstyled page, the mobile-friendly check fails and layout assessment suffers. This is among the most frequent audit findings.
The file must return 200 with content type text/plain at https://domain.uz/robots.txt. A 404 does not block crawling, but Google treats a 5xx on robots.txt as a temporary site-wide crawl ban — prolonged server failures lead to deindexing.
sitemap.xml: what belongs in it
Include only canonical, 200-returning, indexable URLs. A noindexed or redirecting URL in the sitemap is not fatal, but it pollutes reports and obscures the diagnosis of index gaps.
lastmod must reflect the real date of the last meaningful change. If the CMS stamps the current date on every URL at each build, the field becomes worthless and stops being used. Google ignores changefreq and priority entirely.
Format limits: 50,000 URLs and 50 MB uncompressed per file. A 200,000-SKU shop needs multiple files plus a sitemap-index.xml pointing at them. Split by entity type — categories, products, articles — so indexation can be tracked per section.
Multilingual sites and the sitemap
For a site with Russian, Uzbek and English versions it is convenient to declare language alternates directly in the sitemap via xhtml:link rel="alternate" hreflang="...", avoiding a full tag set in every page head.
Every language version must reference all the others and itself. Asymmetric markup is the most common reason Google ignores hreflang wholesale and serves the wrong version.
Use correct language values: ru, uz, en, or ru-UZ and uz-UZ when splitting by country. Set x-default on the version served to users whose language is not covered, usually Russian or English.
Verification and common incidents
After editing robots.txt, check it in Google Search Console, which reports the last fetch date and parsing errors. Yandex.Webmaster offers an equivalent tool for testing whether a specific URL is crawlable.
Incident one: shipping a staging User-agent: * with Disallow: / to production. The site vanishes from results within days. Add a robots.txt content check to the release checklist and to uptime monitoring.
Incident two: sitemaps listing http URLs on an https site, or URLs from another domain. Google rejects such entries as out of scope. Re-verify absolute URLs after every protocol or domain change.