How to set up sitemap.xml and robots.txt for a new website
Without a correct sitemap.xml and robots.txt, a new site gets indexed more slowly and risks having pages you meant to hide end up in the index anyway.
What sitemap.xml is and why it matters
A sitemap is an XML list of a site's URLs that tells search engines which pages exist and when they last changed. Without one, Google discovers pages through links, which is slower — especially for a new site with few backlinks.
On Next.js sites, the sitemap is generated programmatically through a sitemap.ts file at the root of the app directory — it should update automatically as pages are added, not be edited by hand each time.
What to include in the sitemap
Include only pages meant to be indexed and genuinely reachable without a 404: the homepage, service sections, product pages, blog posts. Leave out filtered URLs with parameters, thank-you pages after a form, and internal utility pages.
Set lastmod to the actual modification date, not the current date on every regeneration — artificially bumping the date doesn't speed up reindexing and can read as manipulation.
Configuring robots.txt
The robots.txt file lives at the domain root (site.uz/robots.txt) and specifies which sections are off-limits to crawlers: admin panels, on-site search pages, sort-parameter duplicates, staging subdomains. Always include a line pointing to the sitemap: Sitemap: https://site.uz/sitemap.xml.
Check that the entire site isn't accidentally blocked with Disallow: / — a common mistake after moving from a staging domain to production, when the staging block never gets removed.
Submitting to Google Search Console
After launch, verify site ownership in Search Console (via a verification file, DNS record, or Google Tag Manager) and submit the sitemap under Sitemaps, entering the full path: sitemap.xml.
Check the processing status after 1-2 days: if Google reports errors reading the file, the usual culprits are malformed XML or sitemap URLs that return a 404 or redirect.
Verifying the setup
Run robots.txt through the tester in Google Search Console (the robots.txt Tester tool) — it shows whether a specific URL is blocked from crawling before you find out the hard way, when a page falls out of the index.
Every quarter, compare the URL count in the sitemap to the number of pages actually indexed in Search Console's Coverage report — a wide gap signals indexing issues worth investigating separately.