Contents
In short. Check in order, from cheapest to costliest. First see whether the page is in the index, the search engine's database: use Google Search Console and Yandex Webmaster, not the site: operator. Then rule out your own blocks: robots.txt, noindex, the response code and canonical. If nothing blocks the page, read the status: “Discovered” means Google knows the address but has not visited it, “Crawled” means it visited and has not added the page. Sitemaps and IndexNow only announce addresses. A new site needs days to weeks; manual requests are quota-limited.
Is the page in the index? What the cabinets show and what site: misses
Indexing means the search engine stored the page in its database after its robot visited it; only pages from that database can appear in results. Do not test this with site:yourdomain: Google says the operator does not necessarily return all indexed URLs and, for a URL missing from it, points to URL Inspection.
Verify ownership in two free cabinets: Google Search Console and Yandex Webmaster (useful if Russian-speaking customers search in Yandex). Look at the status of each important address, not at whether the site exists.
- Google, Page indexing report: how many addresses are indexed and why the rest are not; the goal is canonical versions of important pages.
- Google, URL Inspection: an indexed URL may appear in results (not guaranteed); a URL not on Google will not appear; an unknown URL is one Google has not seen.
- Yandex, Indexing, Pages in search (translated from the Russian interface): which pages take part in search and which are excluded, and why. A page can be crawled and still be absent from search.
Did you block the site yourself? Five causes you can check without a contractor
Google names three conditions: the crawler is not blocked, the page returns HTTP 200 (success), and it has content that can be indexed. Meeting them does not guarantee indexing, but if one fails, waiting is pointless.
Scroll right to see all columns
| Cause | How to check | What the help pages say |
|---|---|---|
| robots.txt block (the crawler rules file) | Open yourdomain/robots.txt and look for Disallow | Controls crawling, not results: a blocked address can still appear without a description if linked. Yandex also allows such pages in search. If the file itself returns 5xx, Google stops crawling the site for 12 hours |
| noindex (do not show the page) | Look for meta name="robots" and the X-Robots-Tag header | Google and Yandex both drop the page from results. If robots.txt blocks the page, the robot never reads the directive |
| Server response code | URL Inspection in Google; the server response check in Yandex Webmaster | Must be 200. Google treats 4xx (except 429) as a missing page, and 5xx slows crawling. Yandex excludes 3XX, 4XX and 5XX answers |
| Canonical (main address among similar pages) | Look for link rel="canonical"; in URL Inspection compare declared and Google-selected | If Google selects another address as canonical, it will not show this page |
| Access and content | Open the page in incognito; find the text in the source | Google lists login pages, a blocked Googlebot, and pages made of images or video without text |
What “Discovered” and “Crawled - currently not indexed” mean
This is the editors' inference: the status alone cannot tell what holds the robot back
“Discovered - currently not indexed”: Google found the address but has not crawled it, so the report has no last crawl date. The help names a typical reason: the crawl was rescheduled to avoid overloading the site. It gives no actions for this status.
Google's crawl budget guide, on how much attention the robot gives a site, targets sites with 10,000+ pages changing daily or 1 million+ changing weekly (rough numbers). For a few dozen pages, overload fits poorly. This is the editors' inference: the status alone cannot tell what holds the robot back.
“Crawled - currently not indexed”: Google visited and did not add the page. It may be indexed later, and there is no need to resubmit it. The help names no reason.
Scroll right to see all columns
| Status | Meaning | What to do |
|---|---|---|
| Google: Discovered | Known, not crawled | Check that another page links to it (an a element with href) and that it is in the sitemap. Build external links: Google lists their absence as a possible reason a site is not found |
| Google: Crawled | Read, not added | Do not resubmit. Compare the page with those already found for its query: what does it add for a buyer (editors' suggestion) |
| Yandex: low-value or low-demand page | Automatic decision, can change, not a violation | Decide whether the page is needed; if so, add what similar pages lack and send it for recrawl: it may appear within two weeks after the robot learns of the change |
Example: 5 of 60 pages of this journal's new site were in Google's index on 3 November
This points to the stage to examine, not to an established cause
Journal observation, 3 November 2026. The site customersprofit.com, which you are reading, is new; its sitemap lists 60 addresses. In Search Console URL Inspection, 5 of 60 are in Google's index, 48 are “Discovered - currently not indexed” and 7 are unknown to Google. Googlebot, Google's robot, has visited 5 pages.
Ruled out: search bots are allowed, and robots.txt has Allow: / for Googlebot and YandexBot. On 3 November all 60 addresses returned HTTP 200 without noindex or X-Robots-Tag, and each is two clicks from the home page.
Google knows 53 of the 60 addresses and has visited 5. This points to the stage to examine, not to an established cause: the status does not say why the robot has not reached the other 48. A snapshot, not a forecast.
Do you need a sitemap and IndexNow, and what do they do?
A sitemap lists the addresses you consider important. Google calls it a hint with no crawling guarantee; it helps most a new site with few external links, and a well-linked site of about 500 pages or fewer needs it less. In Search Console “Success” means the sitemap was processed, not that pages are indexed.
IndexNow is an open protocol for announcing new, changed and deleted addresses: a key file on your site, a list of addresses in a request. Participants (Bing, Yandex, Seznam, Naver, Yep, Internet Archive, Amazonbot) share accepted addresses; Google is not among them. A 200 response means only that the address was received.
Scroll right to see all columns
| Method | Who is told | Limit |
|---|---|---|
| Sitemap | Google, Yandex | One file up to 50 MB or 50,000 addresses (Google); crawling not guaranteed |
| IndexNow | Bing, Yandex and other participants; not Google | Up to 10,000 addresses per request; daily limits not disclosed; 429 when exceeded |
| Request indexing (Google) | A daily limit exists, the value is not disclosed | |
| Recrawl pages (Yandex) | Yandex | A daily quota exists per the API description, the value is not disclosed |
How long should a new site wait, and how to send manual requests
Neither Google nor Yandex promises a deadline. Google's guides: about a week for a new site or page, a few days to a few weeks for crawling, a week or two for indexing after a request. Yandex: sites can, as a rule, appear within two weeks of being added to Webmaster; indexing after a recrawl request can take 3 to 7 days.
If the time is up, return to the blocks: Yandex's help names robots.txt, meta tags and server errors. Manual requests are for a few addresses: Request indexing in Google's URL Inspection (owner or full user rights), Recrawl pages in Yandex (a priority visit on the robot's next pass). For large lists Google recommends a sitemap.
- Fix the cause first: a block, a response code or a canonical. A request for a blocked page is useless.
- Pick 3-5 addresses that should bring customers: home page, key service, a couple of articles.
- Check each in URL Inspection (Google) and the page check (Yandex) and record the status and date.
- Send each address once: Request indexing in Google, Recrawl pages in Yandex.
- Come back in 7-14 days. Leave the rest to the sitemap and internal links.
What not to do: fake signals, mass requests and reindexing for its own sake
A request announces a page; it does not remove the cause.
- Do not repeat a request for one address. Google says repeats do not speed up crawling, and “Crawled - currently not indexed” needs no resubmission. IndexNow asks not to resend unchanged addresses and treats exceeding the limit (429) as potential spam.
- Do not use Google's Indexing API for ordinary pages: it is for job postings and livestreams, and abuse can cost you access.
- Do not buy inflated signals or links for quick indexing. Yandex treats such inflation as search spam and warns of significant ranking restrictions (2018 webmaster blog post, archived); Google says violating sites may rank lower or not appear at all.
- Do not believe a promise of guaranteed indexing: Google guarantees no crawling, indexing or serving, even for pages that meet its requirements.
A one-hour task for your team: a table of addresses with status, cause and action
The result of the hour is a table of 5-10 important addresses, each with a status in both engines and a next step. If pages are already indexed but customers do not come, see “Your site is indexed but search brings no customers: what to check”.
Disclosure: the journal's publisher is connected to WEBNOSTRA, a company that sells SEO and GEO services. Every check above runs in free cabinets without a contractor. If you hire one, ask for address statuses and dates in the report, not the number of submissions.
- Verify the site in Search Console and Yandex Webmaster and add the sitemap to both.
- Build a table: address, Google status, Yandex status, date checked.
- Check the blocks: robots.txt, noindex, response code, canonical, access without login.
- For every “Discovered” address, check that another page of the site links to it.
- For every “Crawled” address, write down how it differs from the pages already found for its query.
- Send one manual request each for the 3-5 most important addresses, then set a recheck in 7-14 days.
Key decisions
- check the index in the search cabinets, not with the site: operator, and rule out your own blocks first
- read “Discovered” as “the robot has not come yet” and “Crawled” as “it came and has not taken the page”: the actions differ
- spend manual requests on a few key addresses and never repeat them: quotas exist and repeats do not help
What to discuss with your team
- Key pages are checked in Search Console and Yandex Webmaster, not with the site: operator
- Robots.txt, noindex, the 200 response and canonical are checked for every important address
- The sitemap is in both cabinets, and IndexNow sends only changed addresses
- Manual requests are sent once each for 3-5 addresses and dated
- A recheck is set 7-14 days out, with a decision for “no change”
Terms in plain language
- Indexing
- Storing a page in the search engine's database after crawling; only stored pages can appear in results
- Crawling
- A search robot visiting an address and downloading the page; Googlebot is Google's robot
- Canonical page
- The address a search engine treats as the main one among similar pages
- noindex
- A directive in the page code or response header asking not to show the page in results
- IndexNow
- An open protocol for telling participating search engines about changed addresses
Sources
- Google Search Central: Google Search technical requirementsdevelopers.google.com · checked 03/11/2026
- Google Search Central: In-depth guide to how Google Search worksdevelopers.google.com · checked 03/11/2026
- Search Console Help: Page indexing reportsupport.google.com · checked 03/11/2026
- Search Console Help: URL Inspection toolsupport.google.com · checked 03/11/2026
- Search Console Help: Sitemaps reportsupport.google.com · checked 03/11/2026
- Google Search Central: Ask Google to recrawl your websitedevelopers.google.com · checked 03/11/2026
- Google Search Central: Block search indexing with noindexdevelopers.google.com · checked 03/11/2026
- Google Search Central: Robots meta tags specificationsdevelopers.google.com · checked 03/11/2026
- Google Search Central: Robots.txt introduction and guidedevelopers.google.com · checked 03/11/2026
- Google Search Central: How Google interprets the robots.txt specificationdevelopers.google.com · checked 03/11/2026
- Google Search Central: How HTTP status codes affect Google's crawlersdevelopers.google.com · checked 03/11/2026
- Google Search Central: How to specify a canonical with rel="canonical" and other methodsdevelopers.google.com · checked 03/11/2026
- Google Search Central: What is a sitemapdevelopers.google.com · checked 03/11/2026
- Google Search Central: Build and submit a sitemapdevelopers.google.com · checked 03/11/2026
- Google Search Central: Crawl budget managementdevelopers.google.com · checked 03/11/2026
- Google Search Central: SEO link best practices for Googledevelopers.google.com · checked 03/11/2026
- Google Search Central: How to get information on Googledevelopers.google.com · checked 03/11/2026
- Google Search Central: How to use the site search operatordevelopers.google.com · checked 03/11/2026
- Google Search Central: Indexing API quickstartdevelopers.google.com · checked 03/11/2026
- Google Search Central: Spam policies for Google web searchdevelopers.google.com · checked 03/11/2026
- Yandex Webmaster Help: How Yandex Search works (in Russian)yandex.ru · checked 03/11/2026
- Yandex Webmaster Help: How to tell Yandex about new and changed pages (in Russian)yandex.ru · checked 03/11/2026
- Yandex Webmaster Help: How to check that important pages take part in search (in Russian)yandex.ru · checked 03/11/2026
- Yandex Webmaster Help: Why pages are excluded from search (in Russian)yandex.ru · checked 03/11/2026
- Yandex Webmaster Help: Using the robots.txt file (in Russian)yandex.ru · checked 03/11/2026
- Yandex Webmaster Help: Meta tags (in Russian)yandex.ru · checked 03/11/2026
- Yandex Webmaster Help: Sitemap files (in Russian)yandex.ru · checked 03/11/2026
- Yandex Webmaster Help: IndexNow protocol support (in Russian)yandex.ru · checked 03/11/2026
- Yandex Webmaster API: Sending a page for recrawl (in Russian)yandex.ru · checked 03/11/2026
- Yandex Webmaster API: Checking the recrawl quota (in Russian)yandex.ru · checked 03/11/2026
- Yandex webmaster blog: Dubious promotion services and the risks of using them (2018, archived, in Russian)webmaster.yandex.ru · checked 03/11/2026
- IndexNow: protocol documentationindexnow.org · checked 03/11/2026
- IndexNow: FAQindexnow.org · checked 03/11/2026
- IndexNow: list of participating search enginesindexnow.org · checked 03/11/2026
- WordPress documentation: Settings Reading screenwordpress.org · checked 03/11/2026
- The journal's sitemap: 60 addresses (checked 3 November 2026)customersprofit.com · checked 03/11/2026
- The journal's robots.txt (checked 3 November 2026)customersprofit.com · checked 03/11/2026