Crawler IP verifier

Crawler IPs everywhere

Server resources are finite. It makes sense to spend them on people and crawlers that bring something useful to your digital presence, rather than on unknown or disguised crawlers, some of which scrape your content. A client request calling itself Googlebot deserves a check before you treat it as Googlebot. The name is easy to copy; the source IP gives you something more to work with.

This tool compares IPv4 and IPv6 addresses with crawler ranges published by their operators, I tried to find as many as possible, hope didn't miss many. Supply the full user-agent as well, and it examines the declared identity alongside the published IP range. You can check one address, analyze an export containing thousands of records, inspect the original entries and investigate a few selected IPs through DNS.

TL;DR: a user-agent is a claim, a published IP range is evidence, and DNS is an optional second opinion. Load your IPs, see which claims hold up, and decide what to do about the rest.

This verifier is a companion to my robots.txt checker. The checker tells you what a crawler is allowed to fetch; this one helps you check who is actually asking. Before even deciding which rules apply to a user-agent, the natural first step is to look at what is requesting your pages. The two cover different ground on purpose: the checker handles well over a hundred declared crawler names, while the verifier can only check operators that publish evidence, meaning IP ranges or DNS hostnames.

The tool compares IPv4 and IPv6 addresses with crawler ranges published by their operators. I looked for as many as I could find; if I've missed one, send me a note. Some operators document only DNS verification, which is why Yandex appears in the DNS check but not in the range registry. Supply the full user-agent as well, and it examines the declared identity alongside the published range. You can check one address, analyze an export containing thousands of records, inspect the original entries and investigate a few selected IPs through DNS.

I put considerable weight on server logs, because almost everything passes through them. Log analysis even predates Google Analytics: Urchin, the product GA grew out of, processed server logs before GA launched in 2005 (see Urchin's history and its log-processing documentation). Analytics interfaces have changed a lot since, UA to GA4 being a debatable step, but checking what actually requested a page is as useful as ever.

For this investigation, logs are the starting point. They can record automated requests without depending on the client running a tracking script. An origin log shows requests that reached the origin; CDN logs can include traffic handled earlier. Logging settings and caches affect what you can see, so use the right observation point rather than assuming one file contains every interaction. The purpose here is practical: identify unsupported crawler claims, examine unwanted behaviour and make better decisions about what consumes your server's time. The verifier supplies evidence; your server or security service enforces any restrictions.

The analysis stays in your browser. Importing, interpreting user-agents, matching ranges, filtering and generating reports all happen locally. The page downloads the public crawler dataset; it doesn't upload your batch to perform those checks. Optional DNS investigation is a separate action, with an explicit acknowledgement before selected IPs leave your browser.

Check an address or analyze a log export

1. Provide your input. Paste one IP per line, or load a UTF-8 TXT, CSV, TSV, JSON or JSONL file. For addresses accompanied by user-agents or response codes, use a structured format such as CSV.
2. Check the mapping. File selection inspects the content locally. Open Settings to review the detected format, header handling and selected columns. With pasted input, Analyze Locally performs inspection first and continues automatically when the mapping is clear.
3. Analyze locally. The tool loads the available source dataset and compares your addresses with it.
4. Read the evidence. Review the Analysis summary and Traffic results.
5. Investigate or export. Narrow the results with filters, select the records you need, and download a report. For an additional DNS check, open Selected IPs & Exports and use the separate DNS action.

An IP-only check can establish a published infrastructure association. Adding the user-agent lets the tool examine whether a recognised crawler claim is supported by the applicable ranges. An optional HTTP-status column adds response context and filtering.

Load example and the CSV template provide starting points. Format detection uses the content, so a filename alone does not settle how a file is parsed. If necessary, choose the format explicitly. Cancel processing stops the active browser operation without generating an analysis report; Clear removes the active input and results. Editing input or mapping requires a new analysis.

ip,user_agent,http_status
66.249.66.1,"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",200

This is a formatting example, using an address shown in Google's verification documentation. It is not a captured request or a promise of what a future dataset will return.

Prepare your log data

Keep enough log history to investigate

Disabling all request logging, without an equivalent record elsewhere, leaves an avoidable gap in website maintenance. When a page starts returning errors, a security rule blocks a crawler or an unfamiliar client begins probing paths, you lose the record needed to reconstruct what happened. Turning logging on afterwards can't recover requests that were never recorded. To my surprise, I've seen quite a few sites keep less than a week of logs (if at all) because the files get too big...

For a first look at crawler activity, a rolling 90-day window is a useful starting point, not a universal minimum or a guarantee of enough history. It's close to what I've worked with, where log-file analysis typically meant around 50GB of filtered data over three months, used to spot technical and contextual issues before they surfaced in Search Console. It gives you room to compare periods before and after a change. Then adjust it to your website:

• Crawl frequency and review habits: rarely crawled sections, or issues discovered late, can need a longer window.
• Migrations and seasonal changes: keep the before-and-after evidence. Year-over-year comparisons may need long-lived summaries rather than a year of raw requests.
• Volume and storage: a busy site can keep recent records searchable and older ones in a compressed archive. Note any sampling, since it limits conclusions about missing requests and totals.

Collect the fields you need, check that logging keeps working, rotate and delete on a schedule instead of letting files grow forever, and avoid recording secrets such as authentication tokens. OWASP's logging guidance recommends matching collection to its purpose and defining retention; it doesn't prescribe a universal 90-day period for website access logs.

Start with the client IP

A perfectly valid address can still be the wrong address to check. Behind a reverse proxy or CDN, the immediate network peer recorded by your origin (the server behind the CDN, or "edge") may belong to that intermediary. Comparing it with crawler ranges tells you about the intermediary, not necessarily the visitor.

For Cloudflare exports, look for the client-address field. At an origin behind Cloudflare, CF-Connecting-IP can carry the visitor's address, but its reliability depends on the request path and your trusted-proxy configuration. Worker subrequests have documented exceptions. With Pseudo IPv4 set to Overwrite Headers, the original IPv6 address is preserved in CF-Connecting-IPv6. Check Cloudflare's header documentation before deciding which field is authoritative.

A forwarded header is not trustworthy merely because it has a familiar name. Direct requests to an exposed origin can supply their own headers. The verifier asks for an acknowledgement when certain forwarded-IP fields are selected; that acknowledges your configuration review, it does not perform one. It accepts one address per field and does not choose an address from an X-Forwarded-For chain.

Keep the fields that explain the request

Choose a defined time window and check whether the export is complete, sampled or aggregated. Keep the original file separately if you transform it. A sampled security report and a complete request log can be useful for different purposes, but their row counts are not interchangeable.

Log exports in Cloudflare, nginx and Apache

Cloudflare: export request records containing ClientIP, ClientRequestUserAgent and EdgeResponseStatus where available; the importer recognises those names regardless of capitalisation. Retain the path and timestamp. For Logpush, select the relevant HTTP request fields and import the resulting JSONL after decompressing it. Dashboard exports can expose different fields, so inspect the actual file rather than assuming it contains a user-agent.

nginx: convert from your configured access-log format. Useful values include $remote_addr, $http_user_agent, $status, $request_uri and $time_iso8601; verify that the address reflects your intended client after any proxy handling. If you control logging, nginx's JSON escaping option produces structured records that import cleanly. Existing combined-format logs need to be parsed and exported to a supported format first.

Apache: the combined access-log format includes the user-agent; the common format doesn't. Check your configured LogFormat and export the client address, user-agent, final response status and useful request context through a parser that understands it. A missing user-agent can't be recovered by renaming columns.

Raw Apache/nginx combined-log text isn't accepted by this tool. Don't turn it into CSV by splitting lines on spaces: request lines and user-agents contain spaces, and quoted values need proper handling. CSV also needs correct escaping for commas, quotes and embedded line breaks.

Google Cloud (BigQuery): a useful starting point when your HTTP request logs are already there. Cloud Logging can route log entries into BigQuery tables, and an analytics-enabled log bucket can expose its data through a linked BigQuery dataset. Neither means every Google Cloud project automatically has a complete website access log; the request logging, routing and fields must exist.

Remove repeats without losing request context

You don't usually need to deduplicate first. Repeated records with the same normalised IP (the address written in one canonical form), exact user-agent and mapped HTTP status are checked together, and their original occurrences remain available. Ten identical combinations become one result group containing ten entries; a different status or user-agent produces another group.

If a file is too large, start with a shorter time window or fewer unrelated columns. To reduce it to distinct identity checks, deduplicate on IP + full user-agent + HTTP status, and keep a copy of the original: the reduced file loses repeated observations and may discard different paths or timestamps within the same combination. An exported count column is preserved as metadata; it doesn't restore the discarded occurrences to the verifier's totals. For an IP-only lookup list, deduplicating by address is fine; for a crawler-claim investigation, it can hide the fact that one address presented several identities.

Exclude private and owned addresses before import

If you only need external clients, filter the export before loading it, and keep an unchanged copy of the logs. The private IPv4 ranges are 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16. Don't exclude every address starting with 172 or 192, as that removes public addresses too; use an IP-aware filter that understands CIDR ranges. For IPv6 and other special-use categories, consult the IANA registry. Your own monitoring may use public IPs, so exclude only addresses you've confirmed. And don't remove your CDN's range as a shortcut for a wrongly selected origin field: first check which field identifies the client.

The verifier marks nonpublic and special-use inputs, but it doesn't keep a personal exclusion list or remember which IPs you reviewed in a previous batch. Filtering results changes what you see; filtering the source export reduces what you import and process.

Formats and limits

TXT accepts one address per nonblank line. CSV and TSV support column mapping. JSON accepts an object, an array of objects or an array of IP strings; JSONL/NDJSON accepts one complete record per line. Mapped JSON fields must be at the top level. Nested context can remain in the original record, but nested IP properties aren't automatically flattened. Mixing object and string records in one JSON collection isn't supported.

Oversized inputs are rejected rather than silently shortened. If the file can't be parsed, import stops and no usable analysis is produced. A correctly structured file can still contain individual invalid address records; these remain in the report as Invalid input. Supply individual addresses without URL schemes, port numbers, CIDR suffixes or IPv6 zone identifiers. See how addresses and CIDRs differ for the notation and matching rules.

What a user-agent tells you

The HTTP user-agent header describes the software making a request. Its standard syntax accommodates product names, versions and comments. It is supplied by the client, so writing a recognised name into it doesn't tell you who operates that client.

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

In user-agent terms, Googlebot/2.1 is a product (name and version), and the name alone, Googlebot, is the product token that robots.txt groups refer to. Google lists this shorter form alongside its other Googlebot user-agents. A different documented form isn't automatically suspicious, and a perfectly copied form isn't automatically authentic. Both a genuine crawler and an impersonator can send exactly the same text above.

A browser header contains several names

This older Chrome example from MDN's user-agent reference shows why reading just the first product name isn't enough. It illustrates the structure, not a current browser version.

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36

X11; Linux x86_64 describes the advertised platform. Chrome/51.0.2704.103 identifies the declared browser and version. Mozilla, AppleWebKit, KHTML, Gecko and Safari appear for compatibility; this isn't a list of browsers simultaneously visiting your page. Modern browsers can also reduce the detail exposed in their headers. Either way, an automated client can send the same string.

An AI crawler can look browser-like too

OpenAI has the following GPTBot example, noting that its version number can change:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

The familiar compatibility tokens don't make this an ordinary browser visit. GPTBot/1.4 is the declared crawler identity, and its documented purpose concerns content that may be used for model training, which differs from search and user-triggered fetching. OpenAI also notes that when its crawlers fetch robots.txt itself, they may add an extra marker to the string, which helps you tell those requests apart when your logs omit paths. The AI crawler section covers the distinctions. As with Googlebot, match the claim against provider evidence before attributing a request.

A tool can declare its own identity

My robots.txt fetcher for example, uses a more direct description:

RobotsTxtChecker/1.0 (+https://alexanderkyriakou.eu/ak/robots-txt-checker)

The name identifies the tool, the version identifies its release, and the URL explains what it does. It is useful information to find in a log. As with any user-agent, the declaration still belongs to the sender.

The verifier retains the raw header and interprets recognised crawler claims, product versions, advertised browsers, platforms and automation libraries. Browser-like means the text resembles browser software; automation-like can indicate a library such as curl or Python requests. Neither label says anything about intent. Missing, malformed and unfamiliar headers are distinguished, and multiple crawler claims stay ambiguous instead of being resolved by choosing a convenient one. A crawler name appearing only inside a supplied URL isn't treated as an identity claim.

How crawler IP verification works

The central check is address membership. Providers publish individual IP ranges, the tool retrieves and validates those lists, and your browser checks whether each imported address falls inside them. When a supported crawler name is supplied, the verifier considers the sources applicable to that identity.

Understand IP addresses and CIDRs

IPv4 addresses contain 32 bits, usually written as four decimal octets separated by dots. IPv6 addresses contain 128 bits, normally written as hexadecimal groups separated by colons. With IPv6, zero compression and letter case allow the same address to have several valid spellings. The verifier normalises valid addresses for comparison while keeping the original input in Details. It rejects IPv4 shorthand and leading-zero forms rather than guessing, since some software reads a leading zero as octal and 010.1.1.1 becomes ambiguous.

A CIDR prefix combines a network address with the number of leading bits that identify the range. In 192.0.2.0/24, the first 24 of IPv4's 32 bits are fixed, leaving 256 addresses from 192.0.2.0 to 192.0.2.255. In 2001:db8::/32, the first 32 of IPv6's 128 bits are fixed. Both examples are reserved for documentation, not crawler ranges. The notation is defined in the IPv4 CIDR and IPv6 addressing specifications.

You supply individual IP addresses; publishers supply the ranges. Matching compares the relevant address bits, not the first few characters of the text. The result shows the matched CIDR so you can see which published range provided the evidence.

IPv4 and IPv6 are separate address families. A source containing only IPv4 ranges says nothing about IPv6 coverage. An IPv4-mapped IPv6 address, such as ::ffff:192.0.2.10, is a particular case: the verifier retains its IPv6 identity and records when a range comparison uses the embedded IPv4 address. It doesn't silently merge that representation with a native IPv4 entry or treat the IPv4 comparison as general IPv6 coverage. This example also uses a documentation address.

Private, loopback and other special-purpose addresses get their own explanations from bundled IANA IPv4 and IPv6 classifications. Being listed in the registry doesn't automatically exclude an address, because eligibility depends on the specific assignment. An address absent from it is only a candidate for public matching, which says nothing about whether it's routable or who operates it.

What a match can tell you

Some feeds describe one crawler, while others describe shared provider infrastructure or a broader category. A match against a shared Google range can show a Google infrastructure association (the address belongs to Google's published ranges) without pinning down the exact Googlebot subtype. Where ranges overlap, the evidence keeps all relevant matches. A Google-hosted customer fetcher also needs to be distinguished from a Google-operated search crawler.

A current, applicable match supports the infrastructure association and, where the feed is specific enough, the supplied crawler claim. It doesn't mean every request was useful, harmless or permitted by your own policy. Similarly, an address outside the included ranges may belong to unsupported infrastructure. A supported claim with no matching evidence deserves investigation, though the public dataset isn't complete enough for the tool to call it impersonation automatically.

Identifying a crawler doesn't give it permission to access protected content. A publisher shouldn't bypass a paywall solely because a request claims to be Googlebot. An API should still require valid credentials and the appropriate permissions, even when the IP matches a known provider's published ranges. Use the verification results when configuring crawler access and firewall rules; keep subscriber and API access subject to their own checks.

Reading your results

Read the crawler claim, published provider and finding together. None established in the provider column means this analysis found no published-range provider for the address. That doesn't mean the address has no network operator: the tool doesn't perform WHOIS or ASN attribution.

Two labels are easy to mix up. Could not verify is about a claim or evidence the tool couldn't back up, while No published range match means the address wasn't inside any included range and there was no recognised claim to hold against it.

Investigate a crawler claim that cannot be verified

A finding is the start of the investigation, not a substitute for it. For a request claiming to be Googlebot, this is a useful sequence:

1. Confirm the client IP. Check the original log field and proxy setup. Otherwise, you may be investigating your own intermediary.
2. Review the range evidence. Open Details, check the applicable provider and source health, and distinguish a current nonmatch from unavailable or outdated data.
3. Check the provider's DNS evidence. Use the documented hostname and forward-confirmation checks. No PTR or a failed confirmation means this method didn't verify the claim; it doesn't, by itself, identify the operator.
4. Examine the requests. Review paths, methods, times, response codes and repetition. Look for unwanted endpoint probing, excessive requests or attempts to use access intended for a recognised crawler.
5. Decide on a proportionate restriction. If the investigation supports it, use your firewall, rate limits or application rules, then check the effect on legitimate traffic. Keep the relevant evidence with the decision.

The verifier reports a recognised crawler claim with no observed published provider match as Unverified crawler claim, rather than automatically calling it an impersonator. The evidence details distinguish a nonmatch from incomplete or unavailable source data. You don't have to ignore the surrounding behaviour, though, and you don't need a definitive name for the operator before restricting requests that violate your site's policy.

But Alex, it's classified as Unverified crawler claim, why not just block it?

Not on that finding alone! “Unverified” means the user-agent claims a recognized crawler, but the available published IP-range evidence doesn't establish its provider. That can reflect a nonmatching IP or incomplete or unavailable source data; it doesn't establish impersonation or abuse. The point is to know what you're blocking and why. A rule based on the user-agent alone can block legitimate crawlers while missing impersonators that change their name. Always review the requests first, block as narrowly as the evidence allows, and follow closely the effects afterwards.

Rows, groups and repeated entries

The summary distinguishes input rows, distinct addresses and result groups. Grouping uses normalised IP, exact user-agent and mapped response status. The Input entries column counts occurrences within that group, and selecting a group includes every one of those occurrences.

Top claimed crawlers counts supplied claims across imported rows, including repeats. A row containing several distinct crawler claims can contribute to several claim totals. These are counts of claims, not independently verified crawler visits, and an aggregated or sampled export can't tell you total request volume just because it contains numbers.

Inspect the original evidence

Open Details to see the raw user-agent and parsed components, applicable source health, matched CIDRs, publisher references and the assessment time. Original occurrences preserve additional imported fields, including requested paths and timestamps when supplied. Use Previous occurrence and Next occurrence to move between records grouped into the same result. Invalid records retain their original content and error explanation.

This is where a bland-looking summary can become useful: six similar requests might turn out to have targeted six different directories. The verifier makes that context available; the interpretation of their sequence and behaviour remains yours.

Search, filter and compare

Search matches addresses, user-agent text, user-agent type, recognised claims, observed providers, CIDRs, purposes, finding labels and response status. It doesn't search every arbitrary field retained inside valid original records; use Details for paths and other context, or filter those fields in your log source before import.

Filters work together. Start with Finding, Published provider or Crawler claim, then use HTTP status when mapped. Additional controls cover purpose, user-agent type, IPv4/IPv6, applicable source health and investigation views for unknown clients and invalid addresses. Use Could not verify or directly Unverified crawler claim and the crawler-claim filter to find unsupported claims worth reviewing. A purpose such as AI training describes a recognised claim or feed; it doesn't prove what that individual request was doing.

Sort by input order, numeric IP address, finding, repeated entries or user-agent type, in either direction. Show family, CIDR and evidence columns for a denser view. Pagination offers 25, 50 or 100 rows. View the total number of input entries per page and filtered results, as also the total. Select all filtered results applies across every matching page; selections stay selected when another filter hides them. Clear selection and Clear filters perform different jobs.

Use response codes as context

The status filter appears when a response field is mapped. Codes from 100 to 599 are retained; 0 means no response was recorded in that field. Missing values and unrecognised values are shown separately. A bad status value doesn't invalidate an otherwise usable IP.

For an SEO investigation, filter a crawler claim alongside 404, 403, 429 or 5xx responses, then check its infrastructure evidence and your firewall or server logs. A 200 doesn't authenticate the client, and a 403 doesn't prove impersonation. With Cloudflare records, EdgeResponseStatus describes the response sent to the client; OriginResponseStatus describes a different part of the request path and is retained as original context when supplied.

A 404 isn't automatically a problem to fix. Whether the URL should exist, who linked to it, and whether it was simply invented are different situations, and the requested path and its context tell you which one you're looking at.

Googlebot, country blocks and crawl access

A website can work perfectly for you in X (e.g. Greece) and still refuse the crawler trying to retrieve it. That's worth checking before spending hours on content or metadata that the crawler can't reach.

A local audience does not imply a local crawler IP

Google says its crawlers primarily use US IP addresses, although they also crawl from other countries. A firewall rule allowing only Greek IPs, for example, can therefore block legitimate Googlebot requests. The crawler's location doesn't have to match your customers' location.

It would be too absolute to say that blocking US traffic automatically makes the entire website invisible to Google. Google may try another country when it detects US blocking, and existing indexed URLs don't necessarily disappear immediately. The practical risk remains: a country or even a US state rule can prevent Google from fetching or refreshing important pages. You shouldn't rely on Google eventually finding an allowed route.

If you intentionally restrict content by country, account for that crawl limitation. Google's locale-adaptive crawling guidance says to treat Googlebot consistently with users from the same country. If the public content is meant to be accessible more widely, correct an accidental country block. For region-specific versions, use accessible locale URLs and the appropriate language annotations instead of relying only on IP-based redirection.

Check what the crawler actually received

1. Find requests claiming Googlebot and verify the client identity. A US address alone doesn't make it Googlebot.
2. Open Details for the requested URL, timestamp and response code. Compare the original record with your CDN or firewall event to identify the rule that acted on it.
3. Check representative pages and necessary resources after changing the rule. A successful request to the homepage doesn't show that article pages, scripts or images are accessible.
4. Use Search Console's URL Inspection or Crawl Stats, alongside the logs. A live inspection is one test; continue checking ordinary crawl traffic afterwards.

For ordinary content URLs, persistent 403 responses can affect indexing; 429 and 5xx responses can slow crawling. Google documents these effects in its HTTP status guidance. Also inspect the content when a response is 200: a challenge or error page can return a successful status without delivering the intended page. This verifier retains the response code, but doesn't fetch and inspect response bodies.

Common misconceptions about crawler logs and SEO

“Googlebot fetched it, so Google indexed it.”

A verified request shows that the crawler requested that resource, and even a 200 doesn't guarantee indexing. Use the indexed-URL information in Search Console to investigate indexing, rather than turning a successful log entry into an indexing verdict.

“More crawler requests always mean better SEO.”

Look at what was fetched. Repeated requests to duplicate parameters or unimportant URLs can consume resources without helping your useful content get discovered. Google's crawl-budget guidance focuses on crawl capacity, demand and useful URL inventory, particularly for large or frequently changing sites. Improving server capacity doesn't guarantee Google will request more pages.

“Every 404 needs a redirect.”

Fix broken links to content that should exist, restore a page that went missing by accident, and redirect a moved page to its actual replacement. A removed page without a suitable replacement, or a URL invented by a scraper, can correctly return 404. Redirecting every unknown path to the homepage doesn't restore the missing resource. Google warns that redirects to an irrelevant destination may be treated as soft 404s.

“One IP address means one visitor or one crawler.”

Shared addresses and proxies make that assumption unreliable; one crawler can also use many IPs. Keep the user-agent and request context instead of treating distinct IP counts as people counts. An IP-only export is useful for address checks, but loses distinctions that matter in a traffic investigation.

“If it is absent from my origin log, it never reached the website.”

A CDN can serve a cached response or block a request before contacting the origin. Check edge and origin records in context, including logging exclusions and sampling. A browser can also reuse a local cached resource without making a new request to your infrastructure. Logs describe the activity recorded at their observation point.

“A Google IP means Googlebot, and a non-US IP rules it out.”

Google publishes different crawler and fetcher categories; a broad provider association doesn't settle the exact client. Google also crawls from outside the US. Use the applicable range, declared identity and documented verification method, rather than country or company name alone.

Why crawlers request paths that do not exist

A request doesn't need a link on your website before it can reach a path. Anyone can construct a URL and send it. Automated clients can try familiar installation directories, administration interfaces, API routes, backup filenames or exposed configuration files. OWASP describes this kind of footprinting as enumeration of the application, including guessed file and directory names. A request to such a path can be reconnaissance without an exploit having succeeded.

There are also ordinary explanations. Another website may link to a mistyped URL. A crawler may remember a removed page, encounter a malformed generated link or test how the site handles an unknown address. Google has documented these reasons for requests to nonexistent URLs. “It's not linked anywhere on my site” is useful context, but it doesn't tell you how the requester found or constructed the URL.

For resources that don't exist, a real 404 provides a useful record. Filter those responses, open the original occurrences and compare the paths, source addresses, claims and times. Repeated requests to plausible CMS installations on a site that has never used that CMS deserve a closer look. Don't redirect every invented path to the homepage just to make the errors disappear; it hides the very signal you're trying to read.

For a shop or classifieds site, the same method can help distinguish catalog requests from unrelated endpoint probing. For an API, it can expose requests to guessed routes. This verifier doesn't detect credential attacks, correlate rotating proxies or assign behavioural risk scores. Those questions need the surrounding logs and the controls protecting the application.

A Googlebot claim that did not pass verification

One example from my own logs, analysed on October 24, 2026, involved 195.178.110.155, which sent six requests carrying the Googlebot header. All six received 404 responses; the paths included /cms/wp-json/ and /news/. The same address also sent requests under other user-agents, asking for paths like /wp-content/plugins/web-to-print-online-designer/readme.txt.

Evidence

Check	Observed result
Declared identity	Googlebot/2.1; one recognised, unambiguous crawler claim.
Published ranges	No applicable range matched. No published provider was established.
Applicable source	Google common-crawler source marked Fresh at assessment time.
Optional DNS check	No PTR record returned.
Imported responses	Six entries with HTTP status 404.
Crawler IP verifier's finding	Could not verify.

At first I read this as a Googlebot impostor. Then I noticed that the same IP also sent other user-agents and requested plugin readme files, which looks more like a generic scanner rotating identities than someone faking a single crawler. Either way, the Googlebot claim had no supporting range or DNS evidence, and the requested paths gave me solid reason to investigate.

The tool reports the narrower result, Could not verify or Unverified crawler claim, because it doesn't turn missing evidence into an automatic impersonation verdict. That leaves room for you to weigh the actual requests.

PS: The selected IP got blocked through Cloudflare's Security Rules.

Robots.txt, crawler identities and access controls

Robots.txt publishes instructions for crawlers that follow the Robots Exclusion Protocol. It has defined matching rules, but publishing a Disallow line doesn't make the web server refuse the request. Cooperation happens on the client side.

The User-agent: line selects the crawler group to which rules apply. It normally uses a product token such as Googlebot, rather than the full HTTP header with its version, compatibility text and documentation URL. My robots.txt checker lets you test those rules against URLs and different crawler identities before changing the live file.

A firewall rule that matches user-agent text is different: it can refuse matching requests. But a client can simply send another value and avoid the match. Robots.txt preferences, header-based rules and verified infrastructure checks therefore have different strengths. For private content, use authentication and authorisation; for unwanted request patterns, consider appropriately scoped rate limits and security rules. The same goes for paywalled articles: a condition that grants access whenever the header says Googlebot hands that access to anyone who copies the header. Any SEO who has had a crawl blocked has seen this from the other side, as the next section shows.

When your own crawl gets blocked

If you've run an SEO crawl with Screaming Frog or a similar tool, you've probably met this one: the crawl starts, then comes back full of 403 or 429 responses, or challenge pages instead of content. The site's firewall or bot protection is treating your crawler as unwanted automation, which, to be fair, is what it is.

The usual first workaround is switching the crawler's user-agent to Googlebot or Googlebot Smartphone, which Screaming Frog offers as a preset. Sometimes that works, and what it tells you is that the firewall trusts the name alone. That is the weak rule this page is about: anyone can do exactly what you just did. When it doesn't work, the site is probably also checking the source IP, or limiting your request rate. A Googlebot name arriving from your office or home connection fails a published-range check, and in this tool it would show up as Could not verify or Unverified crawler claim. It's also worth saying plainly that impersonating Googlebot to get past protection on a site you don't own isn't a neutral move, even when the intent is harmless.

The cleaner route is to ask the site owner to allow your crawl. In practice that means allowlisting your IP address or, if your IP changes, a custom HTTP header carrying a secret value that a firewall rule recognises (Screaming Frog can send custom headers). Cloudflare, for example, lets a rule skip its security checks for matching requests. Slow down as well: fewer threads and a lower request rate avoid 429 responses and unnecessary load. And check what the crawler actually received, because a challenge page can come back as a 200 without being the real page.

If you own the site, the same logic works in reverse: give agencies and tools an allowlist entry by IP or token, never by user-agent. Otherwise the rule that lets your SEO in lets everyone else in too.

Tokens that look right, and tokens that never show up

Copying a crawler token from documentation can bring surprises: Perplexity‑User with a non-breaking hyphen (U+2011) isn't Perplexity-User with the ASCII one, and my robots.txt checker flags that. There are also control tokens that don't name separate HTTP crawlers. Google-Extended and Applebot-Extended are examples, so don't expect every robots.txt control to appear as a request identity in your logs.

AI crawlers: deciding what to allow

“Block AI” leaves several questions unanswered. Do you object to training, automated search indexing, a user asking an assistant to retrieve your page, or the volume of requests? Those are different decisions, and providers don't necessarily use one identity for all of them. OpenAI documents GPTBot for potential training use, OAI-SearchBot for search and ChatGPT-User for certain user-initiated actions, and says robots.txt rules may not apply to the last. Anthropic likewise distinguishes ClaudeBot, Claude-SearchBot and Claude-User. Consult the policy for the particular client rather than carrying one company's behaviour over to another.

A publisher might permit search discovery while declining training collection. A product catalogue might benefit from discovery while needing tighter limits on costly endpoints. Your decision can weigh visibility, referrals, content value and server cost, without assuming that a provider match obliges you to permit access. The Purpose filter helps organise the documented categories; it can't reveal the intended downstream use of an individual request.

Robots.txt groups, Search Console's Search generative AI setting and Cloudflare's AI Crawl Control are separate layers, and only some of them enforce anything; my robots.txt checker guide goes through them. Whatever you configure, review the actual file served at /robots.txt, the security policies and the logs afterwards. If traffic can reach an origin without passing through the intended edge controls, those controls don't govern that route.

When the declared identity changes

In August 2025, Cloudflare reported Perplexity-associated requests that changed user-agents and source networks after blocking. That is Cloudflare's published investigation, not a conclusion this verifier can independently reach from a browser-looking header.

It illustrates why blocking a name alone can be insufficient. Once a client uses an ordinary browser header and an address outside the included published ranges, identifying the operator may require timing, request patterns and other evidence beyond this tool. The same limitation applies to competitors scraping prices through rotating proxies. A missing association shouldn't be quietly converted into either “human visitor” or a named company.

One direction worth watching is Web Bot Auth, where crawlers sign their requests with HTTP Message Signatures (RFC 9421), so a site can verify the operator cryptographically instead of leaning on user-agents, IP ranges or reverse DNS. It still rests on IETF drafts and adoption is early, though Cloudflare supports it in its verified bots program. This verifier doesn't check signatures.

Published IP sources and optional DNS checks

Published-range coverage

The included registry contains 21 feeds from eleven providers. The table describes the scope used by this version of the verifier, not every crawler those companies may operate. Live availability and source health are shown in Official source data.

Claims and feed categories cover web indexing, AI search, potential training collection, user-triggered fetching, SEO analysis, advertising-related crawling, site verification, link previews, open web data and other documented automation. One provider or identity can have more than one purpose. More matching feeds don't necessarily mean more independent confirmation.

Updates, source health and older requests

Source retrieval runs daily for all included providers. Each feed is validated before publication, and a failed update can leave the previous validated ranges available, clearly marked as noncurrent evidence. Source health is the tool's label for whether a feed is current, outdated or unavailable.

“Fresh” means the source data passed the tool's format, IP-range and update-consistency checks, remains within its configured freshness interval, and has no disqualifying refresh or timestamp error. It doesn't mean the publisher's list is exhaustive. Older validated ranges may remain available as evidence when a source is no longer current.

Reload published dataset checks the saved dataset served to the page. It doesn't contact every publisher or repair a failed publisher update. Analyze again after reloading. The most recent retrieval timestamp shown for the dataset doesn't mean every included feed was successfully refreshed at that time.

Request timestamps remain available in the imported records, but verification uses the dataset evaluated for that analysis. The tool doesn't maintain a historical ownership archive or reconstruct DNS at the time of an old request. Save the evidence with an investigation when the observation time matters.

Validated public datasets can be cached in your browser for reuse. If the service can't be reached, a retained bundle may be available as explicitly noncurrent fallback evidence. Imported traffic is never written to that dataset cache. Clearing the workflow removes the active input and report.

Investigate selected IPs with DNS

Select rows representing up to five distinct public IPs, open Selected IPs & Exports, acknowledge sharing and choose Check selected IPs. No DNS requests are triggered by ordinary local analysis.

For example, a hostname beneath googlebot.com can provide Google crawler evidence when it resolves back to the submitted address. googlebot.com.attacker.example can't pass that suffix check, and a forward-confirmed hostname under an unrelated domain doesn't verify Googlebot either. See Google's documented verification procedure.

DNS evidence only works where the operator documents which hostnames to expect, which is why only some providers appear above. Operators that publish just IP ranges, OpenAI for one, can't be confirmed this way.

The check can return a forward-confirmed hostname, no PTR record, or hostnames whose forward confirmation wasn't established. A resolver error is a separate failure. At most three distinct valid PTR hostnames are checked per IP; additional unchecked hostnames are reported. DNS can succeed without identifying a supported crawler provider.

Only the selected IPs are sent to the lookup Worker and Cloudflare's public DNS service. Your files, user-agents and full records stay local. Results may be shared from a short-lived server cache, bounded by DNS lifetimes and capped at one hour. Missing-PTR and unconfirmed results can be cached for up to one minute; resolver errors aren't cached. The displayed check time distinguishes a retained observation from a new lookup.

The Crawler IP Verifier allows five new IP checks per minute and 500 per day, shared across visitors. Unexpired cache hits don't consume a new lookup allowance. An accepted attempt can consume allowance even if it subsequently fails. Private/special-use addresses and IPv4-mapped IPv6 selections aren't eligible for this DNS action.

DNS evidence stays separate from the published-range finding, provider filters and analysis exports. It can strengthen an investigation without telling you anything about the intent of an individual request.

Check DNS from Windows or Linux

You can also check an IP directly from your computer. Microsoft's Bingbot verification guide explains how to do this with nslookup on Windows and host on Linux. The process has two steps: find the hostname associated with the IP, then check whether that hostname resolves back to the original IP.

Windows - Command Prompt or PowerShell

Replace IP_ADDRESS with the client IP from your logs:

nslookup IP_ADDRESS

If a hostname is returned, replace RETURNED_HOSTNAME with that hostname and run:

nslookup RETURNED_HOSTNAME

Read the answer section. The initial Server and Address lines identify your DNS resolver, not the crawler.

Linux - terminal

Use host, if installed, for the same two steps. First, look up the IP:

host IP_ADDRESS

Then look up the hostname returned by that command:

host RETURNED_HOSTNAME

For Bingbot, both checks must agree: the returned hostname must end in .search.msn.com, and its forward lookup must include the original IP. A trailing dot in DNS output is normal. A name such as search.msn.com.example.org does not qualify.

No PTR record, a lookup error or a different forward address leaves the IP unverified by this method. Compare that result with the published ranges and the original log evidence before drawing a conclusion. Other crawlers require their own documented hostname endings.

These commands query your computer's configured DNS resolver. They don't send the IP through this tool's lookup service or consume its shared lookup allowance.

For a Bingbot claim, you can also check one IP at a time with Microsoft's Verify Bingbot tool. Microsoft's instructions place it under Tools & Enhancements → Verify Bingbot in Bing Webmaster Tools, with a public version available at the link above. This is a separate check on Microsoft's site; the IP you submit is sent to Microsoft.

For larger DNS batches

An honourable mention to Goodbots by eywu, a command-line project for bulk reverse and forward DNS checks. It supports several crawler families besides Googlebot, and configurable concurrency. It's a useful option when the small DNS allowance here doesn't suit the size of your check.

Read its verification rules before relying on the output: its domain matching may not enforce each provider's exact domains, so review returned hostnames against the operator's documented ones. Also, this verifier's selected-IP TXT export contains comment lines, so prepare a plain one-IP-per-line file for Goodbots.

Export your findings

CSV provides one row per original occurrence, with relevant invalid records; JSON preserves fuller structured evidence and raw values. Choose the whole batch, all filtered results across pages, or selected results. The quick CSV/JSON buttons export the whole batch. TXT always uses explicitly selected eligible public IPs, deduplicated, with explanatory comments.

Exports are generated locally and can include private original data. CSV protects spreadsheet-sensitive cells with a prefix; JSON retains the original strings. Selecting a row records your choice, not a finding of abuse. Later DNS results aren't bundled into these reports. Exports exceeding 64 MiB stop without a partial download; select a smaller scope.

A few practical questions

Why does a Googlebot claim say “Could not verify”?

The name was recognised, but the available evidence didn't back up the claim. Follow the investigation steps: confirm the client IP, review source health, check supported DNS evidence and examine the requests.

Why does an official infrastructure match still need investigation?

A range association doesn't decide whether you want those requests. Check the feed scope, paths, responses and your access policy. The section on what a match can tell you explains the distinction.

Why are several requests grouped into one row?

They share the same normalised IP, exact user-agent and mapped HTTP status. Different paths and timestamps can exist within the same group; see how grouping and counts work.

Where can I see the requested paths?

In the original records inside Details, when your export included them. The verifier can't recover fields omitted from the export, and the main search doesn't search every preserved metadata field. See original evidence and which fields to export.

Why was my file rejected or mapped incorrectly?

Check the supported formats and limits, then review the header checkbox and column mapping. Raw combined access logs need conversion; nested JSON IP fields need flattening. In a headerless two-column file, check whether the second column was interpreted as a user-agent when it actually contains response codes.

Why am I seeing private or proxy IP addresses?

The selected field may describe an internal hop. First check the client-IP source and proxy configuration. If it is the intended field, you can exclude private and owned addresses before import.

Does browser-like mean a human visited?

No. It describes the supplied header, which automated software can copy. See the browser user-agent example; this tool doesn't perform human verification.

Why did DNS return no PTR record?

The lookup returned no usable reverse hostname, so that verification route couldn't establish a provider. Compare it with the range evidence and request context. The DNS section explains the required checks.

Why did DNS stop before checking every selected IP?

Checks run sequentially. A service limit, timeout or resolver error stops later submissions; completed results remain visible. Review the message and the shared lookup limits before retrying.

Why did the finding change after reanalysis?

The ranges, source health, input or selected fields may have changed. Compare source versions and assessment times. Daily updates and evidence freshness explain what a new analysis is comparing.

Can I turn selected IPs directly into a blocklist?

The TXT export is a review list. Complete the investigation before choosing a restriction, and check its effect on wanted traffic. The export section explains what the file contains.

Can country restrictions block Googlebot?

Yes. A rule aimed at visitors outside your market can also affect legitimate crawling. See Googlebot and country blocks.

Is three months of logs enough?

It's a useful starting window for a first look, but crawl frequency, migrations, seasonal patterns and your review schedule can justify a different period. See log history and retention. You only need to import the relevant subset into this tool.

Are my files uploaded?

No loaded file is uploaded for ordinary analysis. The browser downloads public datasets and processes your records locally. Only the optional DNS action shares selected IPs; files and user-agents stay local. Downloaded reports remain on your device.

Resources worth browsing around

Provider links in the coverage table lead to their documentation or published feed, and exact range sources, versions and retrieval times are available within the tool's source evidence.

• RFC 4180: CSV format and Google's exporting BigQuery query results, for preparing a clean import file.
• Cloudflare's verified bots documentation and Cloudflare Radar's bots directory, if your traffic already passes through Cloudflare and you want to compare its classification with the evidence here.
• Cloudflare AI Crawl Control, for enforcing policies on identified AI crawlers.
• Screaming Frog Log File Analyser, a desktop tool for larger log files and ongoing analysis.
• Tame the Bots by Dave Smart, for technical SEO explanations and experiments on how search engines crawl, render and index content.
• HTTP Archive's 2025 Web Almanac, SEO chapter, for measurements from real websites.
• OWASP: broken access control, on why access decisions shouldn't rest on a header.

What a total headache. The more I read how the fundamentals work on the matter, the better I undedratand that we're online by sweat & tears, and mere luck.

PS: If you know of a crawler range or DNS suffix that should belong here, send me a note.