---
title: "Before You Buy an AI Visibility Tool, Check Your Server Logs"
description: "A 60-minute tutorial to find verified ChatGPT, Claude and Perplexity requests in 30 days of logs, identify blocked pages and check your fix."
url: "https://groas.com/post/read-your-server-logs-like-a-search-term"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/04fe3c98-5796-476f-a32d-90eed633d089.png"
published: "2026-10-15T05:48:38.684Z"
modified: "2026-10-15T05:48:38.754Z"
---

October 15, 2026 · 10 min read

# Before You Buy an AI Visibility Tool, Check Your Server Logs

[Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

![A doorman at a data centre door shines a UV lamp on a visitor's hand-drawn "GPTBot" badge, exposing a fake bot trying to get counted.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/04fe3c98-5796-476f-a32d-90eed633d089.png)

In this article

1. [Step 1: Export 30 days of requests, not a dashboard summary](#step-1-export-30-days-of-requests-not-a-dashboard-summary)
2. [Step 2: Filter for AI bot names, then read the status](#step-2-filter-for-ai-bot-names-then-read-the-status)
3. [Step 3: Check IP ranges before you count a bot](#step-3-check-ip-ranges-before-you-count-a-bot)
4. [Step 4: Use one site’s numbers to choose what to inspect](#step-4-use-one-sites-numbers-to-choose-what-to-inspect)
5. [Step 5: Find the block at robots.txt, the CDN or the page](#step-5-find-the-block-at-robotstxt-the-cdn-or-the-page)
6. [Step 6: Fix the specific failure and redeploy](#step-6-fix-the-specific-failure-and-redeploy)
7. [Step 7: Verify the fix after seven full days](#step-7-verify-the-fix-after-seven-full-days)

On groas.com, **Claude-User fetched `/robots.txt` 1,218 times in 30 days** from verified IP ranges, according to our [30-day crawler teardown](https://groas.com/post/teardown-what-30-days-of-our-own-ai-craw). That is a better reason to open your server logs than another pitch about “technical issues hurting AI visibility.” Before you buy a tool, check whether ChatGPT, Claude and Perplexity can reach the pages you want them to read.

I ignored logs for years. I lived in the search terms report and treated access logs as hosting support’s problem. I was wrong. Logs show the request, the page, the response and the claimed User-Agent. They also show something a dashboard count can hide: anyone can claim to be an AI bot. Your job is to separate verified requests from costumes, then find the requests that failed.

**At the end of this hour, you will have** a list of verified AI requests by page and status, plus a short list of access problems to fix. You need 30 days of raw access logs, a terminal or text editor, and read access to `robots.txt` and your CDN firewall settings. If your CDN handles requests before they reach your server, you need its logs too. No paid tool. No new tag.

## Step 1: Export 30 days of requests, not a dashboard summary

Pull the line-by-line access log for the last 30 days. The menu depends on your stack:

- **cPanel:** `Metrics > Raw Access`
- **Plesk:** `Logs`
- **Your own server:** `/var/log/nginx/access.log` or `/var/log/apache2/access.log`
- **Cloudflare:** `Analytics & Logs`

[Requests blocked at the CDN do not appear in your origin log](https://siteseoaiaudit.com/ai-crawlers-server-logs/). Check both sources before declaring a bot absent. Export each domain separately, keep the originals untouched and work on copies.

**Expected result:** a `.log` or `.gz` export containing an IP address, timestamp, requested path, HTTP status and User-Agent for each request. Open the first 20 lines. If you cannot find User-Agent strings, you have the wrong report or a log format that needs changing before the next measurement period.

**Common mistake:** exporting seven days and treating a quiet week as proof that nobody visits. Requests can arrive in bursts. Use the full 30 days for the first pass; later, compare equal periods from the same log source.

## Step 2: Filter for AI bot names, then read the status

In the directory containing your copied origin logs, run a substring match. User-Agent strings can carry extra text, so an exact-string match is too brittle. Change `access.log*` if your files have different names.

```bash
zgrep -iE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User' access.log* > ai-hits.log
```

Open `ai-hits.log`. Read the **bot name, requested path and status together**. A `200` says the server returned a successful response; it does not prove the page contained useful text. A `403` or `429` points to a denial or rate limit. A `301` calls for another look at the redirect destination, not an automatic verdict that the page is blocked. [Status and URL together tell you more than a bot count](https://siteseoaiaudit.com/ai-crawlers-server-logs/).

Keep three working categories in mind as you scan:

- **Training crawlers:** `GPTBot` and `ClaudeBot`
- **Search crawlers:** `OAI-SearchBot` and `Claude-SearchBot`
- **User-triggered fetchers:** `ChatGPT-User`, `Claude-User` and `Perplexity-User`

`PerplexityBot` is also worth tracking, but do not label every one of its requests a user-triggered fetch. The categories help you distinguish model training, search discovery and a page requested around an answer. They are not proof that a particular visit produced a citation. [The agents have different jobs](https://www.hostarmada.com/blog/can-ai-read-your-page/), and the logs show requests, not the finished answer.

Make a quick tally by bot, path and status. If your file uses the common combined access-log layout, where the path is field 7 and the status is field 9, start with:

```bash
awk '{print $9, $7}' ai-hits.log | sort | uniq -c | sort -rn | head -30
```

Check those field positions against a few original lines first. CDN exports can use different layouts; do not run this command on a different format and trust its output. Also, do not hunt for `Google-Extended` in the access log. [It is a `robots.txt` control, not a separate HTTP User-Agent](https://devtoollab.com/blog/how-to-find-ai-crawlers-in-logs/).

**Expected result:** a manageable list of claimed AI requests and the pages they asked for. **Common mistake:** calling this list “verified AI traffic.” It is only a list of names supplied in request headers. Verify the IPs next.

## Step 3: Check IP ranges before you count a bot

Anyone can send `User-Agent: GPTBot`. I filter first and verify second; otherwise, a scraper in a bot costume can make an account look busier than it is. One [demonstration found three spoofed hits among 15](https://devtoollab.com/blog/how-to-find-ai-crawlers-in-logs). That example is a warning about the method, not an estimate for your site.

For a log with the client IP in its first field, extract the addresses:

```bash
awk '{print $1}' ai-hits.log | sort -u > suspect-ips.txt
```

**Check the first field before using the file.** A CDN or proxy can change which address your origin records. If that field is the proxy rather than the requester, use the appropriate client-IP field from your export instead.

Now take one claimed bot at a time. Get its published IP ranges from the matching vendor list, including [OpenAI’s bot-specific lists](https://devtoollab.com/blog/how-to-find-ai-crawlers-in-logs/) or [Anthropic’s published ranges](https://devtoollab.com/blog/how-to-find-ai-crawlers-in-logs/). Put the relevant CIDR prefixes in `vendor-prefixes.txt`, one per line. Do not mix vendors’ ranges and then treat any match as proof of the User-Agent’s identity. Run this check:

```bash
python3 - <<'PY'
import ipaddress
from pathlib import Path

ranges = [
    ipaddress.ip_network(line.strip(), strict=False)
    for line in Path('vendor-prefixes.txt').read_text().splitlines()
    if line.strip() and not line.lstrip().startswith('#')
]

for line in Path('suspect-ips.txt').read_text().splitlines():
    try:
        address = ipaddress.ip_address(line.strip())
    except ValueError:
        continue
    result = 'in range' if any(address in network for network in ranges) else 'unverified'
    print(f'{address}\t{result}')
PY
```

Match each `in range` address back to the lines in `ai-hits.log` that claim the corresponding bot. Repeat with the next bot’s published ranges. A reverse lookup can be a useful spot-check, but a hostname alone is not your count. Keep the unmatched lines as **unverified**, not as verified bot activity.

**Expected result:** a defensible set of requests to tally by bot, path and status. **Common mistake:** checking an IP against _some_ AI vendor’s ranges while reporting it as the bot named in its User-Agent. Match the vendor and the claimed bot before it enters your verified column.

![Access-log lines being separated into verified AI requests and unverified User-Agent claims](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/c4f48f3f-113c-4554-8a0c-13bd8f4031c1.png)

## Step 4: Use one site’s numbers to choose what to inspect

Here is why the IP check matters. In our 30-day groas.com teardown, `/robots.txt` had 13,319 raw hits carrying AI bot tokens. The reported verified count was 3,806. Within that page’s activity, Claude-User alone accounted for 1,218 verified fetches. Those figures describe requests to **one file on one site**, not an industry-wide spoofing rate or a count of citations.

| Page in the 30-day teardown | Reported verified hits | Reported user-triggered fetches | What I check next                                          |
| --------------------------- | ---------------------: | ------------------------------: | ---------------------------------------------------------- |
| `/robots.txt`               |                  3,806 |               1,218 Claude-User | Whether permission checks lead to successful page requests |
| Homepage `/`                |                   900+ |                802 ChatGPT-User | What text the homepage returns                             |
| Category and service pages  |               Hundreds |    0 reported live-answer pulls | Whether those pages are reachable and readable             |

The homepage count is a useful lead, not evidence that ChatGPT cited the homepage 802 times. Likewise, zero reported user-triggered pulls on category and service pages does **not** prove the content is poor. It might mean no matching request arose in that period. The data runs out at the request log.

**The decision:** inspect access and returned HTML before rewriting a page. If the page cannot be fetched, copy changes will not fix delivery. If it can be fetched and requests still do not appear, you have learned something narrower: access is not the blocker you found in these logs.

## Step 5: Find the block at `robots.txt`, the CDN or the page

Open `https://yoursite.com/robots.txt` beside your CDN firewall settings. Look for `User-agent: *` with `Disallow: /`, or disallow rules covering a key `/shop/`, `/blog/` or pricing path. Read the bot-specific groups too; do not infer the rule for every agent from one line. If access for a search crawler is blocked, that can affect its ability to discover pages for [ChatGPT search](https://geolikeapro.com/blog/openai-crawlers-comparison).

In Cloudflare, check `Security > Bots` for Bot Fight Mode or Block AI Bots. If a rule appears to catch the requests you care about, test a narrowly chosen path rather than switching off protection everywhere. The log tells you whether the response came from the CDN or made it to origin; an origin log alone cannot diagnose a request it never received. **`robots.txt` and the firewall are separate checks.**

Return to the non-`200` lines in your verified set. Look for `403`s, `429`s and redirects on pages you want assistants to read. Trace a redirect to its final response; a `301` is not a failure by itself. If you need a fuller pass on the access controls, use the [AI crawler swipe file](https://groas.com/post/the-ai-crawler-swipe-file-copy-paste-rob) after you have named the blocker.

Then inspect what your server actually returns for a key page. This command follows redirects, prints the final status, saves response headers and writes the HTML to `page.html`:

```bash
curl -sS -L -A 'ChatGPT-User' -D headers.txt -o page.html -w '%{http_code}\n' https://yoursite.com/key-page/
head -c 2000 page.html
```

Look for the page’s headline and a plain-language answer in the returned HTML, not just a JavaScript shell or challenge page. A request made with a bot User-Agent is **not** proof that the real bot’s IP can get through; confirm that with your verified logs. [Major agents examined in this context read returned HTML rather than executing page JavaScript](https://www.hostarmada.com/blog/can-ai-read-your-page/). The [curl and CDN audit](https://groas.com/post/can-chatgpt-actually-read-your-site-a-45) goes deeper on an empty response.

**Expected result:** you can name the failing layer: permission rule, firewall response, rate limit, redirect destination or missing HTML content. **Common mistake:** editing copy while the fetcher still receives a challenge page. Fix delivery first.

![Laptop showing a robots.txt file beside CDN bot settings](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/deb6ffdc-fa05-4e08-a4a5-7c2c9f44ae1b.png)

## Step 6: Fix the specific failure and redeploy

If `robots.txt` blocks the agents you want to admit, edit the relevant groups. If your choice is to block the listed training bots while allowing the listed search and user-requested agents, start with this example and adjust it to your policy and paths:

```txt
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /
```

Do not paste that file without checking the rest of your existing rules. In particular, [OpenAI says `robots.txt` rules may not apply to a user-triggered ChatGPT-User fetch](https://www.searchenginejournal.com/openai-says-robots-txt-may-not-apply-to-chatgpts-fetch-bot/585864/). Your verified responses remain the better test of whether requests get through.

If the failure sits at the CDN, change the rule for the affected path and agent instead. If delivery works but `page.html` is an empty shell, put a **40- to 60-word answer block in returned HTML** near the top of the key template. State what the offering costs, who it fits and how long it takes where those answers belong on the page. Do not leave that text behind a client-only fetch. Add key pages to `sitemap.xml` and submit it in Search Console if they are missing.

**Expected result:** your test URL returns a successful response containing the answer text, and subsequent verified requests no longer show the access failure you fixed. **Common mistake:** changing `robots.txt` while a firewall rule still drops the request first. Redeploy, then check both layers.

## Step 7: Verify the fix after seven full days

Export the next seven days from the **same sources** and run the same filter, IP-range check and status-by-path tally. Rename the output so you do not overwrite the first pass:

```bash
zgrep -iE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User' access.log* > ai-hits-week2.log
awk '{print $9, $7}' ai-hits-week2.log | sort | uniq -c | sort -rn | head -30
```

As before, the `awk` command assumes the common combined-log field positions. Verify IPs before comparing bot counts, and compare equal-length periods if you want a count to mean anything. A delivery fix is visible when a previously denied, verified request to a key path receives a successful response. A new user-triggered fetch to that page is useful evidence of activity, but you cannot schedule one: it depends on someone making a relevant request.

**Expected result:** you can show which blocked paths changed status and whether the returned HTML contains the answer. **Common mistake:** checking after 24 hours, seeing no new user-triggered fetch and calling the fix a failure. Give the requests time to arrive. If statuses improve but those fetches stay at zero, report exactly that; the logs do not tell you whether demand, discovery or the page’s content explains it.

Before you close the laptop, check three things in the second export and your saved HTML: verified successful responses on the homepage and a key money page, no remaining `403`s for the agents and paths you meant to admit, and answer text present in the returned page. If a money page also gets its first verified user-triggered fetch, record it. Do not manufacture one as a success criterion.

I used to tell clients to buy the audit tool first. I was wrong. **Read the logs, fix the fetch, then change the page:** if access is clean and the page still gives a thin answer, make that answer worth quoting. Only then decide whether the ongoing work is yours to run or worth handing to groas and its named strategist.

## Frequently Asked Questions

### Should I buy an AI visibility tool before checking whether AI crawlers can reach my site?

No. The recommendation is to read your server logs first: export 30 days of raw access logs, find verified ChatGPT, Claude and Perplexity requests, and identify any blocked pages. Only after access is clean and the page still gives a thin answer should you decide whether ongoing work needs a tool or an outside partner.

### Can I rely only on my origin server's access log to see whether AI bots visit my site?

No. Requests blocked at the CDN never reach your server, so they will not appear in the origin log. If your CDN handles requests before they reach your server, you need to check its logs as well before declaring a bot absent.

### How do I find ChatGPT, Claude and Perplexity crawler requests in my server logs?

Run a substring match on the log files for the bot names, for example with zgrep -iE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User' access.log\*. A substring match is used because User-Agent strings can carry extra text, so exact-string matching is too brittle.

### How can I verify that a request claiming to be GPTBot or ClaudeBot is really from that AI company?

Anyone can send a fake User-Agent header, so extract the client IPs from the filtered requests and check them against the vendor's published IP ranges, one claimed bot at a time, such as OpenAI's bot-specific lists or Anthropic's published CIDR prefixes. Do not mix vendors' ranges; keep unmatched lines as unverified rather than counting them as bot activity.

### What do HTTP status codes for AI bot requests tell me about whether my pages are accessible?

A 200 means the server returned a successful response, though not that the page contained useful text. A 403 or 429 points to a denial or rate limit. A 301 is not an automatic failure; trace the redirect to its final response before deciding the page is blocked.

### Where should I look when AI bots get blocked or receive an empty page from my site?

Check three layers separately: your robots.txt rules for User-agent groups that disallow key paths, your CDN firewall settings such as Cloudflare's Security > Bots rules, and the actual HTML your server returns for a key page. This lets you name the failing layer, whether it is a permission rule, firewall response, rate limit, redirect destination or missing HTML content.

### How long should I wait after fixing a blocked AI crawler request before checking if the fix worked?

Wait seven full days, then export the next seven days from the same log sources and run the same filter, IP-range check and status-by-path tally. A fix is visible when a previously denied, verified request to a key path receives a successful response. Checking after 24 hours and seeing no new user-triggered fetch is not proof the fix failed, because those fetches depend on someone making a relevant request.

### Can I find Google-Extended requests in my server access log?

No. Google-Extended is a robots.txt control, not a separate HTTP User-Agent, so it will not appear as a request in your access logs. To control it, use the relevant directives in your robots.txt file instead of searching the logs for it.

## Pay For Results, Not For Hours

Businesses buy the outcome, agencies resell it, and groas answers for it either way.

[See If You Qualify](https://groas.typeform.com/to/xC1bQNUT)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/read-your-server-logs-like-a-search-term#webpage","url":"https://groas.com/post/read-your-server-logs-like-a-search-term","name":"Before You Buy an AI Visibility Tool, Check Your Server Logs","description":"A 60-minute tutorial to find verified ChatGPT, Claude and Perplexity requests in 30 days of logs, identify blocked pages and check your fix.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/04fe3c98-5796-476f-a32d-90eed633d089.png"},"breadcrumb":{"@id":"https://groas.com/post/read-your-server-logs-like-a-search-term#breadcrumb"},"dateModified":"2026-10-15T05:48:38.754Z","mainEntity":{"@id":"https://groas.com/post/read-your-server-logs-like-a-search-term#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/read-your-server-logs-like-a-search-term#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"Before You Buy an AI Visibility Tool, Check Your Server Logs","item":"https://groas.com/post/read-your-server-logs-like-a-search-term"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/read-your-server-logs-like-a-search-term#article","url":"https://groas.com/post/read-your-server-logs-like-a-search-term","mainEntityOfPage":{"@id":"https://groas.com/post/read-your-server-logs-like-a-search-term#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"Before You Buy an AI Visibility Tool, Check Your Server Logs","description":"A 60-minute tutorial to find verified ChatGPT, Claude and Perplexity requests in 30 days of logs, identify blocked pages and check your fix.","datePublished":"2026-10-15T05:48:38.684Z","dateModified":"2026-10-15T05:48:38.754Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/04fe3c98-5796-476f-a32d-90eed633d089.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"logs, log, check, requests, page, bot, before, access","wordCount":2194,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"Ex Goldman Sachs and Ex Stanford Computer Science","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/read-your-server-logs-like-a-search-term#faq","mainEntity":[{"@type":"Question","name":"Should I buy an AI visibility tool before checking whether AI crawlers can reach my site?","acceptedAnswer":{"@type":"Answer","text":"No. The recommendation is to read your server logs first: export 30 days of raw access logs, find verified ChatGPT, Claude and Perplexity requests, and identify any blocked pages. Only after access is clean and the page still gives a thin answer should you decide whether ongoing work needs a tool or an outside partner."}},{"@type":"Question","name":"Can I rely only on my origin server's access log to see whether AI bots visit my site?","acceptedAnswer":{"@type":"Answer","text":"No. Requests blocked at the CDN never reach your server, so they will not appear in the origin log. If your CDN handles requests before they reach your server, you need to check its logs as well before declaring a bot absent."}},{"@type":"Question","name":"How do I find ChatGPT, Claude and Perplexity crawler requests in my server logs?","acceptedAnswer":{"@type":"Answer","text":"Run a substring match on the log files for the bot names, for example with zgrep -iE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User' access.log*. A substring match is used because User-Agent strings can carry extra text, so exact-string matching is too brittle."}},{"@type":"Question","name":"How can I verify that a request claiming to be GPTBot or ClaudeBot is really from that AI company?","acceptedAnswer":{"@type":"Answer","text":"Anyone can send a fake User-Agent header, so extract the client IPs from the filtered requests and check them against the vendor's published IP ranges, one claimed bot at a time, such as OpenAI's bot-specific lists or Anthropic's published CIDR prefixes. Do not mix vendors' ranges; keep unmatched lines as unverified rather than counting them as bot activity."}},{"@type":"Question","name":"What do HTTP status codes for AI bot requests tell me about whether my pages are accessible?","acceptedAnswer":{"@type":"Answer","text":"A 200 means the server returned a successful response, though not that the page contained useful text. A 403 or 429 points to a denial or rate limit. A 301 is not an automatic failure; trace the redirect to its final response before deciding the page is blocked."}},{"@type":"Question","name":"Where should I look when AI bots get blocked or receive an empty page from my site?","acceptedAnswer":{"@type":"Answer","text":"Check three layers separately: your robots.txt rules for User-agent groups that disallow key paths, your CDN firewall settings such as Cloudflare's Security > Bots rules, and the actual HTML your server returns for a key page. This lets you name the failing layer, whether it is a permission rule, firewall response, rate limit, redirect destination or missing HTML content."}},{"@type":"Question","name":"How long should I wait after fixing a blocked AI crawler request before checking if the fix worked?","acceptedAnswer":{"@type":"Answer","text":"Wait seven full days, then export the next seven days from the same log sources and run the same filter, IP-range check and status-by-path tally. A fix is visible when a previously denied, verified request to a key path receives a successful response. Checking after 24 hours and seeing no new user-triggered fetch is not proof the fix failed, because those fetches depend on someone making a relevant request."}},{"@type":"Question","name":"Can I find Google-Extended requests in my server access log?","acceptedAnswer":{"@type":"Answer","text":"No. Google-Extended is a robots.txt control, not a separate HTTP User-Agent, so it will not appear as a request in your access logs. To control it, use the relevant directives in your robots.txt file instead of searching the logs for it."}}]}]}
```