Services / AI Visibility Monitoring

Do You Know Which AI Bots See Your Client's Site?

Google Analytics doesn't track AI bot visits. Server logs don't surface actionable data. Most agencies doing AI SEO are working blind. We built BotSight to fix that.

Based on 3 published research papers and 19,556 queries analyzed

Real-time monitoring across 15+ AI crawlers, with AI-attributed revenue tracking, crawl-to-conversion funnels, and exportable reports your team can deliver to clients.

The Problem

Why AI Bot Monitoring Matters

Before an AI platform can cite your client's content, it needs to have crawled it. ChatGPT, Claude, Perplexity, and Gemini each use different crawling strategies — some fetch pages in real time during conversations, others rely on pre-built search indices.

Google Analytics doesn't track AI bot visits. Server logs require technical expertise and don't surface actionable metrics. Most agencies in AI SEO are working blind.

Without crawl-level data, you cannot tell a client whether AI platforms even see their site. That gap becomes a liability the moment a competitor shows up with the numbers.

BotSight identifies every visit from 15 known AI crawlers, calculates a composite AI Visibility Score, integrates with GSC / Bing Webmaster Tools / GA4 ecommerce data, tracks AI-attributed revenue and crawl-to-conversion funnels, and generates styled PDF reports and Excel workbooks for client delivery.

How It Works

Deployment

We install a lightweight monitoring layer on your client's site — Vercel edge middleware, Cloudflare Worker, or custom server integration. It runs asynchronously with negligible latency and doesn't block page rendering.

Every incoming request is checked against our database of known AI user agents. Matches are logged with bot identity, page visited, and timestamp. Human traffic passes through untouched.

Detection

AI Bots We Track

ChatGPT

GPTBot, ChatGPT-User

SearchGPT

OAI-SearchBot

Claude

ClaudeBot, Claude-User, Claude-SearchBot

Perplexity

PerplexityBot

Google AI

Google-Extended, GoogleAgent, Gemini Deep Research

Meta AI

Meta-ExternalAgent

TikTok AI

ByteSpider

Amazon Q

AmazonBot

Apple AI

AppleBot

DuckDuckGo AI

DuckAssistBot

Indirect: Gemini (via Google index) Indirect: Copilot (via Bing index) Security threat detection

New AI crawlers are added as they emerge. Detection rules update across all deployments automatically.

Deep Dive

What We Track: A Detailed Breakdown of AI Crawlers

Not all AI bots behave the same way. Each crawler has a different purpose, crawl frequency, and content extraction method. Some fetch pages in real time during active conversations. Others build search indices ahead of time. Some do both. Understanding these differences is critical for interpreting your crawl data correctly.

The table below shows the 14 primary bots BotSight monitors, the platform or company behind each one, whether it operates as an indexing crawler or a real-time fetcher, and what role it plays in the AI content pipeline. This is not an exhaustive list. New crawlers appear regularly, and BotSight detection rules update across all deployments automatically as we identify them.

Bot Platform Type What It Does
GPTBot OpenAI Index crawler Builds OpenAI's training and retrieval index. Crawls broadly, prioritizes high-authority domains.
ChatGPT-User OpenAI Real-time fetcher Fetches pages during live ChatGPT conversations when browsing is enabled. High signal for active citation.
OAI-SearchBot OpenAI Search crawler Powers SearchGPT and search-grounded responses. Focuses on fresh, factual content.
ClaudeBot Anthropic Index crawler Builds Anthropic's web index for Claude's retrieval capabilities. Crawl frequency varies by domain authority.
Claude-Web Anthropic Real-time fetcher Fetches pages during live Claude conversations for web-grounded answers. Indicates active sourcing.
PerplexityBot Perplexity Hybrid Both indexes and fetches in real time. Perplexity relies heavily on live web retrieval, making this one of the most active crawlers.
Bytespider TikTok / ByteDance Index crawler Builds ByteDance's content index for AI features across TikTok and related products. Very high crawl volume.
Meta-ExternalAgent Meta AI Index crawler Feeds Meta's AI assistant across Facebook, Instagram, and WhatsApp. Growing crawl footprint.
Amazonbot Amazon Index crawler Powers Amazon Q and Alexa's AI-driven responses. Focuses on product, service, and factual content.
Applebot Apple Index crawler Supports Apple Intelligence, Siri, and Spotlight suggestions. Crawl patterns align with Apple's content quality criteria.
DuckAssistBot DuckDuckGo Real-time fetcher Powers DuckDuckGo's AI-assisted search results. Fetches content to generate AI summaries.
Google-Extended Google Index crawler Builds Google's AI training and Gemini retrieval index. Separate from Googlebot so publishers can control AI access independently.
GoogleOther Google Utility crawler General-purpose Google crawler used for research, testing, and supplementary indexing tasks.
SearchGPT OpenAI Search crawler Dedicated crawler for OpenAI's search product. Targets fresh, structured content for search-grounded AI responses.
Why GA4 misses 90%+ of this activity. Google Analytics relies on JavaScript execution to register a page view. AI bots do not execute JavaScript. They fetch the raw HTML, extract the content they need, and move on. From GA4's perspective, these visits simply never happened. Server logs can capture some of this traffic retroactively, but they require manual parsing and do not provide real-time, structured data.

The distinction between index crawlers and real-time fetchers matters for interpretation. An index crawler visit means the platform is building or updating its knowledge of your content. A real-time fetcher visit means someone is actively asking about topics your content covers, and the AI is pulling your page to answer them. Both are valuable signals, but they indicate different things about your AI visibility pipeline.

Architecture

How BotSight Works: Edge-Level Detection

BotSight operates as edge middleware deployed at the CDN level. This means it intercepts every incoming HTTP request before that request reaches the browser, the application server, or any JavaScript runtime. The detection happens at the network edge, in the same layer where Vercel Edge Functions or Cloudflare Workers run.

What happens on each request

When a request arrives, BotSight reads the User-Agent header and compares it against our maintained database of known AI bot signatures. If a match is found, the middleware logs the bot identity, the specific page requested, the timestamp, relevant HTTP headers (Accept, Accept-Language, referrer), and the IP range. Human traffic passes through completely untouched with negligible latency added. The entire check runs asynchronously and does not block page rendering.

Why edge-level matters

Because the detection happens before JavaScript loads, BotSight catches every bot that identifies itself via User-Agent, regardless of whether it executes client-side code. This is the fundamental advantage over analytics tools like GA4 that depend on JavaScript tag execution. Most AI bots never trigger the GA4 tag, making them completely invisible to traditional analytics.

How BotSight Compares to Other Approaches

BotSight (Edge Middleware)

Real-time detection at CDN level. No JavaScript dependency. Structured data with bot identity, page, timestamp, and headers. Automatic updates as new bots emerge. Purpose-built for AI bot tracking.

Server Log Analysis

Retroactive, not real-time. Requires manual parsing of raw log files. No structured bot identification without custom tooling. Can capture bot visits but surfacing actionable metrics requires significant engineering effort.

GA4 / Traditional Analytics

Depends on JavaScript execution. AI bots do not execute JavaScript. Misses 90%+ of AI bot activity entirely. Not designed for bot detection. Useful for human traffic but blind to the AI crawl layer.

BotSight is not a replacement for GA4 or server log analysis. It fills a specific gap: real-time, structured tracking of AI bot activity that other tools cannot see. The data it generates is designed to feed into AI visibility scoring, crawl-to-conversion funnels, and client-facing reports.

Strategic Application

From Crawl Data to Business Decisions

Raw crawl data is only valuable if you know how to interpret it. Here is how agencies use BotSight data to make concrete decisions about their clients' AI visibility strategies.

Identifying Which Pages Bots Prioritize

When an AI bot crawls certain pages repeatedly while ignoring others, it tells you what the platform considers important. Pages with high crawl frequency from multiple bots are the ones AI platforms are most likely to use as source material. This data reveals your client's strongest AI assets, often pages that do not correspond to their top Google rankings.

Agencies use this to prioritize optimization efforts. If a product page gets heavy bot traffic from ChatGPT-User and PerplexityBot, that page is actively being sourced for live conversations. Improving its content structure, adding schema markup, and ensuring freshness can directly increase citation probability.

Reading Crawl Frequency Trends

Crawl frequency over time is a leading indicator. If the total number of AI bot visits to a site increases week over week, it means AI platforms are paying more attention to that domain. Declining frequency can signal that bots are deprioritizing the site, often because content has gone stale or a competitor has published fresher material on the same topics.

Agencies track these trends in monthly client reports to show measurable progress in AI visibility. A rising crawl curve provides tangible evidence that optimization efforts are working, even before citation data confirms it.

Finding Coverage Gaps

Coverage gaps are pages that no AI bot has visited during the reporting period. These pages are effectively invisible to AI platforms. They will not be cited, recommended, or referenced in any AI conversation. Common causes include poor internal linking (bots cannot discover the page), robots.txt blocks, thin content that crawlers deprioritize, or pages behind JavaScript-rendered navigation that bots cannot parse. BotSight surfaces these gaps automatically and includes discoverability recommendations for each one.

Correlating Crawls with Citations

The most powerful use of BotSight data comes from pairing it with citation scraping from our competitive intelligence service. When you know which pages bots crawl and which pages get cited, you can close the loop: identify pages that get crawled but not cited (content quality or structure issue), pages that get cited but show declining crawl frequency (freshness risk), and pages that get neither crawled nor cited (discovery and content gap). This correlation is what turns monitoring data into an actionable optimization roadmap.

Scoring

The AI Visibility Score

A composite metric (0-100) summarizing how well a site is being discovered by AI platforms. Four equally-weighted components:

25 pts
Bot Diversity

How many distinct AI bots visit. Broader diversity = broader platform coverage.

25 pts
Crawl Frequency

Total hits over the reporting period. Higher frequency = active content sourcing.

25 pts
Recrawl Rate

Pages revisited by bots. Recrawling = the platform values your content freshness.

25 pts
Page Coverage

Percentage of important pages discovered by at least one AI bot.

The score gives agencies a single, trackable number to report to clients and measure over time. It also highlights the weakest component so optimization effort goes where it matters most.

Deliverables

What's Included

  • Monitoring deployment — BotSight installed on the client's site (Vercel, Cloudflare, or custom server)
  • Regular visibility reports — Bot activity, page-level performance, and AI Visibility Score trends
  • Bot-by-bot breakdown — Each AI bot's crawling patterns, frequency, and page preferences
  • AI revenue attribution — Track purchases and revenue driven by AI referral traffic, broken down by platform (ChatGPT, Perplexity, Claude, etc.) and landing page
  • AI engagement metrics — Page-level engagement data from AI-referred visitors, including sessions, users, and key events
  • Crawl-to-conversion funnels — Map which pages bots crawled that later generated purchases, connecting bot activity to actual business outcomes
  • Estimated revenue lift — Quantify the revenue impact of AI-driven traffic to demonstrate ROI to clients
  • Coverage gap identification — Pages not yet discovered by AI bots, with discoverability recommendations
  • Freshness alerts — Notifications when popular content hasn't been recrawled in 21+ days
  • Search correlation analysis — Cross-referencing bot crawl data with GSC, Bing, and GA4 to identify optimization targets and AI discovery priorities
  • Security monitoring — Detection of vulnerability scans and attack attempts via the middleware layer
  • Exportable reports — Styled PDFs and Excel workbooks with AI revenue sections, charts, score breakdowns, and trend data
  • Actionable recommendations — Specific technical guidance based on monitoring data

Technology

The Technology Behind It

BotSight is a proprietary monitoring system built specifically for AI bot tracking. It connects to Google Search Console, Bing Webmaster Tools, and GA4 ecommerce data to cross-reference bot activity with real business outcomes.

Learn more on the toolkit page.

Transparency

Limitations & Considerations

Monitoring shows bot visits and AI-referred revenue, not citations directly. Crawling is a precondition for citation, and our crawl-to-conversion funnels connect bot activity to actual purchases. For citation-level analysis, pair with our competitive intelligence service.
Not all AI bots identify themselves. Some platforms use generic user agents or headless browsers. Our detection is based on known user agents and represents a lower bound on actual AI bot activity.
Index-only platforms leave no trace. Perplexity and Gemini rely on pre-built indices. BotSight tracks their indexing crawlers but cannot detect when content is served from the index during a conversation.
Correlation, not causation. A high AI Visibility Score indicates strong bot engagement, which correlates with better citation outcomes — but it is not a direct measure of citation probability. Use alongside technical audits and content strategy for a complete picture.

Explore

Related Services

AI SEO Audits

Technical audits evaluating pages against confirmed citation predictors from 3,251 real websites, tested across 3 independent models.

GEO Content Strategy

Intent-driven content strategy optimized for AI platform citation patterns.

Competitive Intelligence

Find out who AI platforms cite instead of your client — and why.
Best AI SEO agencies Read the research From the blog How we work

FAQ

Frequently Asked Questions

How do you track AI bots on my client's website?

We deploy edge middleware at the CDN level (Vercel, Cloudflare, or custom server). It tracks 15+ AI bots in real time with no JavaScript dependency, so it catches bots that traditional analytics tools miss entirely.

What is an AI Visibility Score?

The AI Visibility Score is a composite metric (0-100) across four equally weighted components: bot diversity, crawl frequency, recrawl rate, and page coverage. It gives agencies a single trackable number for client reporting.

How is this different from Google Analytics?

GA4 misses 90%+ of AI bot activity because AI bots do not execute JavaScript. Our edge middleware operates at the CDN level and detects bots by user agent before JavaScript ever loads, capturing activity that GA4 cannot see.

See What's Crawling Your Client's Site Right Now

Run a free AI visibility check to see which bots are visiting , and which aren't. Or tell us about your client and we'll set up full monitoring.

Free AI Check Start a Conversation