Track search and AI crawlers

This article explains how to see which search engine and AI crawlers read your site and when Google last read a page. You will also learn what your site offers AI assistants: the Markdown version of each page and the llms.txt file.

Before you start

• Your site recognizes a crawler from Google, Bing or an AI by its IP address, among those published by the company that runs it. A recognized crawler is called verified. A program that presents itself as Googlebot from another address is not one.
• Each page read by a verified crawler is a pass. A pass is never a visit: it is not counted in your page views, your visitors or the page views of your plan.
• Passes are read in the Statistics menu, which requires the Editor or Administrator role.

Understand the robots.txt file

The robots.txt file, at the root of your domain, tells crawlers what they may explore. Your site writes it according to the environment. It has no settings screen.

See which crawlers pass

1. In the Statistics menu, open the Bots tab.
2. Read the Search & AI crawlers card. It counts the passes of verified crawlers over the period, then gives one crawler per row.

With no verified crawler over the period, the card shows No verified crawler in the period.

Open the detail of a crawler

1. Click the name of a crawler. Its detail opens in a side panel.
2. Read Pages read: the pages it read over the period, the most read first, with the last status and the last pass.
3. Scroll down to Last passes: its latest passes, one per row.

Clicking a page in Pages read opens its detail instead.

Read the crawler traffic

The Bot traffic card, under the first one, lists the visits of bots that give their name without being verified. For example an SEO tool like AhrefsBot, the preview of a link shared on a social network or a messaging app, or an AI launched from its user’s computer. Each row gives its User agent and its Sessions.

Scanners, command-line tools and rented servers do not appear there: their visits are counted nowhere.

Find out whether Google read a page

1. Open the detail of the page: click its address in a card of the Statistics screen.
2. Scroll down to the visibility card and click Crawl health.
3. Read the three tiles Google, Bing and AI. Each gives the last pass of that family of crawlers on the page, for example 2 hours ago. The date and time show below.

The tiles cover the last 90 days, whatever period is chosen. With no pass, a tile shows No pass in 90 days. When the last status served was not 200, it shows in red next to it: a 404 served to Googlebot is an alert.

Under the tiles, a table gives for each crawler its passes on the page and on its Markdown, its Last status and its Last pass. Further down, Last passes lists the latest passes of the period. Each row gives the Date & time, the Crawler, the Type, the Status and the Document (the page or its Markdown). For Googlebot and Bingbot, the type is Smartphone or Desktop.

How long passes are kept

The detail of each pass is kept 90 days, with its date and time, the crawler, the status served and the version read. No IP address is kept. The count of passes per day stays as long as your visit statistics. Your site’s archive contains both: Take your site with you describes it.

Offer your pages to AIs in Markdown

Every public page of your site also exists in Markdown, the format that AI assistants read best. You have nothing to set.

• The address of the page followed by .md serves it, for example /tarifs.md. For the home page, the address is /index.md.
• The page’s usual address also serves it to an assistant that asks for Markdown.
• Each page announces its Markdown version in its code and declares the normal page as the reference address.

Where the Organization schema shows, the Markdown version ends with the section About the organization. It gives your name, your address, your phone, your email, your opening hours, your price range and your linked accounts.

The View Markdown link, at the top of a page’s detail in the statistics, opens its Markdown version.

Count AI visits

An AI visit is a read of the Markdown version of a page by a program that is not a verified crawler, most often an AI assistant. It is never added to views or to visitors.

• The AI visits indicator of the overview counts them over the period.
• The AI visits column of the All pages tab counts them page by page.
• In the detail of a page, the AI visits view separates them At its .md address and At the page’s own address.

The reads of verified crawlers are counted in Crawl health.

Write the index for AIs

The /llms.txt file, at the root of your site, says in a few lines what the site is and where to read its content. It is built from your site. It gives your name and the address of your sitemap.xml, then one section per content type with the sentence that describes it and its entries. A type with too many entries to name them all gives the address of its list instead, if it has one. Otherwise, it names its most recently modified entries.

1. In the Visibility menu, open AI index.
2. In About this site, write two or three sentences that introduce your site to an AI.
3. Click Save. The message Description saved. appears.
4. Review The file as it will be served, at the bottom of the screen.

Left empty, the field takes the description of your organization, otherwise the site’s tagline. The sentence for each content type is set in the type itself.

See also

• Set a page’s SEO
• Read your analytics
• Connect your AI