# Track search and AI crawlers

This article explains how to see which search engine and AI crawlers read your site and when Google last read a page. You will also learn what your site offers AI assistants: the Markdown version of each page and the `llms.txt` file.

## Before you start

- Your site recognizes a crawler from Google, Bing or an AI by its IP address, among those published by the company that runs it. A recognized crawler is called verified. A program that presents itself as Googlebot from another address is not one.
- Each page read by a verified crawler is a pass. A pass is never a visit: it is not counted in your page views, your visitors or the page views of your plan.
- Passes are read in the **Statistics** menu, which requires the **Editor** or **Administrator** role.

## Understand the robots.txt file

The `robots.txt` file, at the root of your domain, tells crawlers what they may explore. Your site writes it according to the environment. It has no settings screen.

| Environment | Content |
| --- | --- |
| Production | All crawlers may explore everything. A `Sitemap:` line gives the address of your sitemap.xml. |
| Staging | No crawler may explore anything. The pages also carry the `noindex` instruction. |

## See which crawlers pass

1. In the **Statistics** menu, open the **Bots** tab.
2. Read the **Search & AI crawlers** card. It counts the passes of verified crawlers over the period, then gives one crawler per row.

![The Search & AI crawlers card: 15,508 passes over the period, then Googlebot, Bingbot, GPTBot, ClaudeBot and the other crawlers, each with its Search or AI category, its passes, its pages, its 200 responses, its redirects, its 404s, its errors and its last pass.](https://static.lovelyrender.com/uploads/2026/10/983-doc-trouve-robots.png)

| Column | Content |
| --- | --- |
| **Crawler** | The name of the crawler, for example Googlebot, Bingbot or GPTBot. |
| **Category** | **Search** for a search engine, **AI** for an artificial intelligence. |
| **Passes** | The number of pages read over the period. |
| **Pages** | The number of different pages read. |
| **200**, **Redirects**, **404**, **Errors** | The responses of your site: the page, a redirect, an address not found or another error. |
| **Last pass** | The date and time of the last pass, in your site’s time zone. |

With no verified crawler over the period, the card shows **No verified crawler in the period.**

### Open the detail of a crawler

1. Click the name of a crawler. Its detail opens in a side panel.
2. Read **Pages read**: the pages it read over the period, the most read first, with the last status and the last pass.
3. Scroll down to **Last passes**: its latest passes, one per row.

Clicking a page in **Pages read** opens its detail instead.

**Tip:** A crawler that your site serves 404s explores dead addresses. Open its detail to see them, then redirect those that have a replacement: [Redirect an old address](https://lovelyrender.com/documentation/redirect-old-url/) explains how.

### Read the crawler traffic

The **Bot traffic** card, under the first one, lists the visits of bots that give their name without being verified. For example an SEO tool like AhrefsBot, the preview of a link shared on a social network or a messaging app, or an AI launched from its user’s computer. Each row gives its **User agent** and its **Sessions**.

Scanners, command-line tools and rented servers do not appear there: their visits are counted nowhere.

## Find out whether Google read a page

1. Open the detail of the page: click its address in a card of the **Statistics** screen.
2. Scroll down to the visibility card and click **Crawl health**.
3. Read the three tiles **Google**, **Bing** and **AI**. Each gives the last pass of that family of crawlers on the page, for example **2 hours ago**. The date and time show below.

![The Crawl health view in the detail of the Pricing page: the Google, Bing and AI tiles, each with its last pass 2 hours ago, then the table of crawlers with their passes on the page and on its Markdown, their last status and their last pass.](https://static.lovelyrender.com/uploads/2026/10/986-doc-trouve-exploration.png)

The tiles cover the last 90 days, whatever period is chosen. With no pass, a tile shows **No pass in 90 days**. When the last status served was not 200, it shows in red next to it: a 404 served to Googlebot is an alert.

Under the tiles, a table gives for each crawler its passes on the page and on its Markdown, its **Last status** and its **Last pass**. Further down, **Last passes** lists the latest passes of the period. Each row gives the **Date & time**, the **Crawler**, the **Type**, the **Status** and the **Document** (the page or its Markdown). For Googlebot and Bingbot, the type is **Smartphone** or **Desktop**.

**Note:** The **Google** tile counts Google’s search crawlers. Its tools launched by a person, like a link preview, are not included.

### How long passes are kept

The detail of each pass is kept 90 days, with its date and time, the crawler, the status served and the version read. No IP address is kept. The count of passes per day stays as long as your visit statistics. Your site’s archive contains both: [Take your site with you](https://lovelyrender.com/documentation/export-your-site/) describes it.

## Offer your pages to AIs in Markdown

Every public page of your site also exists in Markdown, the format that AI assistants read best. You have nothing to set.

- The address of the page followed by `.md` serves it, for example `/tarifs.md`. For the home page, the address is `/index.md`.
- The page’s usual address also serves it to an assistant that asks for Markdown.
- Each page announces its Markdown version in its code and declares the normal page as the reference address.

| Page | Markdown version |
| --- | --- |
| A content | Its title and its content: headings, paragraphs, lists, images, tables and links. A form is flagged in one sentence. |
| The home page | Its content, without header or footer. |
| A list | The name of the list, its number of entries, the current page and one link per entry to its Markdown version. |
| A search or a page not found | No Markdown version. |

Where the Organization schema shows, the Markdown version ends with the section **About the organization**. It gives your name, your address, your phone, your email, your opening hours, your price range and your linked accounts.

**Warning:** The Markdown version of a content includes only what is written in the content. A list or a field displayed by the template does not appear in it.

The **View Markdown** link, at the top of a page’s detail in the statistics, opens its Markdown version.

### Count AI visits

An AI visit is a read of the Markdown version of a page by a program that is not a verified crawler, most often an AI assistant. It is never added to views or to visitors.

- The **AI visits** indicator of the overview counts them over the period.
- The **AI visits** column of the **All pages** tab counts them page by page.
- In the detail of a page, the **AI visits** view separates them **At its .md address** and **At the page’s own address**.

The reads of verified crawlers are counted in **Crawl health**.

## Write the index for AIs

The `/llms.txt` file, at the root of your site, says in a few lines what the site is and where to read its content. It is built from your site. It gives your name and the address of your sitemap.xml, then one section per content type with the sentence that describes it and its entries. A type with too many entries to name them all gives the address of its list instead, if it has one. Otherwise, it names its most recently modified entries.

1. In the **Visibility** menu, open **AI index**.
2. In **About this site**, write two or three sentences that introduce your site to an AI.
3. Click **Save**. The message **Description saved.** appears.
4. Review **The file as it will be served**, at the bottom of the screen.

Left empty, the field takes the description of your organization, otherwise the site’s tagline. The sentence for each content type is set in the type itself.

**Note:** In Production, anyone can read the file. On Staging, only logged-in members see it.

## See also

- [Set a page’s SEO](https://lovelyrender.com/documentation/set-page-seo/)
- [Read your analytics](https://lovelyrender.com/documentation/read-analytics/)
- [Connect your AI](https://lovelyrender.com/documentation/connect-ai/)