Websites
How the crawler reads your site and the PDFs it links to, what it skips on purpose, and what to do when pages are missing.
A website is usually the fastest way to give an agent what it needs. You enter an address, the crawler reads the pages, and each page becomes a source of its own.
Three ways to read a site
On Sources, under Website, choose one:
- Crawl links starts at the address you give and follows every link that stays on the same site, in the page's menus and footer as well as its text. Best for most sites.
- Sitemap reads the pages listed in a sitemap file and nothing else. It follows no links. Use it when you want exact control, or when your navigation is built in a way the crawler cannot follow.
- Individual link reads that one page.
"The same site" is strict: the same scheme, the same host and the same port. A crawl that starts at https://www.example.com does not wander into https://shop.example.com or into the http:// version. Add those as crawls of their own.
Leaving parts out
In Crawl links mode you can list Exclude paths. Any address whose path starts with one of them is skipped, so /blog leaves out /blog, /blog/2024/a-post and also /blogroll. Excluding the parts an agent has no use for (news archives, job listings, tag pages) makes answers better, not just the crawl shorter, because the search has less noise to wade through.
How it behaves on your server
- It identifies itself as
ChatterLab/1.0 (+https://chatterlab.ai/bot). - It obeys your
robots.txt. If the file cannot be read, it assumes everything is allowed. ACrawl-delayline is not read. - It waits 1 second after every page it stores, and reads one page at a time.
- It reads web pages, up to 5 MB a page, and PDFs, up to 20 MB each, on the standard web ports. It does not run JavaScript.
- A single crawl discovers at most 5,000 addresses. A sitemap is read up to 1,000 addresses, and a sitemap that only lists other sitemaps is not opened further.
PDFs on your site
A PDF that your pages link to, on the same site, is read along with them: data sheets, manuals, certificates, price lists. Each becomes a source of its own, under its own address, named after the title in the document, or after its file name when that title only names the program that made it. In the list under Sources it has a document icon.
When an answer relies on one of them, or a visitor asks for such a document, the agent adds a link to it, so the visitor can open the PDF itself. It links only to the address where the crawl found the file.
A scanned PDF holds pictures of text rather than text, so it is skipped: there is nothing in it to read. See Files for what to do with one.
What it skips on purpose
- Images, videos and other files. A link to a picture, a film, an archive or an office document is not followed, and nothing is downloaded. Only pages and PDFs are read.
- Pages with a password field. A login page has nothing to teach an agent.
- Pages with almost nothing on them once menus and footers are gone (under 300 bytes of text).
- Repeated blocks. Once 8 pages are in, short blocks that appear on at least 80% of them are recognised as navigation, cookie notices and footers, and removed. Otherwise every answer would find your footer.
- Text a visitor cannot see. Elements hidden with
hidden,aria-hidden,display:noneand screen-reader-only classes are dropped, along with invisible characters. Hidden text is where instructions aimed at an AI get planted, and a human visitor never reads it anyway. See Security.
While it runs
A large site takes minutes. You can leave the page. The crawl works in short turns in the background, and if a turn is interrupted it picks itself up again after about 90 seconds. The page that turn was reading waits about 10 minutes before it is tried again: until then it cannot be told apart from a page another turn is still reading. If it ever shows Crawl paused, press Resume crawl: nothing already read is lost.
A crawl stops early when the agent's storage is full. It says so, with the number of pages it found and the number it kept. Exclude paths you do not need, or move to a plan with more room. Limits and freshness.
A crawl you no longer want is stopped with Stop crawl on the card. The pages it has read by then stay, as sources like any other, so delete the ones you do not want. The rest of the site is left unread, and a new crawl starts again from the first page.
When pages are missing
| What you see | Likely cause | What to do |
|---|---|---|
| Pages exist on the site but not in Sources | The page is only reachable through a menu built by JavaScript | Use Sitemap, or add the pages as Individual link |
| Pages are there but nearly empty | The site renders its content in the browser | Put the content in a file or a text source instead |
| Nothing at all was read | robots.txt disallows the crawler, or the site blocks unknown bots | Allow ChatterLab in robots.txt or in your firewall |
| A whole section is absent | It lives on another host, such as shop. or help. | Crawl that host separately |
Reading a site again
Crawling a site the agent already has does not give you every page twice. A page at an address the agent already holds is updated with what it says now, and its old passages go, so answers cannot cite the old version. A page on which nothing changed is left as it is, and new pages are added. An updated page counts against your storage only for the difference in size, so crawling again does not need room for a second copy.
What a crawl never does is remove. A page that has gone from the site, or moved to another address, keeps its source until you delete it, and answers can still cite it. After a big change to the site, look through Sources for pages that no longer exist. For a single changed page, add it again as an Individual link.
A page the agent already has that a new crawl turns away, as a login page or as a page with almost no text, keeps its source too. It is listed at the top of Sources under Turned away by a later crawl, with one button that removes them all once you have looked. A crawl takes any page with a password field for a login page, so on a site with a login box in its header the list can hold every page: check it before you remove them.