Crawl your website into a knowledge base
Point it at pages you already maintain — and understand that the index is a snapshot, not a live mirror.
Crawling is the fastest way to give a bot a lot of context, and the fastest way to give it a lot of noise. The discipline is entirely in choosing which pages.
The thing to internalise: the index is a snapshot. Changing a price on your website does not change what the bot believes. Re-crawling is a step somebody has to remember, which makes crawled content a worse home for volatile facts than an uploaded document you control.
Steps
Add crawled pages to a knowledge base
They sit alongside PDFs and documents in the same base, and are retrieved the same way.
The difference is maintenance. An uploaded document changes when you change it; a crawled page is a snapshot from whenever you last crawled. For volatile facts like prices, the document you control is the safer home.
Choose pages, not your whole site
Product pages, pricing, FAQ, policies, opening hours, locations. Blog archives, news posts and case studies add volume and subtract precision — they contain sentences that look like answers and are not.
Exclude anything time-sensitive you no longer maintain
An old campaign landing page with last year's offer will be quoted as current, with total confidence. This is a common and costly failure.
Fix the page before you crawl it
Crawling a page that is itself wrong makes the problem worse, because now it is wrong in two places and one of them talks to customers.
Re-crawl after you change the site
Put it on the same checklist as publishing a price change. Nothing will remind you.
Test the pages you crawled specifically
Ask the bot the question each page answers and check it gives that page's answer rather than a blend of three pages.
Check it worked
Ask the bot your current price. If it is right, note the date. If your site changes and you do not re-crawl, that answer will drift silently — and this is the single most common way a well-built bot goes wrong months later.
The parts people get wrong
- Pages behind a login cannot be crawled. Upload that content as a document instead.
- For anything that changes often, an uploaded document you control is safer than a crawled page you have to remember to re-index.