URL Sources & Refreshing
URL Sources & Refreshing
When you add a URL to your knowledge base, your agent:
- Fetches the page content via Jina Reader (extracts clean text, strips ads and navigation)
- Compiles it into a structured markdown summary (~2K chars)
- Tags it with topic keywords for relevance ordering
- Stores the compiled summary — not the live page
When Sources Go Stale
URL sources are fetched once at the time you add them. They don't auto-refresh. If the page changes (new pricing, updated docs), your agent won't know until you refresh it.
Your admin coaching session will flag stale sources in its Knowledge status line when a URL source hasn't been refreshed in more than 7 days. It looks like:
Knowledge: 3 docs (8.2K chars) · ⚠️ pricing URL stale (12d)
Refreshing a URL Source
In admin chat:
"Refresh my pricing page"
"Update my docs source"
The agent calls refresh_knowledge, re-fetches the URL, and recompiles if the content has changed. If nothing changed, it says so and doesn't update.
Images Can't Be Refreshed
Image sources (diagrams, screenshots) can't be re-fetched — they were uploaded as files, not URLs. To update an image source, remove the old entry and re-upload the new version.
Tips
- Add your most important URLs first — they get the highest relevance when matching visitor questions
- Give entries descriptive titles so you can find them easily in list view
- If a page has multiple sections you care about (e.g., pricing AND FAQ), you can add it as two separate entries with different titles and tags
- For high-change pages (pricing, changelog), refresh every time you update them