Website
Automatically crawl and index your public website, documentation site, or help center. Pillar visits each page, extracts the content, and adds it to your knowledge base.
When to Use
- You have a public docs site or help center
- You want the co-pilot to answer questions from your marketing site
- Your content is already published on the web
Setup
1
Go to Knowledge in the dashboard
Open the Pillar dashboard and navigate to the Knowledge section.
2
Add a new source
Click Add Source and select Website.
3
Enter your URL
Provide the root URL of the site you want to crawl (e.g., https://docs.example.com).
4
Configure crawl settings
Optionally set a maximum number of pages, or add include/exclude path patterns to control which pages are indexed.
5
Start the crawl
Click Create to start. Pillar will crawl the site and begin indexing content.
Crawl Configuration
| Setting | Description |
|---|---|
| URL | The root URL to start crawling from |
| Max Pages | Limit the number of pages crawled (optional) |
| Include Paths | Only crawl URLs matching these patterns |
| Exclude Paths | Skip URLs matching these patterns |
Path Patterns
Use include and exclude paths to control which pages get indexed:
- Include
/docs/*to only crawl your documentation - Exclude
/blog/*to skip blog posts - Exclude
/changelog/*to skip release notes
Sync Behavior
- First crawl: All pages are crawled and indexed immediately
- Periodic sync: Pillar re-crawls daily to pick up new or updated content
- Manual sync: Trigger a re-crawl anytime from the dashboard
- Incremental updates: Only new or changed pages are reprocessed
Tips
- Start with a specific section (like
/docs) rather than your entire domain - Use max pages to limit the initial crawl while testing
- Check the knowledge items list after a crawl to verify the right pages were indexed
- Remove pages you don't want by toggling them inactive in the dashboard