Web Scrape
Last updated
Was this helpful?
Last updated
Was this helpful?
Web Scrape allows you to import content directly from websites into your Knowledge Base, making it easy to leverage existing pages on your website for use cases like Internal Linking. Choose between scraping individual pages or multiple pages via a sitemap based on your specific needs.
To scrape content from a single webpage:
Click "Add Data" and select "Web Scrape" from the dropdown menu
Choose the "Single Page" option
Click "Start Scraping" to begin the process
AirOps will extract and import the content into your Knowledge Base, making it searchable
To scrape content from multiple webpages at once:
Click "Add Data" and select "Web Scrape" from the dropdown menu
Choose the "Multiple Pages (via Sitemap)" option
Enter the sitemap URL (typically found at domain.com/sitemap.xml)
Optionally, specify URL patterns to include or exclude specific pages (e.g., include "/blog/" or exclude "/support/")
Toggle "Scrape Metadata Only" if you only need page titles and descriptions rather than full content
Click "Start Scraping" to begin the process and import all matching pages
Once a sitemap has been added to your Knowledge Base, you can keep the content up-to-date by clicking the gear icon next to "1 Sync Scheduled" and selecting either:
"Sync Data" to manually trigger an immediate sync and fetch the latest content
"Schedule Data Sync" to set up an automated recurring sync on your preferred schedule (e.g., weekly at 9:30 AM on Mondays), ensuring your Knowledge Base stays current without manual intervention
Enter the complete URL of the webpage you want to scrape (e.g., )
