> For the complete documentation index, see [llms.txt](https://docs.airops.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.airops.com/context/knowledge-bases/add-data/web-scrape.md).

# Web Scrape

Import website content into a Knowledge Base and configure crawling access.

## Using Web Scrape in Knowledge Bases

Web Scrape allows you to import content directly from websites into your Knowledge Base, making it easy to leverage existing pages on your website for use cases like Internal Linking. Choose between scraping individual pages or multiple pages via a sitemap based on your specific needs.

<figure><img src="https://3762890407-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FX2n5yPRPynbnWuO4SH0M%2Fuploads%2Fgit-blob-70816d670c03d75008010d6d7d6c8057fb3f549f%2Fkb-web-scrape-01-menu.png?alt=media" alt="Add Data menu with Web Page highlighted above Sitemap"><figcaption><p>Choose Web Page for a single page or Sitemap for many pages.</p></figcaption></figure>

## Allow AirOps to crawl your site

Your website must allow both sitemap retrieval and the page requests used to import content. If you use a firewall, CDN, or bot-protection service, check its rules for each type of request.

### Allow direct sitemap requests

Allow the following combination for direct AirOps sitemap requests:

| Setting           | Value                                            |
| ----------------- | ------------------------------------------------ |
| Source IP address | `52.71.87.39`                                    |
| User-Agent        | `AirOpsSitemapCrawler/1.0 (+https://airops.com)` |

1. Scope the exception to the source IP and User-Agent above. Do not trust the User-Agent alone, because other clients can send the same header.
2. Apply the exception to the sitemap index and each child sitemap you want to import. Include any destination hostname or path reached through a redirect.
3. Make sure the matching requests can pass your bot-protection rules without a CAPTCHA, JavaScript challenge, or cookie-verification page. An IP exception in one firewall rule may not bypass a separate bot-protection policy.

{% hint style="info" %}
Allowlisting another AirOps crawler does not automatically allow the sitemap crawler. In particular, `AirOpsPageChangeTracker/1.0` is used for Onsite page-change detection, not as the Knowledge Base page-import identity.
{% endhint %}

### Allow page fetching

Knowledge Base page imports, including **Scrape Metadata Only**, use managed scraping infrastructure to fetch page content. These requests, and fallback sitemap requests, can use different source IPs and user agents from the direct sitemap request above.

If the sitemap loads but individual pages remain blocked, contact AirOps Support to identify the applicable page-fetching traffic and coordinate a site-specific exception with your security team. Do not assume the direct sitemap IP and User-Agent cover every page request, and avoid disabling bot protection across your whole site.

## Single Page

To scrape content from a single webpage:

* Click "Add Data" and select "Web Scrape" from the dropdown menu
* Choose the "Single Page" option
* Enter the complete URL of the webpage you want to scrape (e.g., [www.domain.com/blog/article-title](http://www.domain.com/blog/article-title))
* Click "Start Scraping" to begin the process
* AirOps will extract and import the content into your Knowledge Base, making it searchable

<figure><img src="https://3762890407-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FX2n5yPRPynbnWuO4SH0M%2Fuploads%2Fgit-blob-1f55740942f72b89ad54431e90537ee2163db96b%2Fkb-web-scrape-02-single-page.png?alt=media" alt="Add Web Page dialog with a spaceops.io page URL"><figcaption><p>Scrape a single page.</p></figcaption></figure>

### Multiple Pages via Sitemap

To scrape content from multiple webpages using a Sitemap:

* Click "Add Data" and select "Web Scrape" from the dropdown menu
* Choose the "Multiple Pages (via Sitemap)" option
* Enter the sitemap URL (typically found at domain.com/sitemap.xml)
* Optionally, specify URL patterns to include or exclude specific pages (e.g., include "/blog/" or exclude "/support/")
* Toggle "Scrape Metadata Only" if you only need page titles and descriptions rather than full content
* Click "Start Scraping" to start the import in the background. AirOps returns you to the Knowledge Base, where you can follow progress as documents appear.

If the sitemap is invalid or unreachable, the connector shows an error banner in the Knowledge Base. Check the sitemap URL and its accessibility before retrying.

<figure><img src="https://3762890407-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FX2n5yPRPynbnWuO4SH0M%2Fuploads%2Fgit-blob-a517a25c5dc8ef4a98c9359f5fb7a0a312c4582f%2Fkb-web-scrape-03-sitemap.png?alt=media" alt="Configure Web Scrape dialog with a sitemap URL, URL must contain /blog/, and URL must not contain /careers/"><figcaption><p>Scrape many pages from a sitemap.</p></figcaption></figure>

Once a sitemap has been added to your Knowledge Base, you can keep the content up-to-date by clicking the gear icon next to "1 Sync Scheduled" and selecting either:

* "Sync Data" to manually trigger an immediate sync and fetch the latest content
* "Schedule Data Sync" to set up an automated recurring sync on your preferred cadence. Sitemap connectors support **Hour**, **Day**, **Week**, and **Month** schedules. Choose Month for large or slowly-changing sites where weekly resyncs are heavier than you need.

{% @arcade/embed url="<https://app.arcade.software/share/sqgFTsbvoWnJJgwiRYIU>" flowId="sqgFTsbvoWnJJgwiRYIU" %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.airops.com/context/knowledge-bases/add-data/web-scrape.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
