Skip to main content

Website Crawling AI

Learn how to index your public marketing pages and support documentation automatically. Sentrup crawls sitemaps, scrapes content dynamically, checks hashes, and generates high-density vector embeddings.

How Does Sitemap and URL Discovery Work?​

Sitemap and URL discovery works by fetching the public sitemap.xml file of a target website. Sentrup parses the URLs using high-performance regex scanners. If no sitemap is present, the crawler automatically extracts outgoing hyperlinks on the homepage to map the website.

What is the Waterfall Scraper Pipeline?​

The waterfall scraper pipeline is a multi-tier scraping mechanism that optimizes crawl speed and accuracy. It processes static pages via fast HTML fetches, dynamically rendered guides via our dynamic document reader, and React or Next.js single-page applications via our single-page application scraper.

Scraper TierServiceTarget Use CaseLatencyCost
Tier 1 (Cheap Static)Local HTML FetchStatic pages, plain markdown, articles<500msFree
Tier 2 (Medium Dynamic)Dynamic Document ReaderDocumentation sites, guides, API hubs~1.2sLow
Tier 3 (Heavy SPA/CF)SPA ScraperReact/Next SPAs, Cloudflare-protected domains~3.5sManaged

How Does Incremental Hashing and Vectorization Work?​

Incremental hashing and vectorization prevents redundant processing by computing SHA-256 hashes of extracted text. Sentrup only runs text chunking and vector embedding generation if the content hash differs from the stored values in our vector database, saving API calls.

What is the Crawl Request API Payload?​

The crawl request API payload is a structured JSON body sent via POST to initiate an automated crawl. It defines parameters such as target URL, page limit, path filters, and source priority to constrain the scope of the crawl.

POST /api/v1/admin/documents/crawl
Content-Type: application/json
Authorization: Bearer <your-admin-token>

{
  "url": "https://www.sentrup.com",
  "max_pages": 15,
  "source_priority": "high",
  "allowed_paths": ["/docs", "/faq", "/pricing"]
}