web-scraper
/web-scraper
Scrape, crawl, and extract data from websites. Use when users ask to scrape web pages, extract content, crawl websites, or collect data from the internet.
- Author
- shobcoder ↗
- Repository
- shob ↗
- Licence
- MIT
- Last repository change
- 18.09.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 6
No signals detected. Repository changed on 18.09.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Scrape, crawl, and extract data from websites. Use when users ask to scrape web pages, extract content, crawl websites, or collect data from the internet.
Original author description
Scrape, crawl, and extract data from websites. Use when users ask to scrape web pages, extract content, crawl websites, or collect data from the internet.
How it works
- Overview
- When to Use
- Tools Available
- Usage Patterns
- Extraction Modes
- Best Practices
- Data Handling
SKILL.md preview
---
name: web-scraper
description: Scrape, crawl, and extract data from websites. Use when users ask to scrape web pages, extract content, crawl websites, or collect data from the internet.
---
# Web Scraper
## Overview
Extract content and data from websites using various techniques including crawling, scraping, and structured data extraction.
## When to Use
- Extract text content from web pages
- Crawl entire websites
- Collect structured data
- Research and gather information
- Monitor website changes
- Extract tables and lists
## Tools Available
### Content Extraction
```javascript
// Use extract_content_from_websites for structured extraction
// Supports batch processing of multiple URLs
// Returns JSON format with extracted content
```
### Task Format
```javascript
{
tasks: [
{
url: "https://example.com",
prompt: "Extract specific information",
task_name: "optional_name"
}
]
}
```
## Usage Patterns
### Simple Content Extraction
```javascript
// Extract main content from a page
const result = await extract_content_from_websites({
tasks: [{
url: "https://news.example.com/article",
prompt: "Extract the title, author, date, and main content"
}]
});
```
### Batch URL Processing
```javascript
// Process multiple URLs in parallel
const urls = [
"https://site.com/page1",
"https://site.com/page2",
"https://site.com/page3"
];
const results = await extract_content_from_websites({
tasks: urls.map((url, i) => ({
url,
prompt: "Extract all product information, prices, and descriptions",
task_name: `product_${i}`
}))
});
```
### Data Mining
```javascript
// Extract structured data like prices, reviews, specifications
const data = await extract_content_from_websites({
tasks: [{
url: "https://ecommerce.example.com/products",
prompt: "Extract product name, price, rating, and availability for all products listed"
}]
});
```
## Extraction Modes
### Auto Mode (Default)
- Attempts HTTP GET first
- Falls back to browser rendering for CSR pages
- Best for most websites
### Curl Only Mode
- Fast direct HTTP requests
- Best for static HTML pages
- May fail on JavaScript-heavy sites
### Browser Only Mode
- Full browser rendering
- Handles dynamic content
- Slower but more comprehensive
## Best Practices
1. Start with simpler extraction before complex patterns
2. Use specific prompts for targeted data
3. Respect website terms of service
4. Add delays between requests when scraping multiple pages
5. Handle errors gracefully with try/catch
## Data Handling
- Returns JSON format for easy processing
- Handles batch operations efficiently
- Supports pagination when needed
- Maintains data structure in results
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · shobcoder/shob | 100 | MIT |
| TerminalSkills/skills | 100 | Apache-2.0 |
| seaworld008/commonly-used-high-value-skills | 100 | MIT |
| jqaisystems/jqai-ai-skills | 100 | MIT |
| sandbaseai/sandbase-skills | 100 | Apache-2.0 |
| guia-matthieu/clawfu-skills | 100 | MIT |
| zocomputer/skills | 100 | MIT |
| skrun-dev/skrun | 90 | MIT |
| LeoYeAI/openclaw-master-skills | 90 | MIT |
Database snapshot: 13.11.2026 · package version a9e848d7eb57 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100web-scraping
Scrape websites, extract structured data, and automate browsers. Use when asked to scrape, extract, crawl, parse, or pull data from web pages or any URL. - 100firecrawl-scraper
Use this skill when users need to scrape web pages, extract structured page data, take website screenshots, parse PDFs, batch-scrape URLs, or crawl a site with Firecrawl. - 100just-scrape
Search, scrape, crawl, extract structured data, and monitor web pages via the ScrapeGraph AI CLI. Use when the user asks to search the web, scrape a webpage, grab content from a URL, extract JSON from a site, crawl documentation or site sections, monitor a page for changes,… - 100crawl4ai
Web crawling and data extraction using the crwl CLI. Use this skill when users need to scrape websites, extract markdown content, handle JavaScript-heavy pages, or extract structured data from web pages. - 100crawl4ai
Crawl, scrape, question, or extract structured data from live websites with Crawl4AI and optional Trafilatura. Use for fresh web content, multi-page crawls, compact page reading, and JSON extraction. - 95scraper-builder
Build production-ready web scrapers for any website using Bright Data infrastructure. Guides you through site analysis, API selection, selector extraction, pagination handling, and complete scraper implementation. Use this skill whenever the user wants to build a scraper, create…