Skip to content
Skills Library
Data & analytics · Automation

web-scraper

/web-scraper

Scrape, crawl, and extract data from websites. Use when users ask to scrape web pages, extract content, crawl websites, or collect data from the internet.

Repository
shob ↗
Licence
MIT
Last repository change
18.09.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
6
Deprecation

No signals detected. Repository changed on 18.09.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Scrape, crawl, and extract data from websites. Use when users ask to scrape web pages, extract content, crawl websites, or collect data from the internet.

Original author description

Scrape, crawl, and extract data from websites. Use when users ask to scrape web pages, extract content, crawl websites, or collect data from the internet.

How it works

  1. Overview
  2. When to Use
  3. Tools Available
  4. Usage Patterns
  5. Extraction Modes
  6. Best Practices
  7. Data Handling
Original SKILL.md headings · 108 lines, 346 words
SKILL.md preview
---
name: web-scraper
description: Scrape, crawl, and extract data from websites. Use when users ask to scrape web pages, extract content, crawl websites, or collect data from the internet.
---

# Web Scraper

## Overview
Extract content and data from websites using various techniques including crawling, scraping, and structured data extraction.

## When to Use
- Extract text content from web pages
- Crawl entire websites
- Collect structured data
- Research and gather information
- Monitor website changes
- Extract tables and lists

## Tools Available

### Content Extraction
```javascript
// Use extract_content_from_websites for structured extraction
// Supports batch processing of multiple URLs
// Returns JSON format with extracted content
```

### Task Format
```javascript
{
    tasks: [
        {
            url: "https://example.com",
            prompt: "Extract specific information",
            task_name: "optional_name"
        }
    ]
}
```

## Usage Patterns

### Simple Content Extraction
```javascript
// Extract main content from a page
const result = await extract_content_from_websites({
    tasks: [{
        url: "https://news.example.com/article",
        prompt: "Extract the title, author, date, and main content"
    }]
});
```

### Batch URL Processing
```javascript
// Process multiple URLs in parallel
const urls = [
    "https://site.com/page1",
    "https://site.com/page2",
    "https://site.com/page3"
];

const results = await extract_content_from_websites({
    tasks: urls.map((url, i) => ({
        url,
        prompt: "Extract all product information, prices, and descriptions",
        task_name: `product_${i}`
    }))
});
```

### Data Mining
```javascript
// Extract structured data like prices, reviews, specifications
const data = await extract_content_from_websites({
    tasks: [{
        url: "https://ecommerce.example.com/products",
        prompt: "Extract product name, price, rating, and availability for all products listed"
    }]
});
```

## Extraction Modes

### Auto Mode (Default)
- Attempts HTTP GET first
- Falls back to browser rendering for CSR pages
- Best for most websites

### Curl Only Mode
- Fast direct HTTP requests
- Best for static HTML pages
- May fail on JavaScript-heavy sites

### Browser Only Mode
- Full browser rendering
- Handles dynamic content
- Slower but more comprehensive

## Best Practices
1. Start with simpler extraction before complex patterns
2. Use specific prompts for targeted data
3. Respect website terms of service
4. Add delays between requests when scraping multiple pages
5. Handle errors gracefully with try/catch

## Data Handling
- Returns JSON format for easy processing
- Handles batch operations efficiently
- Supports pagination when needed
- Maintains data structure in results

Versions

RepositoryScoreLicence
This page · shobcoder/shob100MIT
TerminalSkills/skills100Apache-2.0
seaworld008/commonly-used-high-value-skills100MIT
jqaisystems/jqai-ai-skills100MIT
sandbaseai/sandbase-skills100Apache-2.0
guia-matthieu/clawfu-skills100MIT
zocomputer/skills100MIT
skrun-dev/skrun90MIT
LeoYeAI/openclaw-master-skills90MIT

Database snapshot: 13.11.2026 · package version a9e848d7eb57 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100web-scraping
    Scrape websites, extract structured data, and automate browsers. Use when asked to scrape, extract, crawl, parse, or pull data from web pages or any URL.
  • 100firecrawl-scraper
    Use this skill when users need to scrape web pages, extract structured page data, take website screenshots, parse PDFs, batch-scrape URLs, or crawl a site with Firecrawl.
  • 100just-scrape
    Search, scrape, crawl, extract structured data, and monitor web pages via the ScrapeGraph AI CLI. Use when the user asks to search the web, scrape a webpage, grab content from a URL, extract JSON from a site, crawl documentation or site sections, monitor a page for changes,…
  • 100crawl4ai
    Web crawling and data extraction using the crwl CLI. Use this skill when users need to scrape websites, extract markdown content, handle JavaScript-heavy pages, or extract structured data from web pages.
  • 100crawl4ai
    Crawl, scrape, question, or extract structured data from live websites with Crawl4AI and optional Trafilatura. Use for fresh web content, multi-page crawls, compact page reading, and JSON extraction.
  • 95scraper-builder
    Build production-ready web scrapers for any website using Bright Data infrastructure. Guides you through site analysis, API selection, selector extraction, pagination handling, and complete scraper implementation. Use this skill whenever the user wants to build a scraper, create…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.