Programowanie · Budowanie i kod
scrapy
/scrapy
Pomaga pisać i debugować projekty Scrapy, czyli spidery, oraz uruchamiać i sprawdzać przebiegi crawli.
Dla kogo
- Autor
- scrapy ↗
- Repozytorium
- scrapy-agent-plugin ↗
- Licencja
- Apache-2.0
- Ostatnia zmiana repozytorium
- 10.09.2026
Ocena i bezpieczeństwo
100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
- Pliki w paczce
- 1
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 4
Przestarzałość
Brak sygnałów. Repozytorium zmienione 10.09.2026, bez odwołań do wycofanych modeli i API.
Licencja
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
Co robi i kiedy
Pomaga pisać i debugować projekty Scrapy, czyli spidery, oraz uruchamiać i sprawdzać przebiegi crawli.
Oryginalny opis (EN)
Use when working on a Scrapy project — writing or debugging spiders, running or inspecting crawls.
Jak działa
- LLM-friendly documentation
- Using the Scrapy MCP server
- Differences in API and behavior between Scrapy versions
- Additional useful libraries
Sekcje z nagłówków SKILL.md, w oryginale · 80 linii, 633 słów
Podgląd SKILL.md
--- name: scrapy description: Use when working on a Scrapy project — writing or debugging spiders, running or inspecting crawls. --- # Scrapy ## LLM-friendly documentation The official documentation for Scrapy and many related libraries is LLM-friendly: `llms.txt` provides a table of contents, and all pages are available in Markdown; just replace `.html` with `.md` in the URL. Fetch this documentation when needed. For example: ``` https://docs.scrapy.org/llms.txt https://web-poet.readthedocs.io/en/stable/page-objects/fields.md ``` ## Using the Scrapy MCP server This plugin configures the Scrapy MCP server (https://github.com/scrapy/scrapy-mcp-official), which connects to running Scrapy jobs via the RemoteControl extension. Only locally running jobs that use Scrapy 2.19.0+ are supported and returned by the `list_jobs` tool. Some tasks where this MCP server can be useful: - listing running crawl processes - investigating why a crawl is slow, doesn't make requests, or makes requests without producing items - checking the progress and statistics of a crawl - modifying the state or even the code of a crawl to fix runtime issues or change its configuration without restarting it ## Differences in API and behavior between Scrapy versions Code written for one Scrapy version may be broken when running with a different one because the Scrapy API and behavior evolves over time, and your training data may be outdated compared to the Scrapy version used in the project. Always check which Scrapy version is used by a project or a running spider and always consult the Scrapy documentation for the specific version when writing code or troubleshooting problems with existing code. Consult https://docs.scrapy.org/en/latest/news.md for the full list of changes in each Scrapy version. Here are the most important changes between recent Scrapy versions, covering 2.8.0–2.18.0. - **Start requests:** since 2.12 `start_requests()` may yield items; since 2.13 spiders should define `async def start()` instead of `start_requests()`; since 2.16 `start_requests()` is never called, so a spider that defines only it produces no requests and no warning. - **Component instantiation:** since 2.14 Scrapy never calls `from_settings()` methods of components (use `from_crawler()`) and `scrapy.utils.misc.create_instance()` is removed in favor of `build_from_crawler()`. - **Component search methods:** 2.12 adds the `get_addon()`, `get_downloader_middleware()`, `get_extension()`, `get_item_pipeline()` and `get_spider_middleware()` methods to `Crawler`. - **Crawler attribute initialization:** since 2.18 `crawler.engine`, `extensions`, `logformatter`, `request_fingerprinter` and `stats` raise `RuntimeError` when read before the crawl starts instead of being `None`. - **Add-ons:** since 2.10 third-party components can be configured through the `ADDONS` setting. - **Spider middleware async output:** since 2.16 `process_spider_output()` must be an async generator or be paired with a `process_spider_output_async()` method; `process_start_requests()` is replaced by the async `process_start()` in 2.13 and removed in 2.16. - **Reactor defaults:** since 2.13 the default `TWISTED_REACTOR` is the asyncio one. - **Process and runner classes:** 2.14 adds `AsyncCrawlerProcess` and `AsyncCrawlerRunner`, coroutine-based counterparts of `CrawlerProcess` and `CrawlerRunner`. - **httpx-based download handler:** 2.15 adds `HttpxDownloadHandler`; it gains proxy support in 2.16, HTTP/2 support and SOCKS proxies in 2.17. - **Compression:** since 2.18 Brotli and Zstandard support is always available (`brotli` and, on Python 3.13 and lower, `backports.zstd` are required dependencies), so `Accept-Encoding` always advertises `br` and `zstd` and such responses are always decoded; on older Scrapy versions the `brotli` and `zstandard` packages must be installed explicitly for this. - **scrapy.utils.url re-exports:** the functions re-exported from w3lib, including `canonicalize_url`, `safe_url_string`, `add_or_replace_parameter`, `url_query_parameter`, `url_query_cleaner`, `parse_url`, `is_url`, `any_to_uri`, `file_uri_to_path`, `path_to_file_uri`, `parse_data_uri` and `safe_download_url`, are removed in 2.16; always import them from `w3lib.url`. ## Additional useful libraries There are many third-party libraries that may be helpful when writing spiders; here are some examples: - **Better code structure:** `web-poet` and `scrapy-poet` (separation of crawling and extraction code), `scrapy-spider-metadata` (structured spider arguments) - **Better data extraction:** `extruct` (getting structured data embedded in HTML), `dateparser` (parsing date strings), `price-parser` (parsing price strings), `number-parser` (parsing number strings), `clear-html` (cleaning and normalizing HTML), `html-text` (extracting text from HTML), `zyte-parsers` (extracting some data like ratings/review counts from HTML) - **Help with bans:** `scrapy-playwright` (using a local headless browser), `scrapy-zyte-api` (using Zyte API) - **QA:** `spidermon` (verifying crawl success and data quality)
Wersje
W bazie nie ma innych skilli o nazwie scrapy.
Migawka bazy z 13.11.2026 · wersja paczki ea4a732601ae (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 100devartslab-scrapy-resources
Pomaga budować, refaktoryzować i debugować pająki Scrapy w projektach city-scrapers. - 100shub-workflow-scripts
Pisze lub poprawia skrypty shub-workflow, które planują spidery, odpytują zadania Scrapy Cloud albo działają jako menedżer crawli. - 87shub-workflow-architectures
Pomaga zaprojektować architekturę projektu Scrapy Cloud opartego na shub-workflow: menedżerów pełzania, konsumentów i dostawę danych. - 100web-scraping
Obejmuje scraping stron pod ingest do RAG: Firecrawl API, Crawl4AI, trafilatura, BeautifulSoup, Scrapy, Playwright, sitemap.xml, robots.txt, uprzejme crawlowanie i deduplikację.