Przejdź do treści
Biblioteka skilli
Programowanie · Budowanie i kod

shub-workflow-scripts

/shub-workflow-scripts

Pisze lub poprawia skrypty shub-workflow, które planują spidery, odpytują zadania Scrapy Cloud albo działają jako menedżer crawli.

Repozytorium
shub-workflow ↗
Licencja
Apache-2.0
Ostatnia zmiana repozytorium
23.10.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
6
Skrypty
4
Komentarze HTML
0
Adresy URL w SKILL.md
2
Przestarzałość

Brak sygnałów. Repozytorium zmienione 23.10.2026, bez odwołań do wycofanych modeli i API.

Licencja

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

Co sprawdzić przed instalacją
  • 4 skrypty w paczce. Przejrzyj je przed instalacją, bo Claude może je uruchomić.

Skrypty: examples/async_loop_script.py, examples/loop_script.py, examples/plain_script.py, examples/project_base_mixin.py

Co robi i kiedy

Pisze lub poprawia skrypty shub-workflow, które planują spidery, odpytują zadania Scrapy Cloud albo działają jako menedżer crawli.

Wyzwalacze z opisu

script
Oryginalny opis (EN)

Use when writing, fixing, or updating a Python script built on the shub_workflow.script base classes (BaseScript / BaseLoopScript / BaseLoopScriptAsyncMixin / ArgumentParserScript) — i.e. any script that operates on Scrapy Cloud: scheduling spiders or scripts, scanning/querying SC jobs, aggregating stats, or running as a crawl manager, monitor, scheduler, consumer/deliverer, or an ad-hoc CLI that talks to SC — whether it runs ON Scrapy Cloud or locally against a project. When asked to create a new "script", first confirm it is a Scrapy Cloud script (see below), since these base classes are the

Jak działa

  1. First: is this a Scrapy Cloud script?
  2. Choose the base class
  3. Core procedure
  4. Critical rules & footguns
  5. Entry point boilerplate
Sekcje z nagłówków SKILL.md, w oryginale · 80 linii, 617 słów
Podgląd SKILL.md
---
name: shub-workflow-scripts
description: >-
  Use when writing, fixing, or updating a Python script built on the shub_workflow.script base
  classes (BaseScript / BaseLoopScript / BaseLoopScriptAsyncMixin / ArgumentParserScript) — i.e. any
  script that operates on Scrapy Cloud: scheduling spiders or scripts, scanning/querying SC jobs,
  aggregating stats, or running as a crawl manager, monitor, scheduler, consumer/deliverer, or an
  ad-hoc CLI that talks to SC — whether it runs ON Scrapy Cloud or locally against a project. When
  asked to create a new "script", first confirm it is a Scrapy Cloud script (see below), since these
  base classes are the right tool precisely when the script deploys to or operates on Scrapy Cloud.
---

# Writing shub-workflow scripts

shub-workflow scripts subclass a base class in
[`shub_workflow/script.py`](https://github.com/scrapinghub/shub-workflow/blob/master/shub_workflow/script.py)
and get, for free: argument parsing (+ reusable `-g`/`-v` "programs"), project-id resolution, the
ScrapinghubClient, job scheduling with flow/name tagging, paginated+retrying job queries, a Scrapy
stats collector and an `FSHelper`. The deep reference is the wiki:
[Appendix B: Script Classes](https://github.com/scrapinghub/shub-workflow/wiki/Appendix-B:-Script-Classes).

## First: is this a Scrapy Cloud script?

A brand-new file isn't yet registered in the project's `setup.py`, so you can't tell from the repo.
**When asked to "create a script", confirm with the user whether it will deploy to or operate on
Scrapy Cloud** (run as an SC job, schedule spiders/scripts, scan/query SC jobs, aggregate stats,
etc.). If yes → use these base classes. If it's just a local utility with no SC interaction, a plain
script is fine and this skill doesn't apply. When editing an existing file that already imports
`shub_workflow.script`, this skill applies directly.

## Choose the base class

| Base class | Use when | Template |
| --- | --- | --- |
| `BaseScript` | one-shot: parse args, do work in `run()`, exit (the default) | [examples/plain_script.py](examples/plain_script.py) |
| `BaseLoopScript` | must repeat work on an interval / run continuously until stopped | [examples/loop_script.py](examples/loop_script.py) |
| `BaseLoopScriptAsyncMixin` (+ `BaseLoopScript`) | the loop cycle is asyncio-based (schedules/awaits many things at once) | [examples/async_loop_script.py](examples/async_loop_script.py) |
| `ArgumentParserScript` | only argparse + `PROGRAMS`, **no** SC access (rare; the base the others build on) | — |

Projects usually add a shared **base mixin** (common CLI options/helpers) that every concrete script
inherits — see [examples/project_base_mixin.py](examples/project_base_mixin.py). Check whether the
project already has one and build on it rather than re-adding shared options.

## Core procedure

1. Confirm it's an SC script (above); pick the base class; reuse the project's base mixin if present.
2. Set `description` (property) and add your arguments in `add_argparser_options()` —
   **always call `super()` first** so `--project-id`/`-g`/`-v`/etc. survive.
3. Implement the entry method: `run()` for `BaseScript`; `workflow_loop()` (returns `bool`; `async
   def` for the async mixin) plus optional `on_start()`/`on_close()` for loop scripts.
4. Use the base API for SC work — scheduling, job queries, tags, stats — see
   [references/api-cheatsheet.md](references/api-cheatsheet.md).
5. Add the `__main__` boilerplate (below).
6. To make it runnable, register it in the project's `setup.py`. Deployment itself is handled by the
   **`scrapy-cloud-deployment`** skill, not here.

## Critical rules & footguns

- **Always `super().add_argparser_options()`** before adding arguments, or you lose the framework
  flags (`--project-id`, `--flow-id`, `-g/-v`, loop flags).
- **Mixin order is `class X(ProjectMixin, BaseScript)`** — mixin first, concrete base last. The mixin
  must inherit the typing-only `BaseScriptProtocol` (never `BaseScript`), so the implementation
  isn't duplicated in the MRO.
- **Async entry point differs:** a `BaseLoopScriptAsyncMixin` script's `run()` is a coroutine —
  launch it with `asyncio.run(script.run())`, and make `workflow_loop` an `async def`.
- **Two project ids:** `self.project_id` is the *target* (where you schedule/query, from
  `--project-id`); the script's own running project can differ. Don't hardcode ids; pass
  `--project-id` or set `default_project_id`.
- **`workflow_loop()` returns `bool`:** `True` keeps looping (with `--loop-mode`/`loop_mode`);
  `False` stops immediately. A loop with `loop_mode = 0` runs its body once.
- **Set `project_required = False`** only for scripts that genuinely never touch SC.
- **File I/O? Use the built-in `self.fshelper`** (an `FSHelper`) — don't construct your own
  `FSHelper`/`S3Helper`. To configure its credentials/role/ACLs, **override the `init_fshelper()`
  hook** (default is a bare `FSHelper()`); `self.project_settings` is available there. See the
  api-cheatsheet's *File operations* and the **`shub-workflow-fshelper`** skill.
- For the `-g`/`-v` `PROGRAMS` mechanism (predefined command-line shortcuts), see the
  **`scanjobs-programs`** skill — `scanjobs.py` is the canonical example of a heavily
  `PROGRAMS`-driven script.

## Entry point boilerplate

```python
if __name__ == "__main__":
    import logging
    from shub_workflow.utils import get_kumo_loglevel

    logging.basicConfig(format="%(asctime)s %(name)s [%(levelname)s]: %(message)s", level=get_kumo_loglevel())
    script = MyScript()
    script.run()          # ... or  asyncio.run(script.run())  for a BaseLoopScriptAsyncMixin script
```
Pliki w paczce (6)
  • SKILL.md 6 KB
  • examples/async_loop_script.py 2 KB
  • examples/loop_script.py 2 KB
  • examples/plain_script.py 2 KB
  • examples/project_base_mixin.py 2 KB
  • references/api-cheatsheet.md 5 KB

Wersje

W bazie nie ma innych skilli o nazwie shub-workflow-scripts.

Migawka bazy z 13.11.2026 · wersja paczki d6213760a9c8 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100shub-workflow-crawl-managers
    Buduje lub poprawia menedżera crawlów shub-workflow, który planuje zadania spiderów na Scrapy Cloud i reaguje na ich wyniki.
  • 100shub-workflow-graph-managers
    Buduje lub naprawia menedżera grafu shub-workflow, który uruchamia zależne zadania spiderów i skryptów na Scrapy Cloud jako DAG.
  • 100scrapy
    Pomaga pisać i debugować projekty Scrapy, czyli spidery, oraz uruchamiać i sprawdzać przebiegi crawli.
  • 100devartslab-scrapy-resources
    Pomaga budować, refaktoryzować i debugować pająki Scrapy w projektach city-scrapers.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.