source-connectors
/source-connectors
Projektuje i sprawdza konektory, które trwale importują treści z zewnętrznych usług do repozytorium kontekstu i wznawiają import po awarii.
- Autor
- ctxpipe-ai ↗
- Repozytorium
- ctxpipe ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 27.10.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 4
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 1
Brak sygnałów. Repozytorium zmienione 27.10.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
Co robi i kiedy
Projektuje i sprawdza konektory, które trwale importują treści z zewnętrznych usług do repozytorium kontekstu i wznawiają import po awarii.
Oryginalny opis (EN)
Source connectors. Use when designing, implementing, or reviewing an integration or attachment/media path that durably imports external provider content into a ctxpipe context repository, when provider requests can multiply per related record, or when a full import must resume after a crash.
Jak działa
- 1. Classify the job
- 2. Thin control plane
- 3. Pipe files into git
- 4. OAuth, webhooks, and the data boundary
- 5. Event and setup lifecycle
- 6. Provider reads scale with the data
- 7. Durable steps
- 8. Implement
- 9. Record the decision
Podgląd SKILL.md
---
name: source-connectors
description: Source connectors. Use when designing, implementing, or reviewing an integration or attachment/media path that durably imports external provider content into a ctxpipe context repository, when provider requests can multiply per related record, or when a full import must resume after a crash.
---
# Source connectors
This skill governs connector content for both new and existing connectors.
Control-plane architecture is not retrofitted: maintain Confluence's legacy
tables and Slack's thin intent-capture lifecycle unless that architecture is the
assigned change.
A **git-native** source connector authorises the provider on **this** deployment, writes selected content as files into a **context repository**, then `claimAndRunRepositoryIngestionChild` (in-workflow) / `enqueueRepositoryIngestionWorkflow` (HTTP) indexes that repo. Connector syncs recover an uncheckpointed Git write via `runConnectorRepositoryIngestionWorkflow`. Same code for hosted and self-host. The provider app and webhook endpoint terminate on this deployment; credentials are deployment-shared or connection-specific according to the provider’s tenant-isolation model.
The store is **git**. Rich operations (open a config PR, `commitFiles`) are implemented today only for **GitHub**. Design against git paths; call the GitHub App for those operations. Do not invent a second git host’s PR API unless you are implementing it.
For new connectors, inherit Linear/Notion’s thin control plane (`connections.config` + `<slug>/config.yaml` via PR). Slack shows **intent capture** as a content kind, not a licence to skip config-in-git. Consult Confluence only when maintaining Confluence.
Canonical decisions: [ADR-018](../../../.ai/memory/decisions/ADR-018-unified-connections-table.md), [ADR-022](../../../.ai/memory/decisions/ADR-022-linear-connector-git-native-mirror.md), [ADR-023](../../../.ai/memory/decisions/ADR-023-notion-connector-git-native-mirror.md), [ADR-031](../../../.ai/memory/decisions/ADR-031-github-pr-scoped-mirror.md), [ADR-034](../../../.ai/memory/decisions/ADR-034-pagerduty-connector-git-native-mirror.md). Slack intent-capture (existing, not a template for skipping config): [ADR-025 on PR #267](https://github.com/ctxpipe-ai/ctxpipe/blob/slack-connector/.ai/memory/decisions/ADR-025-slack-connector-git-native-mirror.md). Self-host Atlassian exception: [ADR-019](../../../.ai/memory/decisions/ADR-019-confluence-forge-self-host-and-per-org-atlassian-3lo.md).
## 1. Classify the job
Read [connector kinds](references/kinds.md), then name the **kind** before any schema or UI:
| Kind | Job | Config SoT | Content write | Anchor |
|------|-----|------------|---------------|--------|
| **Native git** | Source already is git | GitHub picker / repo rows | none (ingest the repo) | GitHub connector |
| **Scoped mirror** | Curated spaces (docs, issues) | `<slug>/config.yaml` via config PR | full reconcile then webhook entity sync to the target branch | Linear, Notion |
| **Intent capture** | User points at a thread/item | `<slug>/config.yaml` via config PR | snapshot commit to the target branch | Slack content path (PR #267); new capture still has config-in-git |
MCP is not a source connector. Query-time provider MCP can complement capture; it does not replace git-native durability.
**Done when:** the kind is named, the anchor is named, kinds.md was read, and a config PR is in the plan unless the kind is native git.
## 2. Thin control plane
One `connections` row (`con_*`, `type = <slug>`). **`connections.config` jsonb** holds identity, encrypted secrets, and repo binding (`repositoryId`, `branch`, `enabled`, setup phase / pending config PR metadata). Read/write config only through a colocated Zod schema in `apps/backend/src/lib/connection-config.ts` (lesson: `connections.config` JSONB).
Keep in git or OpenWorkflow, not Postgres:
- draft or live connector config / scope → `<slug>/config.yaml` in the context repo
- dirty-entity / job queues → OpenWorkflow
- binding (`repositoryId` / branch / enabled) → `connections.config`
**No connector-specific tables.** Confluence’s `confluence_spaces` / `confluence_sync_targets` stay as a frozen legacy path. `repositories` is a product entity, not a connector table. List APIs return metadata only — no `config`. Prefer `connectionId` / list-by-org / resolve via `repositoryId`.
Deployment-shared webhook signing secrets are env (same role as `LINEAR_WEBHOOK_SECRET`). Provider-issued per-connection secrets belong encrypted on the row.
Encrypt tokens with `encryptConnectionSecret` (`*Enc` fields). Never log or return decrypted secrets.
**Done when:** the stored shape is a Zod schema on `connections.config`, uniqueness (if any) is a targeted partial index, and the diff adds no connector table.
## 3. Pipe files into git
Git is the durability and audit store.
For images, files, attachments, and external embeds, read
[connector assets](references/assets.md) before designing paths or download
behaviour.
- **Config** lives in the repo as `<slug>/config.yaml`. Create and change it with a **pull request** (today: GitHub). Draft = yaml on the PR branch; live = yaml on the configured target branch after merge.
- **Content** (mirrors, entity updates, captures) may commit **directly to that target branch**. Content is not a second review PR.
Write under a managed root `<slug>/` in the bound context repository (often `ctxpipe-context`).
- **Plain text first.** Markdown for documents, issues, threads, comments. YAML for `config.yaml`. CSV only as a tabular companion beside canonical row Markdown (Notion databases).
- **Deterministic conversion.** Convert provider-native blocks/markup into readable Markdown (or another agreed plain-text form), preserving all user-authored text, ordering, headings, lists, authors, timestamps, links, and code. No LLM rewrite, no API-JSON dumps, no retained HTML unless HTML is itself the source payload.
- **Stable paths.** Include the provider id so renames do not duplicate. Match a nearby anchor: Linear uses flat `linear/issues/<slug>--<id>.md`; Notion pages use `<slug>--<id>/index.md`; Slack thread directories use `thread.md`.
- **Images and file attachments are files.** Copy provider-declared file attachments and explicit embedded external media through the shared asset boundary, commit them beside their owning content, and rewrite links to relative paths. Ordinary hyperlinks and link-only attachment records stay links and are never crawled. Never persist private or expiring URLs as file sources. Unsafe, unreadable, or oversized blobs leave a permalink/text stub and do not fail the whole write.
- **Provenance.** YAML frontmatter: source, stable ids, canonical URL, timestamps. Connector uninstall does not purge git.
- **Graph contract (ADR-033).** A connector ships its frontmatter contract. Hydrate reads the mirrored Markdown into knowledge units (`domain/workspaces/mirror-units.ts`), never with the instruction LLM, and builds cross-tool references only through `domain/codeIngestion/referenceResolver.ts`. Add a render → hydrate round-trip test. Put provider URLs the content links to (PRs, issues, threads) in frontmatter or body so the resolver can join them.
Writes go through `commitFiles` in `installation-write-client.ts` (`encoding: "base64"` for binaries). Checkpoint the selected repository ID and branch in workflow state before the write; never reload a mutable binding between a checkpointed write and its ingestion hand-off. After the hand-off resolves the repository's GitHub connection, carry that connection through the ingestion child and failure recovery rather than reloading it mid-run. After every successful connector sync, invoke `runConnectorRepositoryIngestionWorkflow` inside the parent workflow (it uses `claimAndRunRepositoryIngestionChild` so the parent sleeps and frees its concurrency slot). HTTP and webhook callers use `enqueueRepositoryIngestionWorkflow`. Do not skip the hand-off for a Git no-op: the shared helper resolves the current branch tip and compares it with `lastIngestedHash` so a replay can recover a write whose prior step result was lost without regressing to an older commit. Do not bypass the repository single-flight claim: overlapping hand-offs persist one coalesced follow-up marker, and both successful and failed ingestion paths drain it through retryable workflow steps. Clear that marker only with a compare-and-set against the terminal ingested hash; an unconditional clear can erase a newer overlap.
**Done when:** a sample tree is specified (config yaml, content paths, attachment files); config is a PR; content commits to the target branch; the mirrored page is readable without a live provider API.
## 4. OAuth, webhooks, and the data boundary
Hosted and self-host run the **same** routes. Difference: who creates the provider app and supplies secrets.
- **Deployment-shared** app credentials live in env (`LINEAR_*`, `NOTION_*`, `SLACK_*`, `PAGERDUTY_*`). **Connection-specific** credentials (and provider-issued secrets) live encrypted in `connections.config`. Linear, Notion, and PagerDuty are Atlassian-shaped: self-host may store the OAuth app (and webhook secret) on that connection and resolve **row, then env**; hosted still falls back to env. Document which model this provider requires. Backend **and** worker resolve the same way when the worker refreshes tokens.
- Callbacks and Event URLs are on **this** deployment’s `AUTH_BASE_URL` (e.g. `/api/v1/integrations/<slug>/callback`, `/api/v1/webhook/<slug>`). Derive redirect URI from `AUTH_BASE_URL`; optional `*_REDIRECT_URI` only when the public URL differs.
- One Event URL per deployment app; route to a connection by workspace/team id in the payload. Per-connection webhook paths only when the provider app itself is per-connection (GitHub App on the connection row).
- Per-org provider apps only when sharing one app would cross tenants (Atlassian 3LO on the Forge row — ADR-019). Still processed on this deployment.
- OAuth start/callback: expiring signed `state` binds `userId` + `orgId` (and `connectionId` when needed). PKCE when the provider’s code flow supports it (Atlassian, PagerDuty). Store the PKCE verifier in an HttpOnly cookie (or short-lived server record), never in readable `state`. Popup completion: tiny same-origin HTML relay + `localStorage` (lesson: Connector OAuth popup completion). Secrets never go in `state`.
- Signed webhooks: verify raw body + timestamp; ACK after OpenWorkflow enqueue. Skip events unless setup is live (no dirty-entity buffer). Failed enqueue → 5xx so the provider retries.
**Self-host data boundary (hard):** customer tokens, webhooks, and source bytes stay on the customer’s deployment. No ctxpipe-SaaS proxy, relay, gateway, webhook forwarder, or “use our hosted OAuth app from your self-host”. Hosted ctxpipe is one deployment; a self-host install is another.
**Done when:** callback + Event URL are documented as `{AUTH_BASE_URL}/…`, each secret is classified env vs `connections.config`, routing key is named, and the design has no SaaS hop for self-host traffic.
## 5. Event and setup lifecycle
Phases: `draft` → `awaiting_merge` → `initial_sync` / first content → `live` (plus `config_failed` / `sync_failed`). Config workflow opens the yaml PR. GitHub push on the target branch after merge starts content (full reconcile, or capture becomes eligible). Live webhooks run entity-sync or capture jobs that commit content to the target branch.
If the binding is `draft` after rebind or context-repo recreate and `<slug>/config.yaml` on the target already matches the selected scope, **do not no-op**. Skip the config PR, start `initial_sync`, enqueue content sync, and return `configPrEnqueued: false`. UI must honour that flag (no phantom “creating PR”). A matching **live** scope stays a no-op. Slack has no yaml PR — leave it. Confluence is heavier (Postgres spaces) but follows the same rule; an unchanged `confluence/config.yaml` starts content sync instead of flipping to `live`.
**Intent capture** still waits on live config. Slack’s shipped path derives `live` from binding with no yaml — leave it; new capture connectors use the config PR.
Rebind (repository/branch change) resets lifecycle. `app_uninstalled` / token revoke sets `status: revoked` and disables binding; git stays. Deleting the bound repository clears binding fields.
Durable work is OpenWorkflow in `apps/backend/src/openworkflow/workflows/` (directory discovery). Webhook handlers stay thin.
**Done when:** phases, the config-merge trigger, and the live content path are named, matching the kind.
## 6. Provider reads scale with the data
Each provider call fetches one page of entities, or the one webhook entity being applied, with the fields the mirror renders inlined in that same call. Use the provider's bulk or connection fields for comments, authors, labels, and parent context. A longer list is another page of that same query. Setup discovery of the selectable catalogue is one query; a further call is only a later page of it.
Request count grows linearly with the amount of data. One page of N entities is one request whether each entity has one comment or fifty. A request per related record (per comment, per author, per parent) is exponential in the shape of the data and exhausts the provider quota during a large import.
When the provider returns a rate-limit reset, wait and retry that same request inside the read. The page step then succeeds, and pages already stored stay stored.
**Done when:** a test counts provider calls. One page of entities, including the relations the files render, is one call. A second page is one more call of the same shape. Adding a related record to an entity does not add a call. Discovery of the catalogue is one call unless a list has another page.
## 7. Durable steps
A durable step is one `step.run({ name })` call in the workflow function. OpenWorkflow stores that function's return value in Postgres. On a crash the workflow function starts from the top, and each finished name returns the stored value instead of running again.
Call `step.run` from the workflow for each page. A loop that lives inside one step is one step: a crash before it returns drops everything that step fetched.
Initial sync of a scoped mirror:
1. One step loads the connection, repository, and `<slug>/config.yaml`.
2. One step per provider page. Name it with the scope id and the page index (`team-<id>-issues-0`). Store the provider cursor in the result so the next step knows where to start. Return the rendered text files for that page and `nextAfter`. A nested connection that still has another page is its own step (`team-<id>-issues-0-issue-comments-<issueId>-0`), not another request inside the parent page step.
3. One commit step joins those stored files and writes one git commit. Download attachment bytes in that commit step. Page results stay text.
4. Ingest once, then finalize the setup phase.
A webhook applies one entity in one step and one commit.
Crash during `team-<id>-issues-1`, after `issues-0` is stored: `issues-0` returns the stored markdown and the provider is not called for it. `issues-1` runs again. The commit step runs once, after every page has a stored result.
Crash after the git host accepts the commit but before that step is stored: the page steps are already stored, so the provider is not called. The commit step runs again, sees the files already on the branch (`omitUnchanged` against the tree), and writes nothing. The branch still has one commit.
Delete stale managed paths in that same commit, and only when every page reported zero failures.
**Done when:** a workflow test records step names. Two pages produce two step names. A second run with the first page already stored does not call the provider for that page. `commitFiles` is called once.
## 8. Implement
Follow [references/file-map.md](references/file-map.md). Inspect the named anchors and reproduce their **responsibilities** for this kind; mark inapplicable surfaces N/A. UI chrome: [product-ui](../product-ui/SKILL.md) + connector-wizard lesson (Linear/Notion/Slack share setup chrome; list rows are `ConnectorListItem`, not a stepper).
If AWS CDK / deploy images change, add a changeset for `@ctxpipe/aws-cdk`.
**Done when:** every file-map row for this kind is implemented or N/A; converter fixtures, asset-boundary safety and reconciliation tests (`pnpm --filter @ctxpipe/backend test:connector-assets`, with the new slug added to `vitest.connector-assets.config.ts`), binary/`base64` commits (including an attachment), raw-body signature tests, workflow discovery, empty optional env parsing, and focused UI tests pass. CDK edits: `pnpm --filter @ctxpipe/aws-cdk test`.
## 9. Record the decision
New kind → [capture-adr](../capture-adr/SKILL.md) and point this skill at it. Confirmed convention → [capture-lesson](../capture-lesson/SKILL.md). New term → [capture-glossary](../capture-glossary/SKILL.md).
Product guide under `apps/docs/content/docs/(guide)/connections/source-connectors/`. Self-host operator page under `apps/docs/content/docs/self-hosting/` (OAuth app, callback, Event URL, env, Compose / Railway / CDK).
**Done when:** docs describe both hosted and self-host, and an ADR exists if the kind is new.
Pliki w paczce (4)
- SKILL.md 17 KB
- references/assets.md 5 KB
- references/file-map.md 7 KB
- references/kinds.md 4 KB
Wersje
W bazie nie ma innych skilli o nazwie source-connectors.
Migawka bazy z 13.11.2026 · wersja paczki 0038c04caf8a (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 100wp-connectors-api
Rejestruje i przegląda integracje WordPress 7.1 Connectors API dla zewnętrznych usług, zwłaszcza dostawców AI: rejestr konektorów, tryby uwierzytelniania, priorytet źródeł poświadczeń i maskowanie kluczy. - 100camunda-connectors-development
Buduje własne konektory Camunda 8 jako szablon JSON na konektorze protokołowym albo w Javie z Connectors SDK, dla własnych API, niestandardowego uwierzytelniania i wejściowych webhooków. - 90custom-connectors
Projektuje konektory Power Platform na podstawie opisu OpenAPI, dobierając uwierzytelnianie i wzorzec wdrożenia. - 100lucidchart-reference-architecture
Projektuje architekturę integracji Lucid opartą na kontraktach, obejmującą REST, Standard Import, rozszerzenia edytora i opcjonalne konektory danych. - 85base44-platform-connectors
Łączy aplikacje Base44 z usługami zewnętrznymi przez OAuth, np. Google, Slack czy GitHub, i odczytuje token na serwerze. - 90integrate
Buduje lub zmienia integrację z systemem klienta, z jawnym mapowaniem danych, uprawnieniami, ponowieniami i uzgadnianiem; obejmuje konektory, importy i zapis zwrotny.