Skip to content
Skills Library
DevOps & cloud · Analysis

workflow-triaging

/workflow-triaging

Triage AEM Workflow issues on AEM as a Cloud Service by classifying symptoms, gathering the right logs and metrics, and mapping to runbooks or Splunk searches. Use when the user asks for workflow activity/errors on a Cloud Service host, needs to classify a Jira ticket, or wants to know what to collect for workflow debugging.

Author
adobe ↗
Repository
skills ↗
Licence
Apache-2.0
Last repository change
24.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 24.10.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

Triage AEM Workflow issues on AEM as a Cloud Service by classifying symptoms, gathering the right logs and metrics, and mapping to runbooks or Splunk searches. Use when the user asks for workflow activity/errors on a Cloud Service host, needs to classify a Jira ticket, or wants to know what to collect for workflow debugging.

Original author description

Triage AEM Workflow issues on AEM as a Cloud Service by classifying symptoms, gathering the right logs and metrics, and mapping to runbooks or Splunk searches. Use when the user asks for workflow activity/errors on a Cloud Service host, needs to classify a Jira ticket, or wants to know what to collect for workflow debugging.

How it works

  1. Audience
  2. Variant Scope
  3. Dependencies
  4. When to use this skill
  5. Step 1: Classify symptom (symptomid)
  6. Step 2: Required inputs for triage
  7. Step 3: Log patterns and Splunk (what to search)
  8. Step 4: Developer Console and Sling Job diagnostics
  9. Step 5: Example triage prompts and responses
  10. Step 6: What logs and Developer Console can and cannot answer
Original SKILL.md headings · 183 lines, 2,176 words
SKILL.md preview
---
name: workflow-triaging
description: Triage AEM Workflow issues on AEM as a Cloud Service by classifying symptoms, gathering the right logs and metrics, and mapping to runbooks or Splunk searches. Use when the user asks for workflow activity/errors on a Cloud Service host, needs to classify a Jira ticket, or wants to know what to collect for workflow debugging.
license: Apache-2.0
---

# AEM Workflow Triaging — Cloud Service

Classify workflow issues, determine what logs and data to gather, and map to the correct runbook or log search. Optimized for **production support** on **AEM as a Cloud Service (AEMaaCS)**.

## Audience

AEMaaCS developers and operators (and the IDE LLM acting on their behalf) classifying workflow incidents across one or more environments — using environment ID + time-range + Cloud Manager Logs (or a log aggregator such as Splunk, if you forward AEMaaCS logs there), before drilling into one instance. Use this skill for cross-environment log mining and symptom classification; switch to `workflow-debugging` once the instance and root cause are identified.

## Variant Scope

- AEM as a Cloud Service only.
- **Not for AEM 6.5 LTS / AMS.** If the target is 6.5 LTS, stop and use the 6.5-lts variant of this skill — Splunk index/sourcetype paths, the JMX surface, and several log signatures there do not apply as written on AEMaaCS.
- Log access via **Cloud Manager** → Environments → Logs (download or streaming), or Splunk if logs are indexed there.
- **No JMX on AEMaaCS production.** Workflow counts and queue metrics come from logs, Developer Console status producers, and the Sling Job Console. JMX MBeans exist on the local AEMaaCS SDK (`localhost:4502`) but must not be assumed available on cloud environments.
- **All remediation lands via Git + Cloud Manager pipeline.** There is no Felix Console write access or Package Manager on production AEMaaCS environments.

## Dependencies

- `workflow-debugging` — once a symptom is classified and an environment/instance is identified, route here for the step-by-step runbook and remediation.
- `workflow-debugging/reference.md` — canonical diagnostic tool pointers, log patterns, and external doc links for AEMaaCS.

---

## When to use this skill

- User asks: "Workflow errors on `<env-id>` for the past X hours", "Workflow activity on `<env-id>`", "Why did workflow X fail?", "What should I collect to debug this workflow ticket?"
- User needs: Symptom classification, log patterns to search, Splunk queries, or required inputs for a runbook.
- Context: AEM Cloud Service (environment ID format: `cm-p<programId>-e<environmentId>`).

---

## Step 1: Classify symptom (symptom_id)

Map the user's description to a **symptom_id** and runbook.

| User says / observes | symptom_id | Runbook |
|----------------------|------------|---------|
| Workflow not moving to next step; stuck in Running | workflow_stuck_not_progressing | runbook-workflow-stuck.md |
| Task should be in Inbox but is not visible | task_not_in_inbox | runbook-task-not-in-inbox.md |
| Workflow should start automatically but no instance created | workflow_not_starting_launcher | runbook-launcher-not-starting.md |
| Workflow in Failed state or step shows error | workflow_fails_or_shows_error | runbook-workflow-fails-or-shows-error.md |
| Step failed after retries; failure item in Inbox | step_failed_retries_exhausted | runbook-failed-work-items.md |
| Instance Running but no current work item (inconsistent) | stale_workflow_no_work_item | runbook-stale-workflows.md |
| Too many instances; slow queries; disk/repo bloat | repository_bloat_too_many_instances | runbook-purge-and-cleanup.md |
| User cannot see work item or complete/delegate/return | user_cannot_see_or_complete_item | runbook-inbox-and-permissions.md |
| Cannot delete workflow model (running instances) | cannot_delete_model | runbook-model-delete-and-update.md |
| Jobs queued a long time; slow completion; queue depth high | slow_throughput_queue_backlog | runbook-job-throughput-and-concurrency.md |
| Auto-advance / timeout jobs not firing; participant step stuck past its configured timeout | workflow_auto_advance_failure | runbook-job-throughput-and-concurrency.md |
| New or changed workflow not starting or step not executing | workflow_setup_validation | runbook-validate-workflow-setup.md |

> **WorkItem vs. TaskManager task — do not confuse these.** Most workflow Inbox items are workflow work items (`WorkItem`), created by Participant steps and managed by the workflow engine; they are stored under `/var/workflow/instances`, not in TaskManager. TaskManager (`/var/taskmanagement/tasks`) only holds tasks created explicitly via the Task API — used by Projects, Assets tasks, and custom integrations. For `task_not_in_inbox` and `user_cannot_see_or_complete_item` symptoms on a workflow: investigate the Participant step assignee configuration, Inbox filters, and workflow permissions — not TaskManager storage. Diagnosing the wrong backend wastes significant time.

---

## Step 2: Required inputs for triage

Before suggesting a runbook or Splunk search, try to obtain:

| Input | Purpose |
|-------|---------|
| **Environment ID** | AEMaaCS format: `cm-p<programId>-e<environmentId>` (e.g. `cm-p163724-e1759416`). |
| **Time range** | e.g. "past 4 hours", "past 10 hours" – for log/Splunk scope. |
| **Workflow model or step name** | e.g. "Dynamic Media Reupload", "DAM Update Asset", "testmodel". |
| **Instance ID** (if known) | From Workflow Console URL or payload; ties logs to one instance. |
| **Payload path** (if known) | e.g. `/content/dam/...`; for path-related errors. |
| **Log source** | Cloud Manager log download, log streaming, or Splunk index/sourcetype. |

If the user only provides environment ID + time, respond with the **generic** workflow error searches and note that narrowing by model or instance ID will improve accuracy.

---

## Step 3: Log patterns and Splunk (what to search)

Logs on AEMaaCS are accessed via **Cloud Manager** → Environments → Logs (download or streaming). The primary file is `error.log`. When logs are indexed in **Splunk** (or any log aggregator), use these patterns.

| Scenario | Primary log pattern(s) | Note |
|----------|------------------------|------|
| Step failed | `Error executing workflow step` | Add instance ID or model name to narrow. |
| Process not found | `getProcess for '*' failed` | Extract process name; check OSGi Components for `process.label` mismatch. |
| Stuck at Process step | Same as step failed + `getProcess` | Combine with payload path. |
| Stale workflow | `Cannot archive workitem` | Correlate time with instance ID. |
| Lock / throughput | `refreshing the session since we had to wait for a lock` | Reduce effective concurrency — on AEMaaCS, job queue settings are not directly tunable at runtime; address via code changes: split workflows, offload heavy steps asynchronously, or externalize processing. Raising concurrency makes lock contention worse. |
| Permission | `Terminate failed` / `Resume failed` / `Suspend failed` + verifyAccess | Or `AccessControlException`. Check `enforceWorkflowInitiatorPermissions`. |
| Payload path | `PathNotFoundException` (workflow/payload) | Payload deleted, or launcher config path missing. |
| Launcher not starting | `Error adding launcher config` / `Error retrieving launcher config entries` | Path: `/conf/global/settings/workflow/launcher/config`. |
| Purge failure | `Workflow purge '*' :` | Filter by repository exception / invalid state. |
| Transient workflow retries exhausted | `retrys exceeded - remove isTransient` | Process step kept throwing after `cq.workflow.job.retry` retries. Fix step code; instance persisted for admin handling. |
| Thread pool full | `RejectedExecutionException` | `default` pool saturated with `blockPolicy=ABORT`; timeout/auto-advance jobs dropped. |
| Operation on finished instance | `Workflow is already finished` | Check logic that calls terminate/resume on a completed or aborted instance. |

**Example Splunk searches (replace index/sourcetype/field names for your environment):**

- All workflow step errors (last 24h):
  `index=aem sourcetype=aem:error "Error executing workflow step" | table _time host message | sort - _time`
- Process not registered:
  `index=aem "getProcess for" "failed" | table _time host message`
- By workflow model or instance:
  `index=aem ("Error executing workflow step" OR WorkflowException) (message=*<modelName>* OR message=*<instanceId>*) | sort - _time`
- Lock contention:
  `index=aem "refreshing the session since we had to wait for a lock" | table _time host message`
- Thread pool exhaustion (auto-advance impact):
  `index=aem "RejectedExecutionException" | table _time host message`

> **Note:** Indexes and sourcetypes vary by organization; adapt queries accordingly.

---

## Step 4: Developer Console and Sling Job diagnostics

On AEMaaCS production, use the **Developer Console** status producers and the **Sling Jobs page** for metrics not available from logs alone. JMX is not available on production AEMaaCS; these are the equivalents.

| What to check | Tool / URL | Purpose |
|---------------|-----------|---------|
| Workflow queue depth and failed jobs | Sling Jobs page: `/system/console/slingevent` | `Queued Jobs > 0` with `Active Jobs = 0` → jobs not being picked up. `Failed Jobs` count per topic. |
| Workflow job topic statistics | Sling Jobs page: topic `com/adobe/granite/workflow/job/var/workflow/models/<modelName>` | High `Failed Jobs` / low `Finished Jobs` → process step throwing exceptions. |
| Sling `default` thread pool saturation | Thread Pools page: `/system/console/status-slingthreadpools` | `active count = max pool size` AND `blockPolicy = ABORT` → new scheduled tasks (including workflow timeout detection) are silently rejected. |
| Thread stack trace | Thread Dump: `/system/console/status-jstack-threaddump` | All `sling-default-*` threads stuck on same stack → blocking culprit for auto-advance failure. |
| Sling Scheduler status | Scheduler page: `/system/console/status-slingscheduler` | Confirm `ApacheSlingdefault` uses `ThreadPool: default`. Note: `com/adobe/granite/workflow/timeout/job` is a Sling Job topic, not visible here — check the Sling Jobs page instead. |
| OSGi bundle / process registration | OSGi Components: `/system/console/components` | Confirm WorkflowProcess component with matching `process.label` is Active. |
| Instance state | Workflow Console: `/libs/cq/workflow/admin/console/content/instances.html` | Instance status, current work item, history. |

**Developer Console access:** AEM Cloud Service → Developer Console. Status producers (thread dumps, Sling Jobs, thread pools) are read-only on all tiers. On the local AEMaaCS SDK (`localhost:4502/system/console/jmx`) JMX MBeans are also available — use them for local development only; do not document JMX steps for production.

**Safety:** Never recommend remediation operations that bypass Git + Cloud Manager pipeline (e.g. Felix Console config changes) on cloud environments. All config changes go in `ui.config` and deploy via pipeline.

---

## Step 5: Example triage prompts and responses

| User prompt | Triage response |
|-------------|-----------------|
| "Workflow errors on `<env-id>` for the past X hours" | Classify as `workflow_fails_or_shows_error` / `step_failed_retries_exhausted`. Download or stream `error.log` from Cloud Manager; search for `Error executing workflow step`, `Error processing workflow job`, `getProcess for … failed`. Check Sling Jobs page for failed job count per topic. Route to `runbook-workflow-fails-or-shows-error`. |
| "Workflow activity on `<env-id>` for the past X hours" | Clarify: counts (started/completed/failed) or list of errors? For errors, use log searches above. For counts on AEMaaCS, use Cloud Manager log aggregation or the Sling Jobs page — no JMX. |
| "Why did `<workflow-or-step>` fail? Show failure details." | Need: environment ID, time range, instance ID if known. Search `error.log` for `Error executing workflow step` + model/step name or instance ID; return exception type, message, and stack. Route to `runbook-workflow-fails-or-shows-error`. |
| "Task not in Inbox" | `symptom_id: task_not_in_inbox`. Route to `runbook-task-not-in-inbox`. Gather: instance ID, assignee, whether user is initiator/assignee. Check Inbox filters and `enforceWorkitemAssigneePermissions` via Developer Console OSGi config view. |
| "Workflow not starting" | `symptom_id: workflow_not_starting_launcher`. Route to `runbook-launcher-not-starting`. Gather: model name, payload path, launcher config path; search logs for launcher errors. |
| "Workflow stuck / not progressing" | `symptom_id: workflow_stuck_not_progressing`. Route to `runbook-workflow-stuck`. First: does the instance have a current work item? If no → stale. If yes, follow decision tree by step type. |
| "Auto-advance / timeout jobs not firing" | `symptom_id: workflow_auto_advance_failure`. Route to `runbook-job-throughput-and-concurrency`. Check Developer Console thread dump for `sling-default-*` thread saturation; check Sling Jobs page for `com/adobe/granite/workflow/timeout/job` topic; search `error.log` for `RejectedExecutionException`. |

---

## Step 6: What logs and Developer Console can and cannot answer

**Can answer (with AEM workflow logs from Cloud Manager + Developer Console on AEMaaCS):**

- Step failures: exception type, message, stack (by environment, time, model, step).
- Process not registered: which `process.label` is missing (logs + Developer Console OSGi Components).
- Stuck: step errors, `getProcess` failures, lock wait, payload/path errors.
- Stale: `Cannot archive workitem` and transition errors in logs.
- Queue metrics: Sling Jobs page (`/system/console/slingevent`) → queued, active, failed per topic.
- Thread pool saturation: Thread Pools page (`/system/console/status-slingthreadpools`).
- Throughput: lock wait, session refresh, JobHandler volume in logs.
- Permission: Terminate/Resume/Suspend failed (`verifyAccess`), `AccessControlException` in logs.
- Payload/launcher: `PathNotFoundException`, launcher config errors in logs.
- Purge: `Workflow purge …` repository exception or invalid state in logs.

**Cannot answer directly (AEMaaCS limitations vs 6.5 LTS):**

| What is needed | AEMaaCS alternative |
|----------------|---------------------|
| JMX `countStaleWorkflows` | Deploy a custom `StaleWorkflowServlet` (see `workflow-debugging` Step 6); call with `?dryRun=true`. |
| JMX `countRunningWorkflows` | Workflow Console UI, or a custom count servlet. |
| JMX `retryFailedWorkItems` | Inbox UI → Retry (single); or a custom bulk-retry servlet (see `workflow-debugging` Step 6). |
| JMX `purgeCompleted` | `com.adobe.granite.workflow.purge.Scheduler-<alias>.cfg.json` deployed via pipeline. |
| JMX `restartStaleWorkflows` | Custom `StaleWorkflowServlet` with `POST ...?dryRun=false`. |
| Config status ZIP | Developer Console status producers; or request from Adobe Support. |
| Console state (current work item) | Workflow Console UI (`/libs/cq/workflow/admin/...`) or custom API. |
| Runtime process step code behavior | Requires code review + log correlation. |
| Pod restart | Adobe Support ticket — Cloud Manager does not expose a customer-facing restart action. |

> **Custom servlets are privileged — secure them before deploying.** The `StaleWorkflowServlet` and any bulk-retry/count servlet above restart, replay, or enumerate workflows. Treat them as admin-only: authorize the caller against an operations group, use a service user (never an admin session), default to `dryRun=true`, scope by model, and keep the endpoint off publish. Follow the secure-write/deploy checklist in `workflow-debugging` Step 6 before shipping one.

Always pair log-based triage with Developer Console diagnostics and the appropriate runbook for actions (Inbox Retry, Purge Scheduler config, Cloud Manager pipeline deploy).

---

## References (in repo)

- **Diagnostic tool pointers and log patterns:** [`../workflow-debugging/reference.md`](../workflow-debugging/reference.md)
- **Step-by-step runbook (per symptom):** [`../workflow-debugging/SKILL.md`](../workflow-debugging/SKILL.md)
- **Cloud Service guardrails (paths, service users, OSGi annotations):** [`../workflow-development/references/workflow-foundation/cloud-service-guardrails.md`](../workflow-development/references/workflow-foundation/cloud-service-guardrails.md)

Versions

RepositoryScoreLicence
This page · adobe/skills100Apache-2.0
adobe/skills100Apache-2.0

Database snapshot: 13.11.2026 · package version c8f122e90854 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100workflow-model-design
    Design and create AEM Workflow models on AEM as a Cloud Service. Use when creating workflow models via the Workflow Model Editor or content package XML, defining step types (PROCESS, PARTICIPANT, DYNAMIC_PARTICIPANT, OR_SPLIT, AND_SPLIT), configuring step properties, declaring…
  • 100workflow-debugging
    Debug AEM Workflow issues on AEM as a Cloud Service — stuck workflows, failed steps, missing Inbox tasks, launcher failures, stale instances, thread pool exhaustion, queue backlogs, purge failures, and permissions errors. Use when the user reports workflow problems on Cloud…
  • 100splunk-dashboard
    Generate Splunk Observability Cloud dashboard Terraform from an existing otel-audit report. Reads .observe/otel-audit.json, groups metrics into dashboard panels, and outputs ready-to-apply HCL (signalfx_dashboard_group + signalfx_dashboard + per-panel signalfx_*_chart resources)…
  • 100aem-doctor
    Check AEM project infrastructure health — component definitions, OSGi configs, dispatcher rules, content structure. Use when AEM config seems broken or after infrastructure changes.
  • 100connect-splunk-observability-cloud
    Open the already-running local Splunk Observability Studio Cloud tab so a user can connect an existing or newly ready Splunk Observability Cloud organization by entering its URL and access token outside agent context. Use when a user asks to connect, configure, or provide…
  • 100splunk
    Covers Splunk Enterprise and Splunk Cloud as a log/metrics observability platform: SPL (Search Processing Language), distributed architecture, data ingestion, dashboards, alerting, ITSI, and license cost optimization. Use for \"Splunk\", \"splunk\", \"SPL\", \"Search Processing…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.