Skip to content
Skills Library
Programming · Automation

text-processing

/text-processing

Choose grep, sed, awk, or jq by job, prefer structured formats, and know when a one-liner should become a script. Use when slicing logs, transforming files, or wrangling command output.

Repository
AI-SKILLS ↗
Licence
MIT
Last repository change
05.09.2026

Assessment and security

90/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content25 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 05.09.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Choose grep, sed, awk, or jq by job, prefer structured formats, and know when a one-liner should become a script. Use when slicing logs, transforming files, or wrangling command output.

Original author description

Choose grep, sed, awk, or jq by job, prefer structured formats, and know when a one-liner should become a script. Use when slicing logs, transforming files, or wrangling command output.

How it works

  1. Method
  2. Boundaries
Original SKILL.md headings · 72 lines, 516 words
SKILL.md preview
---
name: text-processing
description: Choose grep, sed, awk, or jq by job, prefer structured formats, and know when a one-liner should become a script. Use when slicing logs, transforming files, or wrangling command output.
---

# Text processing

The classic tools each do one job at expert level: grep selects lines,
sed edits streams, awk computes over fields, jq queries JSON. Fluency
means picking the right one in five seconds and refusing to parse what
was never meant to be parsed.

## Method

1. **Reach for structured output first.** Modern tools speak JSON
   (`--json`, `-o json`, `--porcelain`): pipe that into `jq`
   instead of regexing human-formatted text that changes between
   versions (the powershell-essentials object principle, POSIX
   edition). Parsing `ls`, human `df`, or colorized output is a
   bug with a delay timer (see bash-robustness boundaries).
2. **Map job to tool.** Find lines: `grep -E` (add `-r` for
   trees, ripgrep for speed and sane defaults). Edit a stream:
   `sed` for substitutions and line surgery
   (`s///`, `/from/,/to/d`). Fields and arithmetic: `awk`
   (`-F,` splits, `$3 > 100 { sum += $2 } END { print sum }`):
   the point where cut+grep+sort pipelines collapse into one
   readable program. JSON: `jq` for select/reshape
   (`.items[] | select(.state=="failed") | .name`); yq for YAML.
   Sort/unique/count: `sort | uniq -c | sort -rn`, the
   frequency-table idiom that answers half of all log questions
   (see log-analysis).
3. **Compose short pipelines, name long ones.** Three or four
   stages pipe fine; beyond that, or with branching logic, write
   an awk script or move to Python with named functions (see
   bash-robustness step 6): pipelines are write-once
   read-never past a width. Intermediate results worth
   inspecting go through tee or temp files (atomically; see
   script-idempotency) so debugging does not mean rerunning
   everything.
4. **Handle real-world text hazards explicitly.** Filenames with
   spaces/newlines: `-print0`/`-0`/`--null` chains end to end.
   Huge files: stream (awk/sed/grep are O(1) memory) rather than
   slurp; `LC_ALL=C` for byte-wise speed and stable sorting
   (see shell-portability locale note). CRLF from Windows files
   breaks matches invisibly: `dos2unix` or `tr -d '\r'` at
   ingestion. Delimiters inside quoted CSV fields defeat awk's
   `-F,`: real CSV gets a real parser (csvkit, python csv),
   full stop.
5. **Test extractions against edge inputs.** Empty input, one
   line, the weird line (unicode, embedded quotes, missing
   field): pipe crafted samples through before trusting a
   one-liner on production data; for anything recurring, save
   the sample and expected output as a regression check (see
   golden-master in miniature). Regex greed and anchoring
   (`.*` spanning too far, missing `^`) are the classic silent
   wrongness: prefer explicit field logic in awk over heroic
   regex in sed.
6. **Promote recurring one-liners into tools.** The pipeline you
   have pasted three times into chat becomes a script with a
   name, `--help`, and stdin/file input (see cli-ux-design,
   python-cli-tools); one-liners in runbooks become tested
   scripts referenced by the runbook (see runbook-writing).
   History is not a package manager.

## Boundaries

- HTML and XML are not line-oriented; grep/sed on them works
  until the formatting changes: use pup/xmllint/a real parser
  (the same rule that bans regexing JSON when jq exists).
- Binary-adjacent formats (logs with control bytes, mixed
  encodings) need inspection (`file`, `xxd | head`) before text
  tools; garbage-in pipelines produce confident garbage out.
- At gigabytes-to-terabytes scale or with joins across files,
  the answer is a database or DuckDB-class local engine, not a
  taller pipeline (see exploratory-data-analysis instincts).

Versions

The database has no other skills named text-processing.

Database snapshot: 13.11.2026 · package version 593f8e1dbd68 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100sed-awk-grep-text-processing-mastery
    Use this skill when filtering, transforming, and extracting structured insights from system logs and text streams using sed, awk, and grep.
  • 95bash-script-generator
    Creates bash scripts with argument parsing, error handling, logging, and input validation following current standards. Use when creating new bash or shell scripts, .sh files, CLI tools, scripting automation, text processing workflows (grep/awk/sed pipelines), or building…
  • 100jq
    Use when processing JSON or YAML data from the command line. Covers jq for JSON and yq for YAML — filtering, transforming, and extracting data from structured outputs of APIs, config files, and CLI tools. USE FOR: jq, yq, JSON processing, YAML processing, CLI data…
  • 100awk
    Expert GNU awk (gawk) one-liner generation and transformation. Use this skill whenever the user asks to process text fields, extract columns, compute sums or aggregates, filter structured output, reformat CSV/TSV, parse logs, generate reports, or do any record/field processing…
  • 100sed
    Expert GNU sed one-liner generation and transformation. Use this skill whenever the user asks to transform text, edit files in-place, refactor code via CLI, process logs, do bulk find/replace, extract lines, delete patterns, or perform any stream editing task. Trigger on…
  • 100server-application-log-analysis-grep-sed
    Use this skill when extract critical error patterns, trace IDs, and anomalies from raw server logs using CLI tools.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.