text-processing
/text-processing
Choose grep, sed, awk, or jq by job, prefer structured formats, and know when a one-liner should become a script. Use when slicing logs, transforming files, or wrangling command output.
- Author
- Amey-Thakur ↗
- Repository
- AI-SKILLS ↗
- Licence
- MIT
- Last repository change
- 05.09.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 05.09.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Choose grep, sed, awk, or jq by job, prefer structured formats, and know when a one-liner should become a script. Use when slicing logs, transforming files, or wrangling command output.
Original author description
Choose grep, sed, awk, or jq by job, prefer structured formats, and know when a one-liner should become a script. Use when slicing logs, transforming files, or wrangling command output.
How it works
- Method
- Boundaries
SKILL.md preview
---
name: text-processing
description: Choose grep, sed, awk, or jq by job, prefer structured formats, and know when a one-liner should become a script. Use when slicing logs, transforming files, or wrangling command output.
---
# Text processing
The classic tools each do one job at expert level: grep selects lines,
sed edits streams, awk computes over fields, jq queries JSON. Fluency
means picking the right one in five seconds and refusing to parse what
was never meant to be parsed.
## Method
1. **Reach for structured output first.** Modern tools speak JSON
(`--json`, `-o json`, `--porcelain`): pipe that into `jq`
instead of regexing human-formatted text that changes between
versions (the powershell-essentials object principle, POSIX
edition). Parsing `ls`, human `df`, or colorized output is a
bug with a delay timer (see bash-robustness boundaries).
2. **Map job to tool.** Find lines: `grep -E` (add `-r` for
trees, ripgrep for speed and sane defaults). Edit a stream:
`sed` for substitutions and line surgery
(`s///`, `/from/,/to/d`). Fields and arithmetic: `awk`
(`-F,` splits, `$3 > 100 { sum += $2 } END { print sum }`):
the point where cut+grep+sort pipelines collapse into one
readable program. JSON: `jq` for select/reshape
(`.items[] | select(.state=="failed") | .name`); yq for YAML.
Sort/unique/count: `sort | uniq -c | sort -rn`, the
frequency-table idiom that answers half of all log questions
(see log-analysis).
3. **Compose short pipelines, name long ones.** Three or four
stages pipe fine; beyond that, or with branching logic, write
an awk script or move to Python with named functions (see
bash-robustness step 6): pipelines are write-once
read-never past a width. Intermediate results worth
inspecting go through tee or temp files (atomically; see
script-idempotency) so debugging does not mean rerunning
everything.
4. **Handle real-world text hazards explicitly.** Filenames with
spaces/newlines: `-print0`/`-0`/`--null` chains end to end.
Huge files: stream (awk/sed/grep are O(1) memory) rather than
slurp; `LC_ALL=C` for byte-wise speed and stable sorting
(see shell-portability locale note). CRLF from Windows files
breaks matches invisibly: `dos2unix` or `tr -d '\r'` at
ingestion. Delimiters inside quoted CSV fields defeat awk's
`-F,`: real CSV gets a real parser (csvkit, python csv),
full stop.
5. **Test extractions against edge inputs.** Empty input, one
line, the weird line (unicode, embedded quotes, missing
field): pipe crafted samples through before trusting a
one-liner on production data; for anything recurring, save
the sample and expected output as a regression check (see
golden-master in miniature). Regex greed and anchoring
(`.*` spanning too far, missing `^`) are the classic silent
wrongness: prefer explicit field logic in awk over heroic
regex in sed.
6. **Promote recurring one-liners into tools.** The pipeline you
have pasted three times into chat becomes a script with a
name, `--help`, and stdin/file input (see cli-ux-design,
python-cli-tools); one-liners in runbooks become tested
scripts referenced by the runbook (see runbook-writing).
History is not a package manager.
## Boundaries
- HTML and XML are not line-oriented; grep/sed on them works
until the formatting changes: use pup/xmllint/a real parser
(the same rule that bans regexing JSON when jq exists).
- Binary-adjacent formats (logs with control bytes, mixed
encodings) need inspection (`file`, `xxd | head`) before text
tools; garbage-in pipelines produce confident garbage out.
- At gigabytes-to-terabytes scale or with joins across files,
the answer is a database or DuckDB-class local engine, not a
taller pipeline (see exploratory-data-analysis instincts).
Versions
The database has no other skills named text-processing.
Database snapshot: 13.11.2026 · package version 593f8e1dbd68 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100sed-awk-grep-text-processing-mastery
Use this skill when filtering, transforming, and extracting structured insights from system logs and text streams using sed, awk, and grep. - 95bash-script-generator
Creates bash scripts with argument parsing, error handling, logging, and input validation following current standards. Use when creating new bash or shell scripts, .sh files, CLI tools, scripting automation, text processing workflows (grep/awk/sed pipelines), or building… - 100jq
Use when processing JSON or YAML data from the command line. Covers jq for JSON and yq for YAML — filtering, transforming, and extracting data from structured outputs of APIs, config files, and CLI tools. USE FOR: jq, yq, JSON processing, YAML processing, CLI data… - 100awk
Expert GNU awk (gawk) one-liner generation and transformation. Use this skill whenever the user asks to process text fields, extract columns, compute sums or aggregates, filter structured output, reformat CSV/TSV, parse logs, generate reports, or do any record/field processing… - 100sed
Expert GNU sed one-liner generation and transformation. Use this skill whenever the user asks to transform text, edit files in-place, refactor code via CLI, process logs, do bulk find/replace, extract lines, delete patterns, or perform any stream editing task. Trigger on… - 100server-application-log-analysis-grep-sed
Use this skill when extract critical error patterns, trace IDs, and anomalies from raw server logs using CLI tools.