text tools

Overview

Once you can find and read files, the next skill is processing text. Linux ships with a toolbox of small, sharp commands that search, filter, transform, and combine text — and they become powerful when you connect them with pipes. In CTFs these tools turn a giant log, a wordlist, or a dump of output into the one line that contains the flag.

Learning objectives

• Search text with grep, including recursive and case-insensitive searches.
• Combine commands with pipes (|) and redirect output to files.
• Slice and reshape text with cut, sort, uniq, wc, and tr.
• Extract flags and patterns efficiently from large or messy input.

Core concept

The Unix philosophy is to build small tools that each do one thing well and then compose them. The glue is the pipe (|), which sends the output of one command as the input of the next. A chain like cat data | grep flag | sort | uniq reads a file, keeps matching lines, sorts them, and removes duplicates — each stage doing one job. Mastering a few of these tools and the pipe lets you answer almost any "find the needle in the text" question.

Two related ideas are redirection — > writes output to a file (overwriting) and >> appends — and standard streams, where normal output (stdout) and errors (stderr) are separate channels you can route independently.

How it works

• grep <pattern> <file> prints lines matching a pattern. grep -i ignores case, grep -r searches recursively through directories, grep -n shows line numbers, and grep -v inverts the match (lines that do not match). Patterns can be regular expressions with grep -E.
• cut extracts columns: cut -d: -f1 /etc/passwd splits each line on : and keeps field 1.
• sort orders lines; sort -n sorts numerically; sort -r reverses.
• uniq collapses adjacent duplicate lines (so you usually sort first); uniq -c counts occurrences.
• wc counts: wc -l counts lines, wc -w words, wc -c bytes.
• tr translates or deletes characters: tr 'a-z' 'A-Z' uppercases; tr -d '\n' strips newlines.
• head/tail show the first/last lines (from the Files lesson) and pair well with pipes.

Practical example

A challenge hands you a noisy log with thousands of lines and says the flag was logged once. You do not want to read it by hand:

$ wc -l server.log
20431 server.log

$ grep -i "flag{" server.log
2026-09-25T10:04:11 INFO  issued token flag{gr3p_finds_the_needle}

One grep reduced twenty thousand lines to the single one that mattered. When the interesting data is buried in a structured field, you slice it out:

$ grep "user=" access.log | cut -d= -f2 | sort | uniq -c | sort -rn | head -3
    412 admin
    118 guest
      3 flag_bot

That pipeline extracted the value after user=, tallied how often each appeared, and showed the top three — a classic way to spot an anomaly. You could save any result for later with redirection: grep -i "flag{" server.log > found.txt.

Cybersecurity relevance

Text processing is the daily bread of security work. Log analysis during incident response, sifting through wordlists, parsing scan output, extracting credentials or tokens from a data dump, and building quick one-off filters all rely on grep, pipes, and friends. The ability to compose a short pipeline on the spot — rather than opening a heavyweight tool — is what makes an analyst fast. In CTFs, many challenges are won or lost on whether you can carve the flag out of a large or ugly blob of text efficiently and precisely.

Common mistakes

• Using uniq without sorting first, so non-adjacent duplicates survive.
• Forgetting grep -r when the target might be spread across many files in a directory tree.
• Overwriting with > when you meant to append with >>, destroying earlier results.
• Writing a regex without -E (or not escaping special characters), so the pattern does not match what you expect.

Key takeaways

• Pipes (|) compose small single-purpose tools into powerful text-processing chains.
• grep searches text (-i ignore case, -r recursive, -v invert, -n line numbers, -E regex).
• cut, sort, uniq, wc, and tr slice, order, deduplicate, count, and translate text.
• > overwrites and >> appends output to a file; sort before uniq, and reach for a pipeline before reading by hand.

Check your understanding

• Why must you usually run sort before uniq to remove duplicate lines?
• Which single command would find every line containing password across all files under a directory, ignoring case?
• What is the difference between redirecting output with > and with >>, and when does the difference matter?