Overview
One of Unix's most powerful ideas is combining small tools into larger workflows. Pipes connect the output of one command to the input of another, letting you build sophisticated processing from simple parts. This lesson explains pipes.
Learning objectives
- Explain what a pipe does.
- Chain commands to process data in stages.
- Apply the Unix philosophy of composing small tools.
Core concept
A pipe, written |, connects the standard output of one command to the standard input of the next. Instead of one program doing everything, you chain focused tools: each does one job and passes its result onward. This embodies the Unix philosophy — "do one thing well" and compose tools together.
Common building blocks in pipelines include grep (filter lines), sort, uniq, wc (count), cut, and head/tail.
How it works
When you write cmd1 | cmd2, the shell runs both and feeds cmd1's output directly into cmd2 as its input, without a temporary file. You can chain many stages (cmd1 | cmd2 | cmd3), each transforming the stream. Because data flows through as text, any tool that reads standard input and writes standard output can be a stage — making pipelines endlessly flexible for filtering, transforming, and summarizing data.
Practical example
$ cat /var/log/auth.log | grep "Failed password" | wc -l
42
This pipeline reads the auth log, keeps only lines containing "Failed password", and counts them — quickly answering "how many failed login attempts are recorded?" Each tool is simple; together they answer a real question. (You could also sort/uniq -c the offending IPs to rank them.)
Cybersecurity relevance
Pipes are indispensable for log analysis and investigation: filtering huge logs to relevant events, counting occurrences, and extracting fields are all pipeline tasks central to detection and incident response. For example, chaining grep, sort, uniq -c, and sort -n can surface the most frequent source of failed logins — a possible brute-force attempt. Fluency with pipelines lets an analyst turn raw data into answers quickly, a core blue-team skill.
Common mistakes
- Building one giant command instead of composing small tools with pipes.
- Forgetting a stage reads standard input — some commands need input redirected or piped, not given as an argument.
- Not filtering early, forcing later stages to process more data than necessary.
Key takeaways
- A pipe (
|) sends one command's standard output into the next command's standard input, with no temporary file. - Chaining focused tools (
grep,sort,uniq,wc,cut) embodies the Unix philosophy of composing small programs. - Pipelines are central to log analysis and investigation — filtering, counting, and extracting from large data sets efficiently.
Check your understanding
- What does the
|operator do between two commands? - Write, in words, a pipeline that counts how many lines in a log file contain the word "error."