Overview
Retention — how long logs are kept — balances investigative need against cost and compliance. This capstone lesson covers retention and consolidates logs & telemetry.
Learning objectives
- Define log retention.
- Explain the trade-offs and drivers of retention.
- Consolidate the logs-telemetry module.
Core concept
Retention is how long collected logs are stored before deletion. It matters because investigations — especially of incidents discovered late — need historical logs: if logs are deleted too soon, evidence of an earlier compromise is gone. But retaining logs has costs (storage) and is shaped by compliance/legal requirements (some regulations mandate minimum retention periods). Retention policy thus balances: keeping logs long enough for investigation and compliance, against storage cost. Because attacks are often discovered well after they begin (dwell time), adequate retention is critical to being able to investigate them.
How it works
The SOC sets a retention policy — how long each log type is kept — driven by investigative need (long enough to investigate incidents discovered later, given attacker dwell time), compliance/legal mandates (minimum periods for certain data), and cost (storage of high-volume logs). Often a tiered approach is used: recent logs kept readily searchable ("hot"), older logs archived more cheaply but still available if needed. Adequate retention ensures that when an incident is discovered, the historical logs needed to investigate its origin and scope still exist. Too-short retention creates gaps (lost evidence); excessive retention wastes cost. Retention policy is a deliberate balance central to investigative capability and compliance.
Practical example
A SOC discovers an incident that, on investigation, appears to have begun months earlier (long attacker dwell time). Because its retention policy kept logs long enough (balancing investigative need and compliance against cost), the historical logs from months ago still exist — letting analysts investigate the incident's true origin and scope. Had retention been too short, those older logs would be gone, leaving the incident's start a blind spot. The adequate retention policy preserved the evidence needed to fully investigate — showing why retention matters.
Cybersecurity relevance
Retention directly affects investigative capability: because attacks often have long dwell time (discovered well after they begin), adequate log retention is essential to investigate an incident's origin and full scope — too-short retention means lost evidence and blind spots. It's balanced against storage cost and driven by compliance/legal mandates (minimum retention). Retention policy (often tiered hot/archive) is a deliberate business-and-security decision. This consolidates logs & telemetry — log sources, fields, timestamps, normalization, and retention — which together define how the SOC turns raw logs into an analyzable, correlatable, adequately-retained dataset: the foundation for SIEM, detection, and investigation throughout the path.
Common mistakes
- Too-short retention, losing historical logs needed to investigate late-discovered incidents.
- Ignoring compliance/legal retention mandates.
- Not balancing retention against cost (or archiving older logs efficiently).
Key takeaways
- Retention is how long logs are stored; it must balance investigative need (historical logs for late-discovered incidents) and compliance mandates against storage cost.
- Attacks often have long dwell time, so adequate retention is critical to investigating an incident's origin and scope; too-short retention loses evidence.
- Often tiered (hot/archive); retention consolidates logs & telemetry (sources, fields, timestamps, normalization, retention) into an analyzable, correlatable, retained dataset — the foundation for SIEM and investigation.
Check your understanding
- Why is adequate log retention critical given attacker dwell time?
- What factors must a retention policy balance?