Log formats

Overview

A log is a timestamped record of something that happened on a system: a user logged in, a request hit a web server, a service started, a firewall dropped a packet. During a Capture The Flag (CTF) forensics or incident-response challenge, you are usually handed a pile of log files and asked to reconstruct what an attacker did. Before you can hunt through them, you have to be able to read them — and logs come in several distinct formats. This lesson surveys the formats you will meet most often: syslog, the Apache/nginx combined access log, and structured JSON logs.

Everything here assumes an authorized, isolated practice setting: the logs you analyze in a CTF or lab are provided for you, or generated by your own systems. Reading and parsing logs is a defensive, analytical skill.

Learning objectives

• Recognize the three most common log formats: syslog, web-server combined logs, and JSON.
• Identify the standard fields in each format and what they mean.
• Explain why structured formats are easier to search and correlate than free-text ones.
• Pick an appropriate parsing approach for a given format.

Core concept

A log format is the agreed-upon layout of each record — which fields appear, in what order, and how they are separated. Formats fall on a spectrum from free-text (human-readable lines with loose structure) to fully structured (every field explicitly named, as in JSON). The more structured a log is, the more reliably a machine can parse it, and the easier it is to filter, sort, and correlate across sources.

Three formats dominate real systems and CTF challenges:

• Syslog — the classic Unix/Linux system-logging format, semi-structured text.
• Apache/nginx combined access log — the near-universal web-server request log.
• JSON logs — structured, self-describing records used by modern applications and cloud services.

How it works

Syslog. A traditional syslog line looks like:

Sep 25 14:03:12 web01 sshd[2451]: Failed password for root from 203.0.113.9 port 51234 ssh2

Reading left to right: a timestamp (Sep 25 14:03:12), the hostname (web01), the process name and PID (sshd[2451]), and then a free-text message. The timestamp and host fields are structured, but the message itself is prose, so extracting details from it usually means matching patterns. Newer deployments use the RFC 5424 syslog format, which adds fields like a priority value and an ISO 8601 timestamp, but the idea is the same: a few fixed fields followed by a message.

Apache/nginx combined access log. Web servers record each HTTP request as one line in the "combined" format:

203.0.113.9 - - [25/Sep/2025:14:03:12 +0000] "GET /admin HTTP/1.1" 403 512 "https://example.test/" "Mozilla/5.0"

The fields, in order, are: client IP address; two identity fields (usually -); the timestamp in brackets; the request line in quotes (method, path, protocol); the status code (403); the response size in bytes (512); the Referer header; and the User-Agent string. Because the order is fixed, this format is very predictable to parse.

JSON logs. A structured JSON record names every field explicitly:

{"time":"2025-09-25T14:03:12Z","level":"warn","event":"login_failed","user":"root","src_ip":"203.0.113.9","port":51234}

Each key (time, event, user, src_ip) labels its value, so there is no ambiguity about which field is which. JSON logs are trivial to load into analysis tools, filter by key, and join with other data.

Practical example

Suppose a CTF challenge gives you three files: /var/log/auth.log (syslog), access.log (combined), and app.json (one JSON object per line). You want every event tied to the IP 203.0.113.9.

• In the syslog file you scan the free-text messages for the IP and see repeated Failed password lines from sshd.
• In the combined access log you match the first whitespace-separated field and find requests to /admin returning 403.
• In the JSON file you filter records where src_ip equals the address and read the named event values directly.

Three formats, one investigative question — and the JSON was the least error-prone to query because its fields were labeled. Recognizing each format told you how to extract the IP from it.

Cybersecurity relevance

Log analysis is the backbone of incident response, threat hunting, and security monitoring. Analysts live in these three formats: system daemons and Linux hosts emit syslog, every web application produces combined-style access logs, and modern microservices and cloud platforms emit JSON. A SIEM (Security Information and Event Management system) exists largely to normalize these different formats into a common schema so they can be searched together. Being fluent in the raw formats means you can investigate even when no SIEM is available — exactly the situation a CTF forensics challenge simulates. Knowing which fields carry the source IP, the action, and the outcome lets you answer "who did what, when" from raw text.

Common mistakes

• Assuming all logs share one layout. Field order and separators differ per format; parse each with its own rules.
• Splitting web logs naively on spaces. The request line, Referer, and User-Agent are quoted and contain spaces; a plain space-split breaks them. Respect the quotes.
• Treating a free-text message as structured. In syslog, the message is prose — extract details by matching patterns, not fixed positions.
• Ignoring the format's timestamp conventions. Syslog's default timestamp omits the year and time zone; the combined log and JSON usually include an offset. (Timestamps get their own lesson.)

Key takeaways

• Logs range from free-text (syslog) to fully structured (JSON); more structure means easier, more reliable parsing.
• Syslog: timestamp, host, process[pid], then a free-text message.
• Combined access log: IP, identity, [timestamp], "request", status, size, "Referer", "User-Agent", in fixed order with quoted fields.
• JSON: self-describing key/value records, ideal for filtering and correlation.
• Recognizing the format is the first step; it dictates how you extract fields to answer investigative questions.

Check your understanding

• List the fields of an Apache/nginx combined access-log line in order. Which ones are quoted, and why does that matter when parsing?
• Why is a JSON log generally easier to search and correlate than a syslog message?
• Given a syslog line, which parts are reliably structured and which part requires pattern matching to interpret?