Binary formats

Overview

Before you can analyze a compiled program in a reverse-engineering or forensics challenge, you need to know what kind of file you are looking at. A binary is a file containing machine code and data meant to be run by a computer, not read by a human. Operating systems use standard executable file formats to describe how that machine code and data are laid out so the system can load and run them. This lesson introduces the three formats you will meet most often — ELF, PE, and Mach-O — and the magic bytes that identify them.

Learning objectives

• Identify the three major executable formats and the platforms that use them.
• Recognize a file's format from its magic bytes.
• Explain the roles of headers, sections, and segments in a binary.

Core concept

An executable file format is a structured container. It is not just a blob of machine code — it has a defined layout that tells the operating system's loader where the code is, where the data is, what memory permissions each part needs, and where execution should begin. The three dominant formats are tied to their operating systems:

• ELF (Executable and Linkable Format) — used by Linux and most Unix-like systems.
• PE (Portable Executable) — used by Windows for .exe and .dll files.
• Mach-O (Mach Object) — used by macOS and other Apple platforms.

Each format begins with a header, and the very first bytes of that header are magic bytes: a short, fixed signature that identifies the format.

How it works

Every one of these formats shares the same broad structure, even though the details differ:

• Magic bytes — the first few bytes act as a fingerprint. A tool reads them to decide how to parse the rest of the file. The magic bytes are fixed values defined by each format's specification:
  • ELF: 7F 45 4C 46 (that is 0x7F followed by the ASCII characters E, L, F).
  • PE: 4D 5A at the very start — the ASCII letters M and Z, a legacy DOS-header marker. Further into the file a 50 45 00 00 (P, E, \0, \0) signature marks the real PE header.
  • Mach-O: FE ED FA CE for 32-bit images and FE ED FA CF for 64-bit images (a "fat"/universal archive bundling multiple architectures uses CA FE BA BE).
• Header — after the magic bytes, the header records global facts about the file: the target CPU architecture, whether it is 32-bit or 64-bit, the entry point (the address where execution begins), and pointers to the tables that describe the rest of the file.
• Sections — named regions used mainly at build and analysis time. Typical ELF sections include .text (executable code), .data (initialized read/write data), .rodata (read-only data such as string literals), and .bss (uninitialized data). Sections are what a disassembler uses to know which bytes are code and which are data.
• Segments — regions the loader maps into memory at run time, each with permissions like read, write, and execute. One segment can contain several sections. Sections are the build-time/analysis view; segments are the run-time/loading view.

Practical example

Suppose a CTF hands you a file called challenge with no extension. You inspect its first bytes and see 7F 45 4C 46. Those magic bytes tell you it is an ELF binary, so it is meant for Linux. Reading a little further in the header, you learn it is a 64-bit executable and you find its declared entry point. Now you know which disassembler settings to use and where the program starts running. Had the first bytes been 4D 5A, you would instead be looking at a Windows PE file; FE ED FA CF would indicate a 64-bit Mach-O for macOS. The magic bytes alone reframed your entire approach — all before running anything.

Cybersecurity relevance

Recognizing binary formats is foundational to reverse engineering, malware analysis, and forensics. Analysts constantly identify files by their magic bytes because file extensions can be missing, wrong, or deliberately misleading — a malware sample may be named invoice.pdf yet begin with 4D 5A, revealing it is actually a Windows executable. Understanding headers, sections, and segments lets an analyst locate the code, find the entry point, and reason about which regions are executable. These same structures underpin how loaders enforce protections like non-executable data regions, which matters for both attack and defense. In authorized CTF and lab practice, this knowledge is the first step before any deeper analysis.

Common mistakes

• Trusting the file extension instead of the magic bytes to determine the real format.
• Confusing sections with segments — sections are the build/analysis view, segments are the run-time loading view.
• Assuming every binary is ELF because you learned on Linux, then misreading a PE or Mach-O file.
• Forgetting a PE file starts with MZ and looking only for the later PE\0\0 signature.

Key takeaways

• ELF (Linux), PE (Windows), and Mach-O (macOS) are the three major executable formats.
• Magic bytes identify the format: ELF 7F 45 4C 46, PE 4D 5A, Mach-O FE ED FA CE/FE ED FA CF.
• A binary's header records the architecture and entry point; sections describe code and data for analysis, and segments describe how regions are mapped into memory with permissions.
• Always identify a file by its magic bytes, not its name or extension.

Check your understanding

• What magic bytes identify an ELF file, and what does a PE file begin with?
• What is the difference between a section and a segment in an executable format?