RedTeam Nexus Labs
Module 22 · Lesson 03 // Binary Analysis Basics

assembly overview

Binary Analysis Basics · CTF

Freereading · 10 min
On this page
LESSON

Overview

When you reverse engineer a compiled program, you rarely have the original source code. What you have instead is assembly language — a human-readable representation of the machine instructions the CPU actually executes. This lesson gives you an overview of assembly: why compiled code ends up as assembly, what registers are, and the handful of common instructions you will see over and over.

Learning objectives

  • Explain why high-level source code becomes assembly and machine code.
  • Describe what CPU registers are and why they matter.
  • Recognize common instructions such as mov, add, cmp, jmp, and call.

Core concept

A CPU does not understand high-level languages like C or Python. It executes machine code: raw binary instructions. Assembly language is a one-to-(nearly)-one, human-readable text form of that machine code, where each line is a mnemonic (like mov or add) plus its operands. When source code is compiled, the compiler translates it down into these machine instructions; a disassembler does the reverse-ish step of turning the machine-code bytes back into readable assembly so an analyst can study them. Assembly is CPU-specific — x86-64, ARM, and others each have their own instruction sets — but the core ideas are shared.

How it works

  • Registers — small, extremely fast storage locations built into the CPU. Instructions operate primarily on registers. On x86-64 you will see general-purpose registers such as rax, rbx, rcx, and rdx, plus special-purpose ones like rsp (the stack pointer, tracking the top of the stack), rbp (the base/frame pointer), and rip (the instruction pointer, holding the address of the next instruction). Registers hold the values a program is actively working with.
  • Instructions — each does one small, well-defined operation. The most common ones a beginner meets include:
    • mov — copy a value from one place to another (register, memory, or immediate constant).
    • add / sub — arithmetic: add or subtract values.
    • cmp — compare two values by subtracting them and setting flags (result indicators) without storing the result.
    • jmp — unconditionally jump to another address, changing what executes next.
    • conditional jumps (for example je "jump if equal", jne "jump if not equal") — branch based on the flags set by a previous cmp.
    • call — call a function: it pushes the return address onto the stack and jumps to the function.
    • ret — return from a function by popping the saved return address and jumping back to it.
    • push / pop — put a value onto, or take one off of, the stack.
  • Control flow — cmp followed by a conditional jump is how high-level constructs like if, while, and for are expressed in assembly. A comparison sets the flags; the jump decides which path to take.

Practical example

In a reverse-engineering challenge you disassemble a function that checks a password. You see a pattern like this (x86-64, Intel syntax):

cmp   eax, 0x1337     ; compare the computed value with a constant
je    0x004011a0      ; if equal, jump to the "success" path
jmp   0x004011f0      ; otherwise, jump to the "failure" path

Reading it, you understand: the program computed something into eax, compared it against 0x1337, and branches to "success" only when they match. This tells you the condition you must satisfy to reveal the flag — without ever seeing the original source. Recognizing cmp + conditional-jump patterns is often the key to solving reversing challenges.

Cybersecurity relevance

Assembly is the language of reverse engineering, malware analysis, and binary exploitation. Since analysts almost never have source code for the programs they study, reading disassembly is how they recover a program's logic — finding the check that guards a secret, understanding what a malware sample does, or locating a vulnerable code path. Registers like the stack pointer and instruction pointer are exactly what memory-corruption exploits aim to influence, so understanding them explains both attacks and the defenses against them. Even a basic reading ability for mov, cmp, jmp, and call unlocks most beginner reversing challenges in authorized CTF practice.

Common mistakes

  • Trying to read every instruction instead of focusing on the control-flow decisions (cmp + jumps) that matter.
  • Ignoring which register holds the value of interest, so you lose track of the data.
  • Assuming one syntax — x86 assembly appears in both Intel and AT&T syntaxes, which order operands differently.
  • Forgetting call and ret use the stack to save and restore the return address.

Key takeaways

  • CPUs execute machine code; assembly is its human-readable form, and compilers translate high-level source down to it.
  • Registers are the CPU's small, fast working storage; key x86-64 ones include rax–rdx, rsp (stack pointer), rbp (frame pointer), and rip (instruction pointer).
  • Common instructions include mov, add/sub, cmp, jmp and conditional jumps, call/ret, and push/pop.
  • cmp followed by a conditional jump is how if/loop logic appears in assembly — the pattern to look for when reversing.

Check your understanding

  • What does the cmp instruction do, and how is it typically used together with a conditional jump?
  • Which register holds the address of the next instruction to execute, and which one tracks the top of the stack?
FINISH THIS LESSON

Sign in to save your progress.