RedTeam Nexus Labs
Module 03 · Lesson 03 // Networking for CTF

TCP/UDP

Networking for CTF · CTF

Freereading · 10 min
On this page
  1. Overview
  2. Learning objectives
  3. Core concept
  4. How it works
  5. Practical example
  6. Cybersecurity relevance
  7. Common mistakes
  8. Key takeaways
  9. Check your understanding
LESSON

Overview

Ports identify services, but TCP and UDP are the two transport protocols that actually carry the data to those ports. They represent two different philosophies: TCP is a careful, reliable, connection-oriented conversation; UDP is a fast, connectionless "fire and forget" delivery. Knowing which one a service uses — and how each behaves — changes how you scan it, how you talk to it, and how you interpret what you get back in a CTF.

This lesson compares the two protocols and shows why the difference matters for enumeration and interaction on authorized targets.

Learning objectives

  • Contrast TCP and UDP in terms of reliability, ordering, and connection state.
  • Describe the TCP three-way handshake.
  • Explain why UDP scanning is harder and noisier to interpret than TCP scanning.
  • Identify which common services use TCP vs UDP.
  • Relate transport behavior to practical CTF enumeration.

Core concept

Both TCP and UDP sit on top of IP and use ports, but they offer very different guarantees:

TCP (Transmission Control Protocol) is connection-oriented and reliable. Before any data flows, the two sides establish a connection with a handshake. TCP then guarantees that bytes arrive in order, without duplication, and retransmits anything lost. It tracks connection state on both ends. This reliability costs a little speed and overhead but is exactly what protocols like HTTP, SSH, and FTP need.

UDP (User Datagram Protocol) is connectionless and best-effort. There is no handshake and no connection state: a sender simply emits datagrams toward a destination port and hopes they arrive. There is no built-in retransmission, ordering, or delivery guarantee. This makes UDP lightweight and fast, ideal for DNS queries, DHCP, and streaming or gaming traffic where speed matters more than perfect delivery.

A useful summary: TCP = a phone call (you dial, the other side answers, you confirm you can hear each other, then talk); UDP = a postcard (you write it and drop it in the mail with no confirmation it arrived).

How it works

The TCP three-way handshake establishes a connection before data transfer:

  1. The client sends a SYN (synchronize) packet to the server's port.
  2. The server replies with SYN-ACK (synchronize-acknowledge).
  3. The client sends ACK (acknowledge). The connection is now established.

This handshake is why TCP port scanning is relatively clear: if you get a SYN-ACK back, the port is open; if you get a RST (reset), it is closed; if you get nothing, it is likely filtered. A connection is later torn down with FIN or RST packets.

UDP has no handshake. When you send a datagram to a UDP port:

  • If the port is closed, the host may reply with an ICMP "port unreachable" message — but firewalls often suppress this.
  • If the port is open, the service may reply with data, or it may stay completely silent if your datagram was not a valid request it understands.

Because "open" and "filtered" both often look like silence, UDP scanning is slower, less certain, and prone to false results. You frequently need a protocol-aware probe (a real DNS query to port 53, for example) to confirm a UDP service is truly there.

Common transport assignments:

  • TCP: HTTP/HTTPS (80/443), SSH (22), FTP (21), SMTP (25), SMB (445), MySQL (3306), RDP (3389).
  • UDP: DNS (53, though it also uses TCP), DHCP (67/68), SNMP (161), TFTP (69), NTP (123). Many VPNs also use UDP.

Practical example

You are enumerating an authorized CTF box at 10.10.10.5. A TCP scan was clean, but a challenge hint mentions "the service that translates names." That points to DNS, which primarily uses UDP/53, so you run a targeted UDP scan:

# Illustrative UDP scan against an authorized target
$ nmap -sU -p 53,161 10.10.10.5

PORT    STATE         SERVICE
53/udp  open          domain
161/udp open|filtered snmp

Reading this:

  • 53/udp open — the scanner sent a valid DNS probe and got a DNS response, so it can confirm the service. You would now interact with it as a DNS server (next lesson).
  • 161/udp open|filtered — the ambiguous open|filtered state is classic UDP: the scanner got no reply, which could mean an open service that ignored the probe or a firewall dropping it. You cannot be sure from the scan alone and would need an SNMP-specific probe to decide.

Contrast that with a TCP scan, where an open port cleanly completes the handshake. The lesson: TCP gives you crisp answers; UDP forces you to reason under uncertainty.

Cybersecurity relevance

  • Scanning strategy: knowing which protocol a service uses tells you how to scan it. Running only TCP scans will completely miss UDP-only services like SNMP or TFTP that often hold valuable information.
  • Interpreting ambiguity: the open|filtered UDP result is a constant source of confusion. Understanding why it is ambiguous (no negative acknowledgment) keeps you from either dismissing or over-trusting a UDP port.
  • Protocol behavior in exploitation: TCP's stateful handshake and sequence numbers matter for many network attacks and defenses; UDP's statelessness is why it is often abused for spoofing and amplification (a topic defenders must understand).
  • Defense: firewalls and IDS treat TCP and UDP differently; reading a packet capture correctly requires knowing which transport you are looking at.

Common mistakes

  • Only scanning TCP and missing UDP services entirely.
  • Treating UDP silence as "closed." With UDP, silence usually means open-or-filtered, not closed.
  • Expecting a handshake on UDP. There is none; there is no connection to establish or reset.
  • Assuming a service uses only one transport. DNS uses both UDP and TCP; picking the wrong one can make a service look absent.
  • Ignoring how noisy and slow UDP scans are, then trusting an incomplete scan as definitive.

Key takeaways

  • TCP is connection-oriented, reliable, and ordered; UDP is connectionless, best-effort, and fast.
  • TCP establishes connections with a three-way handshake (SYN, SYN-ACK, ACK), which makes TCP scan results clear (open/closed/filtered).
  • UDP has no handshake, so scans are slower and often ambiguous (open|filtered), frequently needing protocol-aware probes.
  • Match your scanning and interaction approach to the transport, or you will miss services and misread results.

Check your understanding

  • Walk through the three steps of the TCP handshake and say what each packet means.
  • Why does a UDP scan often return open|filtered instead of a clear open-or-closed answer?
  • A challenge hints at an SNMP service but your TCP scan found nothing. What should you do differently, and why?
FINISH THIS LESSON

Sign in to save your progress.

PreviousportsNextDNS