URL encoding

Overview

URL encoding (also called percent-encoding) is the rule that lets you safely put arbitrary characters inside a URL. URLs may only contain a limited set of characters, and some characters — like /, ?, &, and = — have special structural meaning. When you need to include one of those characters as data rather than as structure, you percent-encode it: replace it with a % followed by the two-hex-digit value of its byte.

You will see percent-encoding everywhere in web security: in request URLs, in form submissions, in query strings, and as a technique attackers use to smuggle payloads past naive filters. Like the other encodings in this module, it is a reversible representation, not encryption.

Learning objectives

• Explain why URLs need an encoding scheme.
• Distinguish reserved, unreserved, and unsafe characters.
• Percent-encode and decode characters using their hex byte values.
• Recognize how URL encoding matters in web security testing.

Core concept

A URL has structure: scheme://host/path?key=value&key2=value2#fragment. Certain characters are reserved because they act as delimiters — / separates path segments, ? starts the query, & separates parameters, = splits a key from its value, and # marks the fragment. If you want one of those characters to appear inside a value (not as a delimiter), you must encode it so it is not misread as structure.

Characters fall into three groups:

• Unreserved — safe to use literally: A–Z, a–z, 0–9, and - _ . ~.
• Reserved — have special meaning (/ ? # [ ] @ ! $ & ' ( ) * + , ; =); encode them when used as data.
• Everything else (spaces, most punctuation, non-ASCII bytes) — must be percent-encoded.

How it works

Percent-encoding replaces a character with % plus the two hex digits of its byte value — which is exactly the hex representation from the previous lesson. Common examples:

Two notes:

• In the query string of HTML form submissions (application/x-www-form-urlencoded), a space is often encoded as + rather than %20. Both decode back to a space in that context.
• Non-ASCII characters are first turned into their UTF-8 bytes, and each byte is percent-encoded (so one character can become several %xx groups).

To decode, find each %xx, convert the two hex digits to a byte, and substitute the character.

Practical example

Suppose a search box value is the phrase a b&c:

• The space becomes %20.
• The & (which would otherwise start a new parameter) becomes %26.
• Result: a%20b%26c.

So a full query might look like https://example.test/search?q=a%20b%26c. The server decodes %20 back to a space and %26 back to &, recovering the original a b&c as a single value — instead of misreading &c as a second parameter.

On the command line:

python3 -c "import urllib.parse; print(urllib.parse.quote('a b&c'))"
# a%20b%26c
python3 -c "import urllib.parse; print(urllib.parse.unquote('a%20b%26c'))"
# a b&c

Cybersecurity relevance

• Payload smuggling. Attackers percent-encode characters (or double-encode, e.g. %2520 for %20) to slip payloads past filters that only inspect the raw string. A filter that blocks ../ may miss %2e%2e%2f.
• Decoding order matters. If different layers decode at different times, a value can mean one thing to a filter and another to the backend — a classic source of bypasses. Defenders should canonicalize (fully decode) input before validating it, then validate against an allowlist.
• Testing web apps. Understanding percent-encoding lets you craft and read requests accurately in a proxy, and understand how special characters reach (or bypass) server-side logic.
• It provides no confidentiality — it only makes characters URL-safe. Anyone can decode %xx instantly.

Common mistakes

• Forgetting to encode reserved characters as data, so a value like a&b is misparsed as two parameters.
• Confusing + and %20 — + means space only in form-encoded query data, not in every part of a URL.
• Ignoring double-encoding when analyzing suspicious requests (%25 is an encoded %).
• Validating before decoding — always canonicalize input first, or a filter can be bypassed.

Key takeaways

• URL (percent) encoding represents a character as % plus its two-digit hex byte value, letting reserved/unsafe characters travel safely in URLs.
• Reserved characters (/ ? & = # …) are encoded when used as data; unreserved characters (A-Za-z0-9-_.~) are not.
• a b&c encodes to a%20b%26c; in form data a space may instead be +.
• It is reversible with no key — a representation for transport, not encryption — and its handling (double-encoding, decode order) is a common web-security concern.

Check your understanding

• Why must the & in a parameter value be percent-encoded, and what is its encoding?
• In what context does + mean a space, and where should you expect %20 instead?
• Why should a web application decode (canonicalize) input before validating it?