character representations

Overview

Every encoding in this module ultimately rests on one idea: characters are numbers. The letter A, a space, an emoji, or a newline are all stored as numeric values, and a character set is the agreed-upon table that maps each character to a number (its code point). Understanding character representations — ASCII, Unicode, and the ways numbers are written (decimal, hex, binary, octal, HTML entities) — is what lets you move fluently between "text" and "raw bytes."

This lesson ties the previous lessons together: Base64, hex, and URL encoding all operate on the byte values that character sets define. There is no secrecy here either — these are open, published mappings.

Learning objectives

• Explain what a character set and a code point are.
• Describe ASCII and how Unicode extends it.
• Recognize the same character written in decimal, hex, binary, octal, and as HTML entities.
• Understand the relationship between Unicode code points and UTF-8 bytes.

Core concept

A character set assigns a number to each character. The foundational set is ASCII, which defines 128 values (0–127):

• 0–31 are control characters (non-printable): e.g. 0x09 tab, 0x0a newline (LF), 0x0d carriage return (CR).
• 32 is the space.
• 48–57 are the digits 0–9.
• 65–90 are uppercase A–Z (so A = 65).
• 97–122 are lowercase a–z (so a = 97).

ASCII only covers 128 characters, which is not enough for the world's writing systems. Unicode extends this to a vast set of code points, written like U+0041 (that is A). Unicode keeps ASCII as its first 128 code points, so ASCII text is also valid Unicode.

How it works

The same character can be written many ways because the underlying number can be expressed in different bases and formats. For the character A (code point 65):

They all name the same byte. Choosing a base does not change the value, only how it looks.

Code point vs bytes (UTF-8). A code point is an abstract number; to store or transmit it you need an encoding. UTF-8 is the dominant one:

• Code points 0–127 (ASCII) use one byte (identical to ASCII).
• Higher code points use 2 to 4 bytes.

So the character é (U+00E9) is one code point but is stored as two UTF-8 bytes (0xC3 0xA9). This is why "one character" can become several %xx groups when URL-encoded.

Named HTML entities

HTML also has named entities for characters that are structurally significant:

• & → &
• < → <
• > → >
• " → "

These are used to display those characters as text rather than have the browser interpret them as markup.

Practical example

You are handed the sequence 72 105. Recognizing these as decimal ASCII code points, you look them up: 72 = H, 105 = i, giving the text Hi. The same text could appear as:

• Hex bytes: 48 69
• HTML entities: Hi
• Binary: 01001000 01101001

All decode to Hi. Quick tools:

python3 -c "print(chr(72)+chr(105))"   # Hi
python3 -c "print(ord('A'))"           # 65

Seeing a run of numbers in the 32–126 range is a strong hint that you are looking at ASCII code points that you can convert straight to text.

Cybersecurity relevance

• Recognizing character representations lets you convert unfamiliar-looking data (fl..., \x66\x6c..., or 102 108 97 103) into readable text and find hidden flags.
• Encoding-based filter bypasses exploit multiple representations: a filter blocking <script> might miss <script> if the value is later HTML-decoded. Defenders canonicalize to one representation before validating, and output-encode correctly for the context (HTML entity-encoding to prevent XSS).
• Unicode pitfalls — visually similar characters (homoglyphs) and normalization differences can be abused in phishing and to slip past naive comparisons.
• These mappings are public and reversible; they organize data, they do not secure it.

Common mistakes

• Assuming everything is ASCII. Non-ASCII text is multiple UTF-8 bytes per character, which affects lengths and encodings.
• Confusing a code point with its bytes — U+00E9 is one code point but two UTF-8 bytes.
• Mixing decimal and hex — 41 in decimal is ), but 0x41 is A; always know the base.
• Forgetting HTML named entities (&, <) when reading or sanitizing markup.

Key takeaways

• Characters are numbers; a character set (ASCII, then Unicode) maps each character to a code point (A = 65 = U+0041).
• The same value can be written in decimal, hex, binary, octal, HTML entities, or escapes — all naming the same byte.
• A Unicode code point is abstract; UTF-8 stores ASCII in one byte and higher code points in 2–4 bytes.
• Recognizing representations is central to decoding data and to understanding encoding-based filter bypasses; these mappings provide no secrecy.

Check your understanding

• What is the difference between a Unicode code point and its UTF-8 byte(s)?
• Write the character A in decimal, hex, and as an HTML numeric entity.
• Why can multiple character representations be used to bypass a naive input filter?