Cryptography for CTF for beginners: practice

Depov

Moderator
Staff member
MODERATOR
ULTIMATE
SUPREME
PREMIUM
MEMBER
Joined
Feb 18, 2025
Messages
548
Reaction score
952
Deposit
0$
Encoding and encryption – the difference that solves crypto-tacks

Before breaking something, you need to clearly separate two concepts. The confusion between coding and encryption is the cause of the lost watch on every second CTF. Read more in our guide to creating ctf tasks.

Coding is the transformation of data from one format to another. There's no secret here. Base64, hex, URL-encoding — ways to present the same data with different characters. Anyone decodes back, knowing the algorithm. In real attacks, coding is not used for protection, but for transport: according to MITRE ATT&CK, the Standard Encoding (T1132.001, Command and Control) technique describes how the malwar encodes C2 traffic in Base64 so that the data does not break when transmitted through text protocols and does not attract the attention of primitive filters.

Encryption is a conversion using a key. Without a key or without knowledge of the weakness of the algorithm, it is impossible to restore the source text. Caesar's cipher is a wildcard with a key-shift from 1 to 25. XOR is a broken operation with a key of arbitrary length.

On CTF, this difference is critical: coded – decode, encrypted – break. The first 10 seconds of the decisions of any crypto-task go to the definition that you are in front of you. See the familiar alphabet Base64 with padding? Decode. See a text similar to English, but the letters “not those”? Substitution code. Chaotic set of bytes in hex? Probably XOR.

The same principle works in the opposite direction: the Obfuscated Files or Information (T1027) technique and the Encrypted/Encoded File (T1027.013) technique describe how malware uses multi-layer coding and encryption to bypass antiviruses. On the CTF, this turns into tasks with nested encodings – Base64 on top of the hex on top of ROT13. Such a doll. The ability to define string encoding is the first and foremost skill for any crypto-task.
Base64 – recognize and decode

Base64 is the most frequent element of entry-level crypto-tasks. Strictly speaking, this is not cryptography at all, but coding, but the CTF authors regularly stuff Base64 into the crypto category. The task is either simply to decode the string, or to untangle the chain from several nested layers of coding.
Base64 visual markers and how to define line encoding

Learning to recognize the type of cipher (and encoding) in a second is a key skill. Here is a table of visual differences of the three most frequent encodings:
Sign Base64 Base32 Hex (Base16)
Register of Letters Mixed (A-Z, a-z) Only the top (A-Z) A-F (any register)
Numbers 0-9 2-7 0-9
Special Symbols +, / No No
Padding = or == Up to 6 = Missing
Length Multi 4 Kratna 8 Even

The line Q1RGe3czbGMwbTNfdDBfY3J5cHQwfQ== – mixed case, numbers, two = at the end. Base64. The line 4A4F4B45 only symbols 0-9A-F, without padding. Hex. The line JBSXSIJAKRUGS4ZA...4=== – only the upper register with the numbers 2-7 and triple padding. Base32.

Remember this table – and the question “how to define line encoding” will cease to arise. That's enough for the eyes to begin with.
Base64 decoding: CLI, Python, CyberChef

Three ways – from fast to universal.

Linux command line The fastest way to the CTF. Team echo 'Q1RGe3czbGMwbTNfdDBfY3J5cHQwfQ==' | base64 -d instantly issues CTF{w3lc0m3_t0_crypt0}. On tournament when every second on the account, faster nothing.

Python – one line in REPL: import base64; print(base64.b64decode('Q1RGe3czbGMwbTNfdDBfY3J5cHQwfQ==').decode()). The result is the same. It is useful when you need to embed decoding in the script.

CyberChef for CTF is a visual transformation builder from GCHQ. Insert a string in Input, drag the “From Base64” block into Recipe, result in Output. For Base64, online decoding CyberChef is the most convenient option: immediately shows the result and allows you to build up the chain without switching between tools.
Invested encodings — typical CTF trap

The authors of the Tasks love to invest in each other. Base64-decoding result is Base64 again. Or Base64 on top of the hex. Or URL-encoded Base64. CryptoHack and picoCTF have tasks with 3-5 layers of nesting.

Detangling algorithm:

Decode the first layer
Look at the result – if it is again similar to encoding on visual markers from the table above, decode again
Repeat until you receive the read text or flag format

In CyberChef, this is done by building blocks in Recipe. Five layers of Base64? Five blocks of "From Base64" each other. Base64 → hex → Base64? Three blocks in the right order. CyberChef here wins the command line - the visual chain is more obvious than the pipeline of five pipe-teams.

The nested encodings refer to real-world scenarios: the Deobfuscate/Decode Files or Information (T1140) technique in MITRE ATT&CK describes how an analyst or automated tool removes coding layers from a malicious file. On the CTF you do the same, only in the role of a cryptoanalyst.
Caesar cheat — Brutforce 26 options per second

Caesar’s cipher is a classic wildcard, where each letter of the alphabet shifts to a fixed number of positions. The key is the shift number from 1 to 25 (shift 0 gives the source text, shift 26 - too). One of the simple ciphers for beginners and at the same time is the first real cipher that you will break into CTF.

Key space is 25 options. By analogy with MITRE ATT&CK – literally Reduce Key Space (T1600.001). The key space is so small that the total overtaking is laid in milliseconds.
How to recognize Caesar's cipher in crypto-task

Encrypted line FWI{u0w_wku33} with a known flag format CTF{...} prompts a shift in three seconds: C→F = +3. Check: T→W (+3), F→I (+3) – match. Shift 3.

Signs of Caesar's cipher in the text:

The structure is saved: spaces, punctuation, word length in place
Only letters, numbers and special characters are usually not touched
The text is visually “almost readable” – the letters of the Latin alphabet (or Cyrillic alphabet), but some “wrong”
If the flag format is known (e.g. CTF{), the shift is calculated by the first letters

Key difference from Base64: Caesar's cipher retains the structure of the text, and Base64 turns everything into a homogeneous line without spaces. This is the main visual marker for recognizing the type of cipher.
Automatic overkill: Caesar cipher key brutforce

Caesar's cipher hacking is a choice of 25 options. Manually do this is meaningless when the script can cope in a moment:

ct = "FWI{u0w_wku33}"
for shift in range(26):
pt = ""
for c in ct:
if c.isalpha():
base = ord('A') if c.isupper() else ord('a')
pt += chr((ord(c) - base - shift) % 26 + base)
else:
pt += c
print(f"ROT-{shift:2d}: {pt}")

Conclusion – 26 lines. With ROT-3 we get: FWI{u0w_wku33} → CTF{r0t_thr33} The flag is found. Check: F minus 3 = C, W minus 3 = T, I minus 3 = F. In the lowercase: u→r, w→t, w→t, k→h, u→r. Read: r0t_thr33 (ROT three). Ready.

Alternatives for those who do not want to write code: dCode.fr automatically detects Caesar's cipher and shows all 26 variants, CyberChef has a "ROT13 Brute Force" block that outputs all shifts at once. For fast crypto tasks CTF solutions on tournament these tools save time.
Frequency analysis of Caesar cipher and ROT13

On a CTF, Caesar's cipher is usually applied to short lines, and the gross force works faster than any analysis. But if a long text is encrypted without a known flag format, frequency analysis helps: in the English text, the most frequent letter is e (~12.7%), then t (~9.1%) a (~8.2%). Find the most frequent letter in the cipher, assume that it corresponds e, calculate the shift. For Russian-speaking tasks: the most frequent letter - о (~10.97%), then е (~8.45%) а (~8.01%).

The frequency analysis of Caesar’s cipher is the basis for the cryptanalysis of all wildcard ciphers. In CTF, it is used less often than a brutforce (25 variants is nothing for a computer), but understanding the principle is critical for more complex ciphers: Vigenère, arbitrary substitution, polyalphavite ciphers.

It is worth mentioning separately ROT13 – Caesar’s code with a shift of exactly 13. His chip: ROT13 is an involution, the application returns the source text twice. In Linux: echo 'текст' | tr 'A-Za-z' 'N-ZA-Mn-za-m'. ROT13 is found on CTF so often that it has a separate block in CyberChef and built-in teams in many text editors.
XOR encryption in CTF – from single-byte key to repeating-key

XOR (excluding OR) is a smash operation that is used everywhere in crypto-tasks. The main property: A ⊕ K = C and C ⊕ K = A The same operation encrypts and decrypts. Beautiful, right? XOR CTF encryption is the second most frequent task category after encoding at the initial level.
How to recognize XOR in hex-dampa

XOR-encrypted data looks like a random set of bytes. But the visual markers are:

If the key is one byte, all zero bytes of the source text turn into the same byte (the key itself). The same byte is repeated suspiciously often in a hex dump is a clue.

With a repeating key, the patterns are repeated with a period equal to the length of the key. The Hex dump is visually less “random” than the real random. On a fairly long hex-dump, you can notice rhythmicity - the eye gets used to after a couple of dozen tassks.

The fastest way is the known plaintext. If you know part of the source text (flag format CTF{, File title PK for ZIP, %PDF for PDF, \x89PNG for PNG), XOR of this fragment with the corresponding part of the cipher text gives a piece of key. When a file has a standard title, it works trouble-free.
Brutforce single-byte XOR key

If the key is one byte, the key space is 256 options. Go over everything and evaluate the result by the frequency of “normal” ASCII symbols – a task on six lines of Python. Counting frequency symbols (etaoinshrdlu for English) automatically identifies the correct key:

ct = bytes.fromhex("1b37373331363f78151b7f2b783431333d")
for k in range(256):
pt = bytes(c ^ k for c in ct)
score = sum(c in b' etaoinshrdlu' for c in pt.lower())
if score > len(ct) // 3:
print(f"Key 0x{k:02x}: {pt}")

The script only outputs those variants where more than a third of the characters are the frequency letters of the English language. The correct key usually gives a score that is multiples of all others. Hex ASCII conversion here occurs automatically through bytes.fromhex().
Repeating-key XOR and known-plaintext attack

If the key is longer than one byte, the XOR still breaks, but you need a different approach. The basics of repeating-key XOR cryptanalysis come down to three steps:

Step 1 – determining the length of the key. Hamming Method: Break the cipher text into blocks of different lengths (2, 3, 4... bytes), calculate the Hamming distance between the first two blocks and normalize to the length of the block. The smallest normalized distance indicates the likely length of the key. This method is the classic Cryptopals (set 1, challenge 6).

Step 2 – split into “bands.” When the length of the key is known (suppose 4 bytes), take every 4th byte of the cipher text - you get a text encrypted with one byte. Apply the one-byte XOR brutfors to each "lane" separately.

Step 3 – build the key. Combine the found bytes of each band - get the full key. Decrypt the entire text.

CyberChef has an “XOR Brute Force” block for short keys. For keys longer than 2-3 bytes, you will have to write a script or use ready-made solutions from CryptoPals.
Tools for crypto CTF – working arsenal

The toolkit for crypto CTF of entry-level solution is compact. Here’s what’s really used on every tournament:
The instrument For what When to grab
CyberChef Encoding Chains, Base64, ROT, XOR First thing on any dug
dCode.fr Auto-determination of the cipher, the Caesar brutforce, Vigenère When you don't know the type of cipher
Python Custom Brutforce, Hex-Parsing, Automation When the finished tools are not pulled
CryptoHack Crypto Taxation Training and Practice Between CTF – observation training
CLI Linux base64 -d, xxd, tr, od Quick one-time operations at the tournament

CyberChef for CTF is the main tool. Its "Magic" unit tries to automatically define the type of encoding and apply decoding. On simple dumps saves time, although on complex chains it is better to collect Recipe manually - Magic sometimes misses, and then you lose time, trusting the automation.

dCode.fr is the second most useful. Download the encrypted text, and the site will offer options: Caesar, Vigenère, substitution, transposition. For quick recognition of the type of cipher is indispensable.

Python – for tasks where custom automation is needed. Single-byte XOR-brutforce, repeating-key analysis, parsing hex-dumps - three or ten lines of code. Libraries base64, binascii, itertools cover 90% of the entry-level needs.


All of the above is reduced to a step-by-step algorithm for solving any entry-level crypto-task:

Step 1 – visual score (5 seconds). See the alphabet and line structure:
What do you see Likely type
Mixed case + numbers + = Padding Base64
Only 0-9 and A-F Hex (Base16)
Top register + digits 2-7 + padding Base32
Points and Dashes Through / Morse code
Readable structure, but the letters "shifted" Caesar cipher (substitution)
Random bytes, unreadable hex-dump XOR or more complex cipher
Numbers via hyphen (14-15-2-15) Letter Numbers (A=1)

Step 2 – hypothesis test (10 seconds). Try decoding to CyberChef or dCode. If the result is read text or other encoding, move further along the chain.

Step 3 – the application of a specific tool. Base64 → base64 -d. Caesar → Brutforce 25 shifts. XOR → brutforce key. Unknown cipher → CyberChef Magic or dCode autodetermination.

Step 4 – checking the flag format. The result should contain the platform flag format (CTF{...}, flag{...}, picoCTF{...}). If the format is found, you give. If not, maybe we need another layer of decoding. Return to Step 1.
 
Top Bottom