Challenge: flag_checker
Category: Reverse Engineering
TL;DR
Extract encrypted constants from .rodata, reverse XOR and Atbash cipher transformations. The flag validation uses simple cryptographic operations that can be reversed statically.
Initial Analysis
bash$ file flag_checker
flag_checker: ELF 64-bit LSB pie executable, x86-64, dynamically linked
checksec --file=flag_checker
RELRO STACK CANARY NX PIE RPATH RUNPATH Symbols FORTIFY Fortified Fortifiable FILE
Full RELRO Canary found NX enabled PIE enabled No RPATH No RUNPATH No Symbols No 0 6 flag_checker
$ ./flag_checker
Enter flag: test
Nope.
Extracting String Constants:
$ strings flag_checker | grep -i bup
BUPCTF{}_-?R3NJOYgh1s7E
This template reveals the flag format and some characters. The binary also contains many anti-debugging strings (gdb, ida, radare2, etc.), suggesting heavy protection. Dumping .rodata Section: Looking at the .rodata using radare2, we can see that:
objdump -s -j .rodata ./flag_checker
./flag_checker: file format elf64-x86-64
Contents of section .rodata:
3000 01000200 3f004c44 5f505245 4c4f4144 ....?.LD_PRELOAD
3010 00676462 006c6c64 62007232 00726164 .gdb.lldb.r2.rad
3020 61726532 00696461 00676869 64726152 are2.ida.ghidraR
3030 756e0069 64617436 34006964 6170726f un.idat64.idapro
3040 0062696e 6172796e 696e6a61 00626e00 .binaryninja.bn.
3050 4c445f41 55444954 00474442 00523250 LD_AUDIT.GDB.R2P
3060 49504500 424e5f43 4f524500 4944415f IPE.BN_CORE.IDA_
3070 50415448 00726164 61726500 67686964 PATH.radare.ghid
3080 72610069 64617400 72002f70 726f632f ra.idat.r./proc/
3090 73656c66 2f737461 74757300 54726163 self/status.Trac
30a0 65725069 643a002f 70726f63 002f7072 erPid:./proc./pr
30b0 6f632f25 732f636f 6d6d002f 70726f63 oc/%s/comm./proc
30c0 2f25732f 636d646c 696e6500 25732f25 /%s/cmdline.%s/%
30d0 7300456e 74657220 666c6167 3a20004e s.Enter flag: .N
30e0 6f20696e 7075742e 000d0a00 666c6167 o input.....flag
30f0 20697320 76616c69 64202100 4e6f7065 is valid !.Nope
3100 2e000000 00000000 00000000 00000000 ................
3110 42555043 54467b7d 5f2d3f52 334e4a4f BUPCTF{}_-?R3NJO
3120 59676831 73374500 00000000 00000000 Ygh1s7E.........
3130 00000000 00000000 01000000 00000000 ................
3140 00000000 00000000 80c3c901 00000000 ................
3150 031c1102 1d0f2113 69056b12 050f1c17 ......!.i.k.....
3160 05136b2e 296d0565 050c370b 6a180513 ..k.)m.e..7.j...
It's already showing some parts of the flag. We need to dig deeper! Looking at Ghidra's decompilation, the validation logic references a global variable:
do {
uVar48 = uVar48 + 1;
*pbVar43 = *pbVar43 ^ 0x5a;
pbVar43 = pbVar43 + 1;
} while (uVar48 < sVar37);
if (sVar37 != 0x22) goto LAB_00101b7a;
lVar30 = FUN_00102450();
auVar19[1] = local_448[1];
auVar19[2] = local_448[2];
auVar19[3] = local_448[3];
auVar19[4] = local_448[4];
auVar19[5] = local_448[5];
auVar19[6] = local_448[6];
auVar19[7] = local_448[7];
auVar19[0] = local_448[0];
auVar19._8_2_ = sStack_440;
auVar19._10_6_ = uStack_43e;
auVar53 = _DAT_00103150 ^ auVar19 | _DAT_00103160 ^ local_438;
auVar53 = auVar53 | auVar53 >> 0x40;
auVar53 = auVar53 | auVar53 >> 0x20;
auVar53 = auVar53 | auVar53 >> 0x10;
lVar32 = FUN_00102450();
bVar27 = (ulong)(lVar32 - lVar30) < 0xbebc201 &&
((local_428 == 'i' && local_427 == '\'') &&
(auVar53[0] == '\0' && auVar53[1] == '\0'));
}
else {
uVar48 = sVar37 & 0xfffffffffffffff0;
auVar53._8_4_ = 0x1a1a1a1a;
auVar53._0_8_ = 0x1a1a1a1a1a1a1a1a;
auVar53._12_4_ = 0x1a1a1a1a;
The symbol _DAT_00103150 indicates data at address 0x00103150. This is our target - the encrypted comparison constant.
bash$ objdump -s -j .rodata flag_checker | grep -A 2 "3150"
3150 031c1102 1d0f2113 69056b12 050f1c17 ......!.i.k.....
3160 05136b2e 296d0565 050c370b 6a180513 ..k.)m.e..7.j...
This 32-byte sequence is what our transformed input gets XORed against and compared to zero. If the XOR result is all zeros, our transformed input must exactly match these bytes.
Decompilation Analysis
Using Ghidra to analyze the main validation function reveals: Anti-Debug Mechanisms (Can be Ignored) The binary implements multiple detection techniques:
Flag Validation Logic
The decompiled pseudocode shows: c// Read input (max 256 chars) fgets(input, 0x100, stdin); length = strcspn(input, "\r\n");
// Apply transformations (simplified)
transform_chars(input, length); // Character substitution
xor_bytes(input, 0x5A, length); // XOR each byte with 0x5A
// Validate length and content
if (length == 0x22) { // Must be exactly 34 bytes
// Compare first 32 bytes with encrypted constants at 0x3150
if (memcmp(transformed_input, encrypted_data, 32) == 0 &&
transformed_input[32] == 'i' &&
transformed_input[33] == '\'') {
puts("flag is valid !");
}
}
Understanding the Character Transform
Examining the transformation loop more carefully:
cfor (i = 0; i < length; i++) {
c = input[i];
// Check if uppercase (A-Z)
if ((c + 0xBF) < 0x1A) {
c = (-0x25 - c) & 0xFF; // Transform uppercase
}
// Check if lowercase (a-z)
else if ((c + 0x9F) < 0x1A) {
c = (-0x65 - c) & 0xFF; // Transform lowercase
}
// Non-alphabetic
else {
c = (-0x65 - c) & 0xFF; // Transform other chars
}
input[i] = c;
}
// Then XOR everything
for (i = 0; i < length; i++) {
input[i] ^= 0x5A;
}
This is actually implementing the Atbash cipher for alphabetic characters:
For uppercase: A↔Z, B↔Y, C↔X, ..., M↔N
For lowercase: a↔z, b↔y, c↔x, ..., m↔n
Non-alphabetic characters remain unchanged (the transform and XOR cancel out)
Solution
Decryption Strategy: Both XOR and Atbash are self-inverse operations, so we can reverse them:
Extract the 32-byte encrypted constant from offset 0x3150 Reverse XOR with 0x5A Reverse Atbash on alphabetic characters only Append the characters that transform to 'i' and '''
Implementation
I've fed LLM all the information that I've collected and asked it Nicely to write a solution scipt. It gave me the solution script.
python#!/usr/bin/env python3
def atbash(text):
"""Apply Atbash cipher (self-inverse)"""
result = []
for ch in text:
if 'A' <= ch <= 'Z':
result.append(chr(ord('Z') - (ord(ch) - ord('A'))))
elif 'a' <= ch <= 'z':
result.append(chr(ord('z') - (ord(ch) - ord('a'))))
else:
result.append(ch)
return ''.join(result)
#Encrypted data from .rodata:0x3150
encrypted = bytes([
0x03, 0x1c, 0x11, 0x02, 0x1d, 0x0f, 0x21, 0x13,
0x69, 0x05, 0x6b, 0x12, 0x05, 0x0f, 0x1c, 0x17,
0x05, 0x13, 0x6b, 0x2e, 0x29, 0x6d, 0x05, 0x65,
0x05, 0x0c, 0x37, 0x0b, 0x6a, 0x18, 0x05, 0x13,
])
#Step 1: Reverse XOR
after_xor = bytes([b ^ 0x5A for b in encrypted])
intermediate = ''.join(chr(b) for b in after_xor)
print(f"After XOR: {intermediate}")
#Step 2: Reverse Atbash
decoded = atbash(intermediate)
print(f"After Atbash: {decoded}")
#Step 3: Find last 2 characters
#The decompiled code checks:
#local_428 == 'i' && local_427 == '\''
#These are positions 32 and 33 in the transformed array
#What original character becomes 'i' after Atbash + XOR?
#Work backwards:
#transformed[32] = 'i' = 0x69
#Before XOR: 0x69 ^ 0x5A = 0x33 = '3'
#Before Atbash: '3' (numbers don't change in Atbash)
#Therefore: original[32] = '3'
#What original character becomes '\'' after Atbash + XOR?
#transformed[33] = '\'' = 0x27
#Before XOR: 0x27 ^ 0x5A = 0x7D = '}'
#Before Atbash: '}' (special chars don't change in Atbash)
#Therefore: original[33] = '}'
flag = decoded + "3}"
print(f"\nFlag: {flag}")
Key Insights
- Static Analysis Wins The binary can be solved entirely through static analysis without ever running it or defeating anti-debug protections. The encrypted constants are stored in .rodata and can be extracted with objdump.
- Transformation Chain Original Flag → Atbash (A↔Z, a↔z) → XOR 0x5A → Encrypted Data Both operations are self-inverse, making decryption straightforward: Encrypted Data → XOR 0x5A → Atbash (A↔Z, a↔z) → Original Flag
- The i' Trap
The decompiled code checks
local_428 == 'i' and local_427 == '\'', which might suggest the flag ends with i'. However, these checks validate the transformed values, not the original characters. Working backwards from the transformed values:
'i' (0x69) ← XOR 0x5A ← 0x33 = '3'
'\'' (0x27) ← XOR 0x5A ← 0x7D = '}'
Therefore, the flag ends with 3}, not i'!
Tools Summary
strings - Initial reconnaissance and flag format identification. objdump - Extract encrypted constants from .rodata section. Ghidra (optional) - Decompilation to understand validation logic. Python 3 - Implement reverse transformations and verification.