Unpacking a layered crack-me: RCTF reverse engineering 300
This Windows crack-me from a 2015 CTF is a good illustration of how reverse-engineering challenges stack obstacles rather than relying on one trick. The outer layer was a standard UPX-packed executable, straightforward to unpack with the public UPX utility.
Anti-disassembly and indirection
Underneath, the binary used two common tricks to slow down static analysis. One function deliberately skipped a few bytes of the instruction stream after it returned, so a disassembler that didn't account for the jump produced a misleading instruction listing. A second routine called Windows API functions indirectly through a passed-in address rather than a direct call, which stops many disassemblers from automatically resolving which API is being invoked, though the call target is still recoverable by tracing the argument.
The unpacked binary then decrypted a second, embedded executable with a single-byte XOR and ran it in memory, a classic loader pattern also seen in malware, used here to hide the real verification logic from static tools and from simple memory scanning.
The verification logic
The embedded program's check was a multi-stage transform of the user's input: a byte-to-hex expansion, followed by a position-dependent XOR that treated repeated substrings differently from unique ones, followed by a final reordering step driven by a lookup table that depended on the input length. The comparison target was a fixed string baked into the binary.
Because the transform was reversible, the practical path to the answer was to reconstruct each stage in reverse: undo the reorder, undo the XOR, undo the hex expansion, recovering the original input. The length-dependent lookup table meant the solver first had to fix the input length by working out how many characters the final transform implied, then derive the table for that specific length using a debugger before running the recovery script.
Why it matters
None of the individual tricks here are exotic, packing, indirect calls, embedded payloads, multi-stage encoding, but combining them raises the cost of analysis considerably. The same layering shows up in real malware, which is why practising on crack-mes like this one is useful preparation.
Key takeaways
- UPX-style packing is just the outer layer; expect further obstacles underneath
- Indirect API calls (calling through a passed-in address) defeat automatic disassembler annotation but not manual tracing
- Decrypting and running an embedded PE in memory is a loader pattern shared with real malware
- Multi-stage, reversible transforms are solved by reconstructing each stage in reverse order
- Combining several ordinary tricks, not one exotic one, is what makes a crack-me genuinely time-consuming
This note has been rewritten and checked for this edition; it is not a copy of the original post.