Reverse engineering an unfamiliar instruction set under CTF time pressure

First published Jun 2012•Rewritten Nov 2026•Reverse engineering CTF write-ups

This note revisits a DEF CON CTF binary challenge that paired a small network capture with an executable in a format competitors did not immediately recognise, and walks through the reasoning rather than the exact disassembly.

Working out what you are looking at

The packet capture showed a short TCP session carrying only ten bytes of payload beyond the handshake, which was an obvious hint that those ten bytes mattered. The accompanying executable, however, did not match any common binary format competitors expected, and searching its internal strings eventually identified it as a binary built for OpenVMS running on the Alpha architecture, a combination rarely seen outside legacy enterprise systems.

Getting tooling to cooperate

A disassembler's auto-detection can misidentify an unusual file as something more familiar, in this case initially reading as a generic DOS executable. Getting a correct disassembly meant manually telling the tool which processor family to target and using a build of the disassembler that supported 64-bit analysis, since the Alpha architecture is 64-bit. Once that was set correctly, the instructions became legible, even though Alpha's instruction set was unfamiliar to most of the team.

Inferring behaviour without full documentation

Several function calls in the disassembly pointed outside the program itself, consistent with operating-system calls rather than ordinary library functions, and could not be resolved precisely from the binary alone. Rather than block on that, the team reasoned from the challenge's networking context: the code's structure matched a typical server flow of opening a socket, binding, listening, and then looping on accept and receive. That loop lined up with where the ten captured bytes would be processed, and the operation performed on them turned out to be a simple XOR against a repeating key.

Closing the loop with a hint

With only partial knowledge of the key, the challenge organisers released an additional clue referencing the word "leet", a well-known piece of hacker-community shorthand for substituting letters with similar-looking numbers. Combining that hint with the handful of key bytes already recovered from known plaintext let the team reconstruct the rest of the key and recover the original message.

Takeaway

When full documentation or tooling support for an architecture is not available, treating the protocol context, and partial known plaintext, as its own source of evidence can get a reverse engineer further than reading every instruction.

This note has been rewritten and checked for this edition; it is not a copy of the original post.