Shellcode can be complex. To effectively write shellcode, you need to understand what the system is actually doing. Binding to a remote listening port, dropping privileges or even restoring system rights are all common but difficult tasks at the system level. Knowledge of a lower level language (such as C) will help at this point.
In time, it will become necessary to recognize what a system call is expecting and how this can be achieved using assembly code. You will also need to come to know which registers the data you seek to manipulate are held in and where your shellcode’s arguments will be stored, that is again which registers.
Shellcode exists for both Linux and Windows based hosts, but for the purposes of this article, we will focus on exploiting Linux.
Shellcode is named from its origin and primary use (Foster, et. al. 2005), spawning a shell. Though it is possible to create machine code directly, it is both more common and also far simpler to write in Assembly code and to use this to create the machine code using an assembler such as NASM. Shellcode can allow an attacker to do nearly anything that the exploited program can do as well as calling external functions (such as spawning a root shell). Some of the more common uses of shellcode include:
· Linked library injection,
· Binding a service or a shell to a listening port (including UDP),
· Tampering with and removing log and audit entries,
· Creating user accounts or changing passwords,
· Drop active users (especially administrative accounts) from the system, and
· Shovelling a shell (forcing a reverse connection back to a remote system).
Shellcode, as with assembly code is architecture specific. This makes it a little more difficult as it cannot be easily ported between dissimilar processor families. As shellcode generally manipulates the various processor calls directly in order to point them to a desired system call in place of the original calls, the author needs to have an in-depth understanding of a particular processor register and the opcodes that are used to manipulate these.
In order to create shellcode, Assembly code is specifically written to accomplish a chosen operation. It is necessary to assemble this into machine code without any “null bytes” (Foster, et. al. 2005).
The Linux and Unix operating systems assign individual system call numbers to each function used. A system calls allows the system to manage the communications between the system kernel and the hardware.
“Rings” are generally used to protect or secure the system separating processes and function (figure 1). In this model, controls are built into the kernel to act as check points. These allow or deny calls from higher level rings and control secure functions. Ring 0 is the most trusted or privileged ring in Unix and is defined as kernel mode. Ring 1 is reserved for device drivers and offers some protection from the hardware layer. Ring 3 is the user or application layer and is the security level where most unprivileged operations reside in Linux. Applications running in a higher level need to request access to lower level functions and hardware.
System calls are a means of allowing kernel level functionality and access to hardware from within a program. Many kernel level functions cannot be directly assigned and allocated into the address space of a ring 3 application. System calls allow for the required levels of access in a safer and more controlled manner.
When a user level application needs to access a function that is not within its address space, it needs to first ascertain the system call number (FreeBSD, 2010) of the function it is seeking to invoke and then issue an interrupt (int 0x80)
The assembly instruction “int 0x80” is used to invoke system calls in the manner displayed below:
int 80h ; Call kernel
Here, if a function needed to access a function with more privileges than are provided in Ring 3, the assembly command “call kernel” which would then issue an “int 0x80” and signals the operating system that an event has occurred.
Figure 1: Privileges and rings.
If the access is allowed, the OS can schedule the tasks and processes and allow the function call to complete. In general, a system call will also require one or more arguments. The system call number is loaded into the EAX register with the associated arguments being loaded into the EBX, ECX and EDX registers as required.
As an example, if a sys_write() function is called, the value “04” will be written into the EAX register with the arguments that are associated with the function being written into the EBX, ECX and EDX registers as needed with the “int 0x80” statement being loaded last. E.g. to use the sys_write() function to write a value of 16 we would use:
Mov EAX 04
Mov EBX 10
Code 1: calling sys_write()
This instruction set loads the system call number “04” for “int 0x80” into EAX and then loads the value we wish to write (16) into EBX as 10h before executing the interrupt 0x80. The Linux Man page for Syscalls(2) has a good list of common Linux system calls and their associated numbers.
 Common string operators [such as strcpy()] will terminate when a null byte is read. As such, any shellcode with null bytes remaining will likely fail unexpectedly but certainly without achieving the desired goal.
 A comprehensive system calls is available online from http://bluemaster.iu.hio.no/edu/dark/lin-asm/syscalls.html or if you are on a Linux system, the file “/usr/include/asm-i386/unistd.h” has a full list of the calls.