Programs may perform additional sanity checks on data that the shellcode must pass in order to succeed, such as the following:
All bytes are printable (less than 0x80) ASCII bytes.
All bytes are alphanumeric (A through Z, a through z, or 0 through 9).
To overcome filtering limitations by the vulnerable program, nearly all shellcode encodes the main payload to pass the vulnerable program’s filter and inserts a decoder that turns the encoded payload into executable bytes. Only the small decoder section must be written carefully so that its instruction bytes will pass the strict filter requirements; the rest of the payload can be encoded at compile time to also pass the filter. If the shellcode writes the decoded bytes back on top of the encoded bytes (as usual), the shellcode is self-modifying. When the decoding is complete, the decoder transfers control to the main payload to execute.
The following are common encoding techniques:
XOR all payload bytes with constant byte mask. Remember that for all values of the same size a,b that (a XOR b) XOR b == a.
Use an alphabetic transform where a single byte of payload is split into two 4-bit nibbles and added to a printable ASCII character (such as A or a).
Shellcode encodings have additional benefits for the attackers, in that they make analysis more difficult by hiding human-readable strings such as URLs or IP addresses. Also, they may help evade network IDSs.