Building a UNIX/Linux Incident response / Forensic Disk

There are many Linux distributions readily available. This however should not stop you creating your own version of a UNIX forensic tools disc. Whether you are on Solaris, HP-UX or any other variety of UNIX it is simple to create a forensic tools CD that can go between systems. The added benefit of this method is that the audit tools do not need to be left on the production server. This in itself could be a security risk and the ability to unmount the CD and take it with you increases security.

The ability to create a customized CD for your individual system means that the analyst can have their tools available for any UNIX system that they need to work with. It may also be possible to create a universal forensic CD. Using statically linked binaries, a single DVD or CD could be created with separate directories for every UNIX variety in use in the organization that you are auditing. For instance, the same CD could contain a directory called “/Solaris” which would act as the base directory for all Solaris tools.

Similarly, base directories for Linux (/Linux), HP-UX (/HPUX10, /HPUX9) and any other variety of UNIX in use in your organization could be included on the same distribution allowing you to take one disk with you but leaving you ready at all times.

The added benefits of creating your own disk that you can update the tools any time you wish and add new ones. On top of this, those audit scripts that you have been creating may be all listed together in one place. If you are using a KNOPPIX distribution it will not have your audit scripts. These tools then become your trusted source of software. As was noted above, a script could be created that runs your trusted tool and also the tool on the host to verify that the results of the same. If there are any differences it is easy to note that the system may have been compromised. The added benefit of this distribution is that you can also use it for incident response and forensic work if required.

When creating your distribution you should include the following binaries and statically linked format where possible:

• “chown”, “chgrp”, “chmod” ‘
• “cp”, “cat” and “diff”,
• “find”, “ls” and “ps”,
• “dd”.
• “df” and “du”,
• “rm” and “mv”,
• “netstat”, “lsof” and “top”

Compression Applications including: “compress”, “uncompress”, “gzip”, “gunzip”, and “tar”.

• Include “shared libraries” and “static system libraries”
• gdb, nm
• ps, ls, diff, su
• passwd
• strace/ltrace
• MD5 or another has tool (preferably a number of these)
• fdisk/cfdisk
• who, w, finger
• dig
• scripts
• gcc, ldd
• sh, csh

It is also advisable to include “losf”, and the “sleuthkit” tools as well as their related libraries.

Dynamically linked executables are commonly used due to space limits. As a large number of applications can use identical basic system libraries, these are rarely stored in the application itself. An attacker could still compromise these libraries. Threat all system libraries as being suspect and compile all tools using “gcc” set with the “-static” parameter. This will create a static binary or standalone executable. The “ldd” command can be used to demonstrate the dependency discovery process:

$ /cdrom/bin/ldd calc
libc.so.6 => /lib/libc.so.6 (0x40020000)
/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)

About ”ldd”
The command, “ldd” may be used to list dynamic dependencies of executable files or shared objects. The “ldd” command can also be used to examine shared libraries themselves, in order to follow a chain of shared library dependencies.

The “pvs” command may also be useful. This command displays the internal version information of dynamic objects within an ELF file. Commonly these files are dynamic executables and shared objects, and possibly reloadable objects. This version information can fall into one of the following two categories:

• Version definitions described the interface made available by an ELF file. Each version definition is associated to a set of global symbols provided by the file.
• Version dependencies describe the binding requirements of dynamic objects on the version definition of any shared object dependencies. When a dynamic object is built with a shared object, the link-editor records information within the dynamic object indicating that the shared object is a dependency.

For example, the command “pvs -d /usr/lib/libelf.so.1” can be used to display version definition of the ELF file libelf.so.1.

Alternatively, if you are starting out, there is still the SIFT workstation from SANS if you do not wish to go to all the trouble of creating your own. This works for general forensics, but in any organization where you have a set *NIX build, I would recommend creating your own system disk based on those systems you actually deploy.

CSU Webinar

Cyber (Crime / Espionage / Terror)
Where: https://www2.gotomeeting.com/register/253921674

We have just seen the largest cyber espionage incident in recorded history and it is only set to get bigger. The attacks were simpler than many thought would be necessary, but simple controls that could have helped stop these attacks had not been applied. We will discuss the how the rise of cyber based groups engaging in hactivism is creating chaos. In some ways it is only the start as these groups start to do more damage. That said, many simple controls that do not cost much money could have helped these organisations.

Al-Qaeda and other pure terror groups have been on the back foot unable to leverage the social aspects of Web 2.0, but will this change as groups such as Anon and LulzSec define a distributed model for social malfeasance?

Add to this criminal controlled botnets of millions of zombie hosts and the decade is set to be the decade of the hack.

The good news, there are many simple things you can do to make your system and organisation more secure and many of these do not cost anything.

We discuss the rise of cyber-activism, cyber-crime and cyber-espionage.

Presented by Dr Craig Wright of Charles Sturt University and the Global Institute for Cyber Security + Research.

Search for IP Theft and misconfigurations

One step in testing the security of your site is Recon. You can use publicly accessible search engines to look for signs of vulnerabilities on systems. Google, Yahoo, and Microsoft’s Live Search all contain a great deal of information that could indicate the presence of vulnerabilities in systems associated with the target environment.

Some of the old systems (such as AltaVista) were actually better for this as they accepted literal strings and did not as Google tries to do add “helpful” information. In coming posts, I will cover web spidering and how to script a spider using python. Commercial spiders (such as Blackwidow will also be covered).

By sending the appropriate queries to the search engines themselves, we may be able to identify vulnerable systems without actually sending any packets to those systems directly. Google has some of the most advanced search queries, including:

• site: The “site:” directive allows an attacker to search for pages on just a single site or domain, narrowing down and focusing the search. For example typing “site:sans.org” will return all the web pages associated with the sans.org domain
• link: The “link:” directive shows sites that link to a given web site. During recon, this directive can be used to find business partners, suppliers, and customers.
• inurl: The “inurl:” directive lets us search for specific terms to be included in the URL of a given site. This can be helpful in finding well-known vulnerable scripts on web servers. For example, searching for inurl:viewtopic.php will find sites with URLs that contain “viewtopic.php” in them.
• ext: or filetype: Both of these directives allow you to search for specific file types. You can use this to identify all files of a given type within a domain by combining it with the site: directive. If you wanted to find all Microsoft Word files on the SANS website for example you could look for site:sans.org filetype:doc.

There are many, many other things you can find using search engines such as Google. Things like system configurations, passwords, and confidential information can easily be found using the right directive and search strings. A list of useful Google searches to find vulnerabilities can be found at the Google Hacking Database (http://www.exploit-db.com/google-dorks/)

Some Google operators
Google allows you to create logical queries using logical operators.

• Exclude terms using NOT operator: "-" (NOT operator)
• Include common words using AND operator: "+"
• Exact phrases must be surrounded by quotes "Printer header type"
• Wildcards are available for a single word: ("*") or character (".")
• Google is not case sensitive in its searchers: Google, GOOGLE, GooGlE, and google are all treated the same by Google
• The maximum number of terms per search is 32 items
• Boolean operators: AND (default) and OR (|)

Using these terms, you can look for Intellectual property from your organisation that could have leaked onto the web (and here the use of hidden metadata strings is recommended) as well as to search for known vulnerabilities and misconfigurations.

Code auditing

In the last few days I have been picking on coding more than anything else. I have done this for a couple reasons:

1. I have been working on a new class,
2. I have been preparing for my GSSP-JAVA exam, and
3. I have seen poorly coded sites time and again.

The GSSP-JAVA exam is the “GIAC Secure Software Programmer-Java”and follows from my GSSP-.NET certification earlier in the year.

So right now, as I prepare to sit this exam, I have a focus on this topic. In this bent, I will note a few coding issues that should be checked for.

One major issue that we need to teach better and seem to commonly discourage in software engineers is that of good code writing skills.

Poorly written code
When writing code, the following are things that should be avoided. When auditing code, finding these following points is a sign that you should be spending more time in analysing the code for vulnerabilities. These things to look for are:

• Undecipherable variable names. When writing large programs and when working in teams, it can be difficult to remember what the variables actually do. Using a descriptive name makes it simpler to return and update code later as well as making it simpler to review and lowers the time needed for another engineer to come into a project enhancing collaboration. You can expect that errors will occur and the wrong variables will be called when poor variable naming has occurred.
• Large and overly complex functions. Ideally, a function should be small and set for a single purpose. As functions are called in multiple places with different inputs and returns, complex multilevel functions can be difficult to update and maintain. They are also far more prone to errors and changes to them can adversely impact other areas in the code.
• Style inconsistency. Inconsistency makes review and updating the code difficult. It displays a lack of organisation and commonly hides errors.
• Poor set-out and visual structure. Again, a block of code needs to be structured well to be reviewed inefficiently and as we see in code sample 1 (unstructured) and code sample 2 (set-out) the same code can be extremely difficult to review if now set-out well.
• A lack of code documentation and comments. Simply put, if you do not document code, you will be wasting a good amount of time trying to remember what you wanted to achieve. For the reviewer, it can be frustrating to have to try and workout what the developer was seeking to do.

Displayed below we have a sample first of poorly set-out code (Code sample 1) and the same code written in a form that is more clearly readable and reviewed (Code sample 2).

Code sample 1
And the same code set-out so we can see it clearly…

Code sample 2
This is the same code segment, but I have come across the spaghetti code of the first function many times. This poor formatting is not just inefficient, it is insecure as it makes it simpler to make errors.
Some of the other aspects of coding I will cover in coming weeks include:

• Poor use and deployment of cryptographic protocols,
• Poor error handling,
• Misapplied credential management, and
• Test and development code that is loaded into production systems.

More on validating input

Continuing from last nights post on securing code with coffee and sleep, I am following up with with more tips of coding that seem to be neglected by the text books.

I will get to picking on others (many others) in time, but right now I am working on and and up to errors in Chapter 4 of the same book which is:
“Teach Yourself C in 21 days” by SAMS.

As noted yesterday, day 4, has a type and run as they call it. You enter the code even if you do not understand all of it yet.

I have modified the code as is listed below. I have not tidied this such that printf() has been replaced and there are many other things I could do to improve this, but the point is that a simple change can make this more resilient.

Figure 1: The modified section of code.

This compiles well enough and we can see the results in figure 2.

Figure 2: Type and Run “Find_nbr.c” now accepting valid data

I do not see the code changes as too onerous. The “Type and Run” was not designed to have students understand this day, but as a prelude in any event.

Figure 3: We can handle non-numerals now and over large values!

Again, why must we teach poor coding skills and then have to relearn secure coding?

C has NO native error handling and bounds checking. You as the programmer have to tell C just what it needs to do and it will try to do what you want no matter how illogical this can be. As such, you need to defend C from bad code. You need to make sure that any input is validated first.

Validating Input

Continuing from last weeks post on securing code, I am following up with with more tips of coding that seem to be neglected by the text books.

I will get to picking on others (many others) in time, but right now I am working on and and up to errors in Chapter 4 of the same book which is:
“Teach Yourself C in 21 days” by SAMS.

Following day 4, they have a type and run as they call it. You enter the code even if you do not understand all of it yet. I have listed the code from pages 93 &94 below:

Code: find_nbr.c

1: /* Name: find_nbr.c

2: * Purpose: This program picks a random number and then

3: * lets the user try to guess it

4: * Returns: Nothing

5: */

6:

7: #include

8: #include

9: #include

10:

11: #define NO 0

12: #define YES 1

13:

14: int main( void )

15: {

16: int guess_value = -1;

17: int number;

18: int nbr_of_guesses;

19: int done = NO;

20:

21: printf(“\n\nGetting a Random number\n”);

22:

23: /* use the time to seed the random number generator */

24: srand( (unsigned) time( NULL ) );

25: number = rand();

26:

27: nbr_of_guesses = 0;

28: while ( done == NO )

29: {

30: printf(“\nPick a number between 0 and %d> “, RAND_MAX);

31: scanf( “%d”, &guess_value ); /* Get a number */

32:

33: nbr_of_guesses++;

34:

35: if ( number == guess_value )

36: {

37: done = YES;

38: }

39: else

40: if ( number < guess_value )

41: {

42: printf(“\nYou guessed high!”);

43: }

44: else

45: {

46: printf(“\nYou guessed low!”);

47: }

48: }

49:

50: printf(“\n\nCongratulations! You guessed right in %d Guesses!”,

51: nbr_of_guesses);

52: printf(“\n\nThe number was %d\n\n”, number);

53:

54: return 0;

55: }

This compiles well enough and we can see the results in figure 1.

Figure 1: Type and Run “Find_nbr.c” compiled and running using “cl”.

I do not think I am asking to much to add a small validation just to ensure that the data input is a number. We see in Figure 2 that the code does not end and loops in error if a character string is entered in place of an integer that we expect.

Figure 2: Oh no… We did not validate input again.

The program fails if the value is a float as well. Basically, we have no input validation. Again, it seems that these fundamental coding practices are simply ignored or forgotten.

I shall not be looking at the fact that we can enter values outside the ranges and get unpredicted results… Not for this post at least. We will even ignore how you can loop incorrect guesses and increment the counter until it crashes. We will ignore all that for now.

There are many ways we could have validated the input and restricted it to an integer. I will not even cover checking that it is in range here (although this would have fit with the point of the lesson more!)

A simple example would be the Boolean function isdigit(). Using a call such as:

if (isdigit(guess_value))

We could have setup the equivalent of a try block in C. That is, we could test the input and not simply trust it. That is, we need to think of abuse cases when we design use cases.

The following is messy (it is late) and I will list a few better examples tomorrow including using functions as a means of validating the input.

A function would allow us to create a check we could use throughout the program.

In figure 3 we see a small and simple example (though by no means a good use of code) that does a simple check to ensure that only integer values have been entered.

Figure 3: A small validation

We now see (figure 4) that we have an error when badly formatted (non-integer values in this example) data is entered into our program.


Figure 4: And we block non-integer values…

So, validation should really be a concept we start to teach from week 1.



An IDE…
No a separate note, I will plug (freely as I am a commercial customer and pay to use with no freebees etc.) UltraEdit Studio. This is a great IDE and text editor and beats notepad for even simple edits. When writing code, you have the text displayed in colour such that you can tell when you have not closed a parenthesis or ended a line correctly(See Figure 5).

Figure 5: UltraEdit Studio… Better than notepad.

It also displays the line number for when you are debugging. Version 11 is out and it is a great product. My choice of editor and using it makes coding on Windows easier and allows for simple checking without loading a complete environment (such as Visual Studio).

User Activity Monitoring

I was recently asked about user activity monitoring software. There are many and varied reasons for monitoring users in an organization from the creation of automated time-sheets (I actually have a system that does this for tax and account tracking of my own systems) to an analysis of illicit activities. Having a primary focus in incident handling and forensics, this is one area I see all too often.

There are many options and products in this space, but I will note a few I have seen and would recommend.

Visual TimeAnalyzer automatically tracks all computer usage and presents detailed, richly illustrated reports. Easily log individual users or specific projects, and compile detailed accounts of time spent within each program. The program helps track work time, pauses, projects, costs, software and internet use and gives parents have control over their children's PC use.

The software has some privacy safeguards and does not monitor all user data (such as passwords and personal documents). Unlike Spyware (click this link for an analysis of such a program), it does not record specific keystrokes or run background screen captures.

Software Functions:
· User supervision: Get detailed accounts of working hours and breaks
· Computer supervision: Monitor the family's PC or the company's network
· Software metering: Determine how often software is actually used and by whom
· Internet use: Control online time and web usage
· Project overview: Summarize time applied to each project milestone
· Compare users: See most active users and what they have done on their computers

Figure 1: Visual TimeAnalyzer in Action


Another approach is the logging and centralized approach used with DAD.

Figure 2: DAD


DAD logs to a MySQL database and for large deployments, this can allow for an anomaly based analysis of the activities of users outside the norm.


The ability to create SQL queries also adds value to this project.


Whowatch
For those using Linux and Apple systems, whowatch is a viable alternative. By default this is a system based process for the administrative (root) user though SUDO can allow other admins (other than root) to access this program.

whowatch allows you to monitor the following:

• Any users that are logged onto the host, (this can be displayed in real-time).
• Users by login name
• The users tty
• The user host name if they are accessing the system remotely (common in Linux)
• The user's active process(es)
• Processes trees and a tree of all system processes.
• Kill user process and more

All this can be done using a number of separate commands (top, ps, w, who etc), but whowatch has this in a single package making it simpler to provide access to non-root administrators (through sudo).

Better yet, whowatch can be ported to a MySQL database for ongoing monitoring.

Other Linux monitoring options include the Audit functions in *NIX. I will cover sending whowatch output to a SQL database and Linux Audit in a subsequent post.