Friday, 12 August 2011

Network Mapping and Organizing the Mapping Results

It is important to plan the scope of all audit and system testing engagements, network mapping is no different. A failure to adequately plan will quickly lead to being overwhelmed. Plan a risk based approach to mapping the network. Start at the perimeter and work in towards the centre, gradually gaining more and more depth as each of the systems is audited.

It is important to ask where the real value lies within the organization. This is not a job for the auditor alone and management should consider the value of the data and information assets. Work from the outside in. With each step go deeper into mapping the weaknesses associated with the organization’s information assets. This should align with the following steps:

  1.  Map the network devices and perimeter,
  2.  Scan the internal systems and Servers,
  3.  Test and map Databases and Applications,
  4.  Create Images and baselines. 
 Creating Network Maps
There are many tools that can be used to scan systems and make a network map. The best known of these tools is nmap. Nmap is available from There are many excellent sources of information for the auditor or security professional wanting to discover more about this tool. Other then the section in the Firewall chapter of this book, the following sites should be one of the first stops in this process:
Though nmap has been ported to Windows, it works best under Linux or UNIX. Too many of the options available within nmap are “broken” by the Microsoft network stack.

We covered using nmap for individual scans in an earlier chapter, “Testing the Firewall”. In this section we look at how to automate the response and make this tool useful for reporting.

The prime limitations with nmap are its reporting capabilities. Nmap does provide output in a “grep’able” format, but there are far more effective tools that can query the data. PBNJ (this package includes ScanPBNJ and OutputPBNJ) can import nmap scan results from an nmap “-oX”, XML format and provides the capability to query this data. The program is written in Perl and provides a means to instantaneously identify changes to the systems and network.

ScanPBNJ can be used directly to scan the network using nmap directly. Using nmap to scan and then import the output into ScanPBNJ requires the use of the nmap XML output format (-oX). ScanPBNJ with the “-x” option can import the results of the nmap XML report.

PBNJ is a suite of tools that provides that capacity to monitor change across a network over time. It has the capacity to save nmap results into a database and check for changes on the target host(s). It saves the details concerning the services running on these hosts as well as the service state. PBNJ can then parse the data from an nmap scan and store the results in the database. PBNJ uses Nmap as a scanning engine. It is available from

The benefits of PBNJ include:
  • The ability to configure automated Internal and external Scans,
  • A configurable and flexible querying language and alerting system,
  • The ability to parse Nmap XML output files
  • The ability to access Nmap output using a database (SQLite, MySQL or Postgres),
  • The ability to use distributed scanning with separate consoles and scan engines, and
  • PBNJ runs on Linux, BSD and Windows (Linux or UNIX are recommended over Windows in this instance).
ScanPBNJ default scan options
By default, ScanPBNJ runs an nmap scan using the command options; “nmap -vv -O -P0 -sS -p 1-1025”. This output is extremely verbose with operating system identification set. It will also not ping host by default. The options above run an nmap SYN scan over TCP ports between 1 and 1025.

It is possible to override the default options in ScanPBNJ using the “-a” switch. For instance to scan all TCP ports on the host the following command could be used;
ScanPBNJ –a “-A –sS –P0 -p 1-65535”
The other options of the previous command include using the SYN scan option, version scanning, not pinging the host and using operating system detection. Any of the standard nmap switches and scan types may be used.

The ability to query the ScanPBNJ results is provided using OutputPBNJ. OutputPBNJ uses a query yaml config file to perform queries against the information collected by ScanPBNJ. OutputPBNJ display the results of the scans using a variety of formats (such as csv, tab and html).

A number of predefined queries have been included with OutputPBNJ. These may be used to query the nmap results. The configuration file “query.yaml” contains default queries that have been defined on the system.

By default, there are only a small number of queries are limited. It is both possible to modify the existing default queries and/or to query the database directly. An ODBC connection to the database could also be used to load data from the database into another tool.

Understanding the Map
Networks change over time. New hosts, servers and services are added and removed. Network maps are not just Visio diagrams. It is nice to have a detailed visual map of what is running on the network, but a representation that can be automatically tested and used as a baseline is better.

The map of the network is the basis of being able to see what is authorized and what is not. Even if the systems on the network are not all tested and verified to be at an acceptable level of security, the map gives a way to get there.

Think of it this way, you have a system that is baselined, but has not been tested and verified. You already have a way to know two things:

  1. You have a starting point to check for unauthorized changes,
  2. You have a set of details about the system such as a list of services that are running on the system and which operating system it is using. 
From here it is easier to make a project to test systems over time. Grouping systems also help. If you have a series of DNS servers, they should be configured in a similar manner. Start with checking the “snowflakes” – why are they different. Each time that you recheck a system, it would then be added to the updated baseline. This way, the network becomes more and more secure over time.

Another way to see changes to the network is with a tool called ndiff.

Ndiff is a tool that utilizes nmap output to identify the differences, or changes that have occurred in your environment. Ndiff can be downloaded from The application requires that perl is installed in addition to nmap. The fundamental use of ndiff entails combining ndiff with a baseline file. This is achieved by using the “-b” option to select the file that is the baseline with the file to be tested using the “-o” option. The “-fmt” option selects the reporting format.

Ndiff can query the system’s port states or even test for types of hosts and Operating Systems using the “-output-ports” or “-output-hosts” options.

The options offered in ndiff include:
ndiff [-b|-baseline ] [-o|-observed ]
[-op|-output-ports ] [-of|-output-hosts ]
Ndiff output may be redirected to a web page:
ndiff –b base-line.txt –o tested.txt –fmt machine | ndiff2html > differences.html
The output file, “differences.html”, may be displayed in a web browser. This will separate hosts into three main categories:
  • New Hosts,
  • Missing Hosts, and
  • Changed Hosts.
The baseline file (base-line.txt) should be created as soon as a preliminary network security exercise has locked down the systems and mapped what is in existence. This would be updated based on the change control process. In this, any authorized changes would be added to the “map”. Any unauthorized changes or control failures with the change process will stand out as exceptions.

If a new host has appeared on the network map that has not been included in the change process and authorization, it will stand out as an exception. This reduces the volume of testing that needs to be completed.

Further, is a host appears in the “Changed Hosts” section of the report, you know what services have been added. This is again going to come back to a control failure in the change process or an unauthorized change. This unauthorized change could be due to anything from an internal user installing software without thinking or an attacker placing a trojan on the system. This still needs to be investigated, but checking an incident before the damage gets out of hand is always the better option.

No comments: