Friday, 8 February 2008

Generic Unix Log Parsing Tools

There are a number of requirements defined in the PCI-DSS for logging. These include:

5.2 Ensure that all anti-virus mechanisms are current, actively running, and capable of generating audit logs

Requirement 10: Track and monitor all access to network resources and cardholder data
Logging mechanisms and the ability to track user activities are critical. The presence of logs in all environments allows thorough tracking and analysis if something does go wrong. Determining the cause of a compromise is very difficult without system activity logs.

12.2 Develop daily operational security procedures that are consistent with requirements in this specification (for example, user account maintenance procedures, and log review procedures).

The following section of this post lists some of the MANY log parsing tools that are available. (Expect this to grow as I enter them one by one. That is this is an organic living post).

ACID (Analysis Console for Intrusion Detection) and BASE
A PHP-based analysis engine that searches and processes a database of incidents generated by security-related software such as intrusion detection systems and firewalls.

awk (and maybe SED)
A tool to really show your Unix prowess – or how difficult you can make things. From the author: "It is not a complete toolkit, but rather an approach that can be adapted for a variety of log analysis tasks."


Colorlogs basically color-codes logfiles for easier reading.

CyberSafe Event Log Analyst (CLA)
The Windows Server Resource Kit includes CyberSafe Log Analyst (CLA) which is a Microsoft Management Console (MMC) snap-in that lets you analyze the Security logs of the systems in a domain. CLA has prebuilt reports that provide useful views of security activity and allows you to design custom reports.

A Macintosh OSX generic logging utility/parser

Uses code to search for a large number of semi-static patterns in system logs returning only the lines that are not matched.


A Perl tool that can identify single-line log anomalies

SLCT (Simple Log Clustering Tool)

Code designed to identify patterns occurring in a logfile more frequently than a given threshold.

Installing, configuring and using swatch 2.2 to analyze log messages on systems running Solaris 2.x Setting up automatic alerting in your UNIX environment


A SyslogScan::Summary object will 'register' a series of SyslogScan::Delivery objects. All registered deliveries are grouped by sender and receiver e-mail addresses, and then added up. Three sums are kept: Total Bytes Recieved, Total Bytes Sent, and Total Bytes Broadcast.


A Python script that summarizes the contents of a syslog output file, by displaying each unique line once (timestamps are not included in the determination of line uniqueness). This script also provides the number of times each unique line appeared in the given file. Lines are displayed in the order they occur in the input file. This code is GPL'ed; it's written and maintained by Lars Wirzenius.


Monitors any plain text log file and identifies user-configurable events (not limited to syslog data). Application is well documented, and includes a sample startup script as well as a sample rule configuration file.


A system monitoring tool that allows administrators to monitor everything that's happening on a system in a very quick and comfortable way. It allows reading logfiles, checking devices or running status-gathering programs, translating all available data, and displaying results with filters and associated actions (including highlighting or lowlighting lines, hiding data, or taking actions on user-defined events.

1 comment:

Ryan said...

SEC is a good one too - Like swatch, except more versatile.