Wednesday, 10 August 2011

Logging and retention

The following documents the logs that are created by a fairly normal company and the related time that these logs should be maintained.

Security and audit logs and the reporting on the systems that produce these logs must be centralised. It is necessary to have these records centralised so that systematic attacks across the servers can be pinpointed in a timely fashion. It also significantly lowers the risk of the logs being compromised.

The security logs must be backed up and kept for the duration of the audit period (e.g. 1 Year minimum). This can be extended with other requirements and should be deemed to be the absolute minimum requirement.

For instance, in Victoria, changes to the Crimes Act (1958) [Crimes (Document Destruction) Act 2006; Act No. 6/2006] have created “a new offence in relation to the destruction of a document or other thing that is, or is reasonably likely to be, required as evidence in a legal proceeding”. This act, punishable by indictment for a term of up to five years imprisonment affects anyone who destroys or authorises the destruction of any document that may be used in a legal proceeding (including potential future legal proceedings).

The distinction between correct and circumstantial evidence is that direct evidence categorically establishes the fact. Circumstantial evidence on the other hand is only suggestive of the fact. Authentication logs are generally accepted as direct evidence short of proof that another party used the access account. The consequence of this is that system logs can be critical to a legal case and hence must be maintained.

A section noting the “minimum document retention guidelines” has been included below.

Logging Options

The logging requirements for many companies are defined in this document. In order to meet these needs, the following type of questions will have to be able to be answered:
  •  “Can you tell me when Joe Bloggs logged in and logged out?”
  •  “Can you see if someone has tried to open this file”
  •  “Can you tell me who the last person to open this file was?”
  •  “We need to know if this person has been chatting (Office Communicator) with this person outside the company”
  •  “We need to know what these people have been chatting about (office communicator) both inside and outside the company”
  • “We need to know when people add external contacts to their communicator list”
  •  “Who had this IP address on
  •  The concern is leaking of information, more people resigning and things of that nature. What can you give me?
Three options that can provide the required solution have been noted.

Systems and Logs to maintain

Ideally, it would be best to be able to save a copy of all system and security log files. The requirements of running a business and efficiently maintaining client systems can make this difficult at best. As such, the following is set as a guide as to the minimum logging that most organisations should maintain.

In addition to normal system logs, an operating log must be maintained that records any significant events and action taken by the system operator or administrator. These logs should be maintained on a centralised and isolated server. Proper recording would indicate whether operators were following instructions for halts in programs, change control, etc.

There are more reasons to maintain logs than for security alone. Some of the reasons to maintain system and network logging include:
  •  Optimizing system and network performance; to record the actions of users
  •  Identifying security incidents, policy violations, fraudulent activities, and operational problems;
  •  Performing audits and forensic analyses;
  •  Supporting internal investigations;
  •  Establishing baselines; and
  •  Identifying operational trends and long-term problems.
Note: Windows Server 2008 supports syslog logging natively. Windows 2000 and Windows 2003 server require a third party logging client. A decision as to the state of logging on Windows 2003 and earlier systems (i.e. the need for an agent or alternatives needs to be made).

General Logging

Where possible, logs for devices should be enabled and sent to a central system.
The following should be used as guideline in storing logs. This table sets the times for local storage. The central logging system should maintain all logs for at least 12 months online and maintain an offline backup for longer periods.
Category Low Impact Systems Moderate Impact Systems High Impact Systems
How long to retain log data? 1 to 2 months 3 to 6 months 12 to 36 months
How often to rotate logs Optional (if performed, at least every week or every 25 MB) Every 6 to 24 hours, or every 2 to 5 MB Every 15 to 60 minutes, or every 0.5 to 1.0 MB
Transferring log data to the log management infrastructure - how frequently that should be done? Every 3 to 24 hours Every 15 to 60 minutes At least every 5 minutes
(If not automatic – e.g. syslog-ng)
How often log data needs to be analysed (through automated or manual means) Every 7 days Every 12 to 24 hours At least 6 times a day
Whether log file integrity checking needs to be performed for rotated logs Optional Yes Yes
Whether rotated logs need to be encrypted Optional Optional Yes
Whether log data transfers to the log management infrastructure need to be encrypted or performed on a separate logging network Optional Yes, if feasible Yes

The analysis of the logs should be automated and use a correlation system to make review simpler.
Entries from the logging system that are deemed to be of particular interest should be retained on the system and also transmitted to the log management infrastructure. Reasons for having the logs in both locations include the following:

· If either the system or infrastructure logging should fail, the other should still have the log data. For example, if a log server fails or a network failure prevents logging hosts from contacting it, logging to the system helps to ensure that the log data is not lost.

· During an incident on a system, the system’s logs might be altered or destroyed by attackers; however, usually the attacker will not have any access to the infrastructure logs. Incident response staff can use the data from the infrastructure logs; also, they can compare the infrastructure and system logs to determine what data was changed or removed, which may indicate what the attacker wanted to conceal.

System or security administrators for a particular system are often responsible for analysing its logs, but not for analysing its log data on infrastructure log servers. accordingly, the system logs need to contain all data of interest to the system-level administrators.

All logs could (should) be sent to a SQL database. Syslog, Syslog-ng and many other logging formats can be sent to MySQL, Informix and other database formats. This provides the capability to create custom queries to a database in order to retrieve selected logs.

Security Software

Most companies (and other organisations) use several types of network-based and host-based security software to detect malicious activity, protect systems and data, and support incident response efforts. Accordingly, security software is a major source of computer security log data. Common types of network-based and host-based security software include the following:

· Anti-malware Software. The most common form of antimalware software is antivirus software, which typically records all instances of detected malware, file and system disinfection attempts, and file quarantines. Additionally, antivirus software might also record when malware scans were performed and when antivirus signature or software updates occurred. Antispyware software and other types of antimalware software (e.g., rootkit detectors) are also common sources of security information.

· Intrusion Detection and Intrusion Prevention Systems. Intrusion detection and intrusion prevention systems record detailed information on suspicious behaviour and detected attacks, as well as any actions intrusion prevention systems performed to stop malicious activity in progress. Some intrusion detection systems, such as file integrity checking software, run periodically instead of continuously, so they generate log entries in batches instead of on an ongoing basis. This section includes Tripwire and a NIDS (such as Cisco Secure IDS or SNORT).

· Remote Access Software. Remote access is often granted and secured through virtual private networking (VPN). VPN systems typically log successful and failed login attempts, as well as the dates and times each user connected and disconnected, and the amount of data sent and received in each user session. VPN systems that support granular access control, such as many Secure Sockets Layer (SSL) VPNs, may log detailed information about the use of resources.

· Web Proxies. Web proxies are intermediate hosts through which Web sites are accessed. Web proxies make Web page requests on behalf of users, and they cache copies of retrieved Web pages to make additional accesses to those pages more efficient. Web proxies can also be used to restrict Web access and to add a layer of protection between Web clients and Web servers. Web proxies can keep a record of all URLs accessed through them. This would include the ISA systems and other Web Filtering systems.

· Vulnerability Management Software. Vulnerability management software, which includes patch management software and vulnerability assessment software, typically logs the patch installation history and vulnerability status of each host, which includes known vulnerabilities and missing software updates. Vulnerability management software may also record additional information about hosts’ configurations. Vulnerability management software typically runs occasionally, not continuously, and is likely to generate large batches of log entries.

· Authentication Servers. Authentication servers, including directory servers and single sign-on servers, typically log each authentication attempt, including its origin, username, success or failure, and date and time. This would include the Security and Application events from the Active Directory DNS servers.

· Routers and Switches. Routers may be configured to permit or block certain types of network traffic based on a policy. Routers that block traffic are usually configured to log only the most basic characteristics of blocked activity (as is the case generally). These logs should be increased to incorporate both allowed and dropped traffic. Switch traffic should log unusual events associated with VLANs and any violations of ACLs.

· Firewalls. Like routers, firewalls permit or block activity based on a policy; however, firewalls use much more sophisticated methods to examine network traffic. Firewalls can also track the state of network traffic and perform content inspection. Firewalls tend to have more complex policies and generate more detailed logs of activity than routers. In many companies, logging is disabled for both allowed and blocked traffic rendering the logs of little use. The Firewall logging needs to be changed to record both ALLOWED and DENIED Traffic to and from the network.

General Servers and Services

The following services used within many organisations create log files that are essential to maintain. The logs for all of these services need to be centralised and maintained for at least 12 months for all production systems that run them.
  • DNS (All active servers)
  • DHCP (All active servers)
  • LDAP
  • Network Devices
  1.  Firewalls (Cisco ASA, Load Balancers)
  2.  Cisco Router and Switch Logging (Both internal and external switches)
  •  SAMBA logging (Unix/Linux SMB Servers)
  •  Web server logs (Apache, IIS, TomCat, PHP etc)
  •  Proxy Logs (Websense, ISA)
  •  Tripwire logs (Servers)
  •  Mail Server logs (Sendmail, Exchange etc)
  •  Databases (MySQL, Informix, MSSQL)

Operating System (Server) logging

Operating systems (OS) for servers, and networking devices (e.g., routers, switches) usually log a variety of information related to security. The following security-related OS data should be maintained for at least 1 year (if not longer):

· System Events. System events are operational actions performed by OS components, such as shutting down the system or starting a service. Typically, failed events and the most significant successful events are logged, but many OSs permit administrators to specify which types of events will be logged. The details logged for each event also vary widely; each event is usually time-stamped, and other supporting information could include event, status, and error codes; service name; and user or system account associated with an event.

· Audit Records. Audit records contain security event information such as successful and failed authentication attempts, file accesses, security policy changes, account changes (e.g., account creation and deletion, account privilege assignment), and use of privileges. OSs typically permit system administrators to specify which types of events should be audited and whether successful and/or failed attempts to perform certain actions should be logged. The Windows Domain systems should be configured to log security events.

Client Systems

In general, the centralised logging of all client services would be too onerous for most companies. As such, it is recommended that organisations maintain the following client logs as a minimum:
  • · AV logging (Malware detected etc)
  • · Deep Freeze logs(and similar products)
  • · Authentication logs (from Domain Controllers)

Minimum Document Retention Guidelines

Australia/NZ USA UK
Basic Commercial Contracts 6 years after discharge or completion 4 years after discharge or completion 6 years after discharge or completion
Deeds 12 years after discharge A minimum of 6 years after discharge 12 years after discharge
Land contracts 12 years after discharge 6 years after discharge 12 years after discharge
Product liability A minimum of 7 years Permanent A minimum of 10 years
Patent deeds 20 years 25 years 20 years
Trade marks Life of trade mark plus 6 years Life of trade mark plus 25 years Life of trade mark plus 6 years
Copyright 75 years after author’s death 120 years after author’s death 50 years after author’s death
Contracts and agreements (government construction, partnership, employment, labour, etc.) A minimum of 6 years Permanent A minimum of 7 years
Capital stock and bond records 7 years after discharge Permanent 12 years after discharge

Any logging from systems associated with the above listed areas should be maintained for the period listed as a minimum.

Issues and reasons to take logging seriously

The maintenance of system and security logs is critical to ensuring the security of both a system and network. In addition, the failure to maintain an adequate audit trail can have severe repercussions. A couple of these are noted below.

Due Care and Due Diligence

Management is required to implement and preserve a suitable set of internal controls to check illegal and unscrupulous goings-on. A failure to implement due care and due diligence can constitute negligence.


Section 10 of the PCI-DSS states that:
Logging mechanisms and the ability to track user activities are critical in preventing, detecting, or minimizing the impact of a data compromise. The presence of logs in all environments allows thorough tracking, alerting, and analysis when something does go wrong. Determining the cause of a compromise is very difficult without system activity logs.”

This section of the PCI requirements then sets out the formats required for logging.

No comments: