Using Syslog Effectively for Security Troubleshooting

Author: Douglas B. McKillip, P.E., Global Knowledge Instructor, CCSI, CCSP, CCIE #1851

Abstract

This paper examines the numerous ways that syslog messages can be used to enhance the secure deployment of an infrastructure of equipment from Cisco®. Syslog can not only be an essential auditing tool for network and administrative events, but also can be an effective troubleshooting tool. The utilization of syslog will be examined in the following key areas:

• Configuration change auditing
• Troubleshooting VPNs and Downloadable Access Lists
• Secure reporting using Secure Logging with TCP and Transport Layer Service

Configuration Change Auditing

Starting with Cisco IOS 12.3(4)T, an administrator can configure a router with a series of commands such that any subsequent configuration commands entered will be sent to syslog. Having a recorded audit trail of changes made can provide a valuable tool to troubleshoot possible unexpected outcomes. A sample router configuration dialog is shown below:

Router# config t
Enter configuration commands, one per line. End with CNTL/Z.
Router(config)# logging trap notifications
Router(config)# logging 10.10.2.10
Router(config)# archive
Router(config-archive)# log config
Router(config-archive-log-cfg)# logging enable
Router(config-archive-log-cfg)# logging size 1000
Router(config-archive-log-cfg)# hidekeys
Router(config-archive-log-cfg)# notify syslog

The purpose of the hidekeys command above is NOT to log the entry of passwords. The display below shows a Kiwi® Syslog Service Manager display, which captures some entered configuration commands. Note that these all appear with the %PARSER-5-CFGLOG prefix.

Previously, aaa commands were needed to audit configuration commands, and they needed a TACACS+ server to receive them. Unfortunately, this functionality has not yet been introduced for the ASA firewall.

Troubleshooting VPNs

Although the appropriate use of the applicable debug command can provide valuable output in troubleshooting VPN connectivity problems, a disadvantage of this approach is that the output cannot be sent to an external server. Instead, administrators either need to spot the key diagnostic message as it quickly scrolls by, or they need to capture the terminal output to disk. By using syslog instead, the information can be kept more easily for later analysis. In addition, syslog servers like the Cisco MARS® appliance can be configured with customized rules to generate incidents when raw messages due to VPN misconfiguration occur.

We will examine the following remote access VPN scenarios:

• Wrong IPSec VPN Client Group Name
• Wrong IPSec VPN Client Pre-shared Key
• Missing IPSec VPN Client Address Pool
• Bad User Password and Missing Address Pool for SSL AnyConnect Client

Troubleshooting Scenario

The following scenario will be used to illustrate syslog for use troubleshooting outlined above. A simple simulation of an external client connection to either a PIX (for IPSec) or an ASA (for AnyConnect®) was done from a laptop on the same external network as the security appliances, appearing in the traces as 192.168.1.2. The IPSec Group name is MYGROUP. Both the PIX and the ASA in the sections below were booted with OS8.0 code.

Wrong IPSec VPN Client Group Name

A partial output setting the log level to debug appearing on the console is shown below: (Note that the message which indicates the problem is indicated in bold font.)

logging on
pixfirewall(config)# %PIX-5-111008: User 'enable_15' executed the 'logging on' command.
%PIX-7-609001: Built local-host outside:192.168.1.2
%PIX-6-302015: Built inbound UDP connection 5 for outside:
192.168.1.2/500 (192.168.1.2/500) to identity:192.168.1.202/500
(192.168.1.202/500)
%PIX-7-713236: IP = 192.168.1.2, IKE_DECODE RECEIVED Message (msgid=0)
with payloads : HDR + SA (1) + KE (4) + NONCE (10) + ID (5) + VENDOR
(13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE
(0) total length : 853
%PIX-7-715047: IP = 192.168.1.2, processing SA payload
%PIX-7-715047: IP = 192.168.1.2, processing ke payload
%PIX-7-715047: IP = 192.168.1.2, processing ISA_KE payload
%PIX-7-715047: IP = 192.168.1.2, processing nonce payload
%PIX-7-715047: IP = 192.168.1.2, processing ID payload
%PIX-7-715047: IP = 192.168.1.2, processing VID payload
%PIX-7-715049: IP = 192.168.1.2, Received xauth V6 VID
%PIX-7-715047: IP = 192.168.1.2, processing VID payload
%PIX-7-715049: IP = 192.168.1.2, Received DPD VID
%PIX-7-715047: IP = 192.168.1.2, processing VID payload
%PIX-7-715049: IP = 192.168.1.2, Received NAT-Traversal ver 02 VID
%PIX-7-715047: IP = 192.168.1.2, processing VID payload
%PIX-7-715049: IP = 192.168.1.2, Received Fragmentation VID
%PIX-7-715064: IP = 192.168.1.2, IKE Peer included IKE fragmentation
capability flags: Main Mode: True Aggressive Mode: False
%PIX-7-715047: IP = 192.168.1.2, processing VID payload
%PIX-7-715049: IP = 192.168.1.2, Received Cisco Unity client VID
%PIX-4-713255: IP = 192.168.1.2, Received ISAKMP Aggressive Mode message 1 with unknown tunnel group name 'MYGROUP1'.

Wrong IPSec VPN Client Pre-Shared Key

While the above output quickly indicated that the configured group name was wrong, once this error is corrected, spotting a misconfigured pre-shared key requires more lines of output. Notice that the correctly configured group name, MYGROUP, appears regularly in the output of the log:

logging on
DOUGPIX-515E(config)# %PIX-5-111008: User 'enable_15' executed the
'logging on' command.
DOUGPIX-515E(config)# %PIX-7-713236: IP = 192.168.1.2, IKE_DECODE RECEIVED
Message (msgid=0) with payloads : HDR + SA (1) + KE (4) + NONCE
(10) + ID (5) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13)
+ VENDOR (13) + NONE (0) total length : 852
%PIX-7-715047: IP = 192.168.1.2, processing SA payload
%PIX-7-715047: IP = 192.168.1.2, processing ke payload
%PIX-7-715047: IP = 192.168.1.2, processing ISA_KE payload
%PIX-7-715047: IP = 192.168.1.2, processing nonce payload
%PIX-7-715047: IP = 192.168.1.2, processing ID payload
%PIX-7-715047: IP = 192.168.1.2, processing VID payload
%PIX-7-715049: IP = 192.168.1.2, Received xauth V6 VID
%PIX-7-715047: IP = 192.168.1.2, processing VID payload
%PIX-7-715049: IP = 192.168.1.2, Received DPD VID