My-Tiny.Net :: Networking with Virtual Machines

Logfile Locations

---

The two key configuration files for logging are /etc/syslog.conf and /etc/logrotate.conf. Both of these have a lot of embedded documentation now to explain how they work; our application configurations have also been changed to be consistent with this scheme.

The important thing is where to look for log messages from various applications. Most of this summary is from the bottom of /etc/syslog.conf, but some comes from the configuration files for applications:

Application	Logs to	See
DHCP Server & DNS	/var/log/dnsmasq.log	syslog.conf
DHCP Client	/var/log/dhcpcd-log	dhcpcd.conf
Mail Transport	/var/log/maillog	syslog.conf
Mailbox Delivery	/var/log/dovecot.LDA	dovecot.conf
Mailbox Access	/var/log/dovecot.IMAP	dovecot.conf
Webserver (Errors)	/var/log/httpd.ERRORS	monkey.conf
Webserver (PHP)	/var/log/httpd.PHP	php.ini config_squirrel_logger.php
LDAP Server	/var/log/slapd.log	syslog.conf
stunnel	/var/log/tunnels.log	syslog.conf
xinetd, OpenVPN	/var/log/acpid.log	syslog.conf
ssh, scp, sftp	/var/log/authinfo.log	syslog.conf
sudo, adduser	/var/log/secure	syslog.conf
iptables	/var/log/syslog   or /var/log/messages	syslog.conf
Other and less important messages	... various ...	syslog.conf

Perhaps it goes without saying, but let's say it anyway: not all of these will be used on all of the hosts. For example, only the Gateway and MailHost will have anything meaningful in maillog; the important one for DHCP on the Gateway is dnsmasq.log, but on all of the other hosts the important file is dhcpcd-log since they are clients and the Gateway is the server.

Be Patient: when you get more familiar with the roles, you will know where to look.

It is important to understand log levels and facilities if you want to get detailed information for debugging - but the default settings should be adequate for our purposes.

logrotate

The purpose of /etc/logrotate.conf is to keep logfiles from getting too big; when the file gets to a certain size (20k for a TinyNet) a new one is created and up to 2 (for a TinyNet) old files are kept. The older logfiles will have names ending with .1, .2, etc. which can safely be deleted.

On TinyNet hosts, logrotate is called from a script in /etc/cron.hourly to check for files that have grown too big, and by /etc/rc.d/rc.local_shutdown to create new logfiles for kernel messages when the system starts up.

A helpful reference here is the Logrotate Cheat Sheet at https://www.jamescoyle.net/category/cheat-sheets/676-logrotate-cheat-sheet

Centralized syslog Server

A centralized log server that receives and processes log messages from multiple sources can be of great value to a system administrator. When a hacker breaks into a system, the first thing she will want to do is to hide and stay hidden. To do this, most hackers will modify system logs to remove evidence of their existence. If a break-in occurs and you want to track the cracker down, the system administrator will first check the log files for evidence. With a machine on the network that collects logs from other machines on the network, we can be sure log files haven't been tampered with. Even if a hacker breaks into one system (the MailServer for example) and tampers with the logs, we still have another set of log files that have not been altered.

You can have the syslog daemon accept remote messages with two steps:

/etc/rc.d/rc.syslog stop

/etc/rc.d/rc.syslog listen

When you check the output of netstat -tulpn you should see it listening on udp-port 514.

To make it permanent, put those two commands in /etc/rc.d/rc.local just above the comment line
# start monkey httpd

To have a client system send log entries to the syslog server, you need to edit /etc/syslog.conf and change the log target from the name of a file to @server.host.name (using the name of your syslog server, of course).

One syslog configuration line can normally specify only a single destination.*
To keep the local logfile and add the syslog server we have to duplicate the line for the extra destination.
* Oddly enough, "a comma-separated list of users" is considered a single destination, but list of files is not.

On the server side, messages are processed according to the facility + priority selectors without regard to origin. The full domain name will be logged as part of each message received: to shorten it, check the -h and the -s switches in the syslogd man page, and make the appropriate changes in /etc/rc.d/rc.syslog. Or we can use multitail with a simple regex, as in the example below ...

Final Note: One feature of syslog is that it condenses repeated messages into a single message:

Last message repeated n times

With the standard linux version there is no way to control this, BUT Slackware uses the FreeBSD version where this CAN be controlled by uncommenting one line at the top of /etc/rc.d/rc.syslog

# Uncomment the following line to use the -c option to syslogd.  This will
# disable suppressing repeated messages, which may be useful with tools that
# parse the logs, such as fail2ban:
SYSLOGD_OPTIONS="-c "  

Do this on both the server and the client to see all messages.

Simple Example

First, get syslog on the Gateway to listen for messages. (Be sure to check it is running!)

Then on the MailHost add a line in /etc/syslog.conf so messages will be saved locally AND sent to the Gateway

    # root login, adduser
    authpriv.info   -/var/log/secure
    authpriv.info   @gw.tinynet.edu

Restart syslog, and then login on new VT a couple of times with [Alt] F2 and [Alt] F3.

Now check /var/log/secure on the Gateway - it should have entries like these

Oct 27 21:32:17 gw-abc login[1242]: invalid password for 'root'  on '/dev/tty1'
Oct 27 21:33:58 gw-abc login[1242]: ROOT LOGIN  on '/dev/tty1'
Oct 28 01:17:23 gw-abc login[1243]: ROOT LOGIN  on '/dev/tty2'
Oct 28 01:31:35 mail-nc.net-c.tinynet.edu login[1484]: ROOT LOGIN  on '/dev/tty2'
Oct 28 01:31:44 mail-nc.net-c.tinynet.edu login[1445]: ROOT LOGIN  on '/dev/tty3'

Let's clean it up! With multitail we can use -ke with a regular expression to omit parts of lines from the output.
This example shows how to drop the domain part if it is net-a or net-b or net-c

multitail -ke ".net-[abc].tinynet.edu" /var/log/secure

Using multitail twice on on a single file, we can separate one set of entries from the others
using -e to select some lines in the first window and -ev to leave them out of the other window:

multitail -ev "gw-abc" /var/log/secure -e "gw-abc" /var/log/secure

You can find all the details in the Logfile Analysis with Regular Expressions section of the Utilities :: Multitail page on the menu.

---

---