Analyze log files from several servers in real-time (update: whois, firewall)

First, we setup a machine to analyze the logs:

# install missing packages (e.g. with Ubuntu 12.10)
apt-get install netcat logtop

# create a ramdisk with 1GB for storing the logs
mkdir /ramdisk
mount -t tmpfs -o nosuid,noexec,noatime,size=1G none /ramdisk

# receive logs on port 8080
ncat --ssl -l -k 8080 > /ramdisk/access.log
# open second terminal
tail -f /ramdisk/access.log | logtop

# clean up the ramdisk from time to time
echo >/ramdisk/access.log

Second, we setup the web servers:

# install missing packages
apt-get install netcat

# send logs to analyzer-ip:8080
tail -f /var/log/apache2/access.log | ncat --send-only --ssl <analyzer-ip> 8080

Besides access.log, we can also monitor other log files using different port numbers.

Let's start watching the requests coming in:

Instead of analyzing each line separately, we can also aggregate all requests by client IPs:

tail -f /ramdisk/access.log | awk -Winteractive '{print $1}' | logtop

Or we can aggregate all requests by URLs:

tail -f /ramdisk/access.log | awk -Winteractive '{print $7}' | logtop

Or filter all requests by a user agent:

# show only iPad
tail -f /ramdisk/access.log | grep iPad | logtop

# don't show Google, Msn, Bing
tail -f /ramdisk/access.log | grep -Ev 'Googlebot|bingbot|msnbot' | logtop

To extend the IP address with its owner, we write a small PHP script:

// whois.php
<?php
$fp = fopen('php://stdin', 'r');
while (!feof($fp)) {
  list($ip, $null) = explode(' ', fgets($fp), 2);
  if (!isset($whois[$ip])) {
    $who = shell_exec('whois '.escapeshellarg($ip));
    preg_match_all('!(?:descr|orgname|organization|country|owner).*:\s+(.+)!im',
      $who, $m);
    $whois[$ip] = ' '.str_pad($ip, 15).'  '.$m[1][0].' '.$m[1][1]."\n";
  }
  echo $whois[$ip];
}
fclose($fp);

and run:

apt-get install whois logtop php5-cli
tail -f /ramdisk/access.log | php whois.php | logtop

To send uptime messages every 5 seconds, we can use:

# @analyzer
ncat --ssl -l -k 8081 > /ramdisk/uptime.log
tail -f /ramdisk/uptime.log
# @webserver
while true; do echo -n `hostname`; uptime; sleep 5; done | ncat --send-only ...
# or free disk space, replace uptime with: df -h | grep sda

To configure a firewall on Ubuntu, we can use ufw:

# start firewall, block incoming connections
ufw enable
# allow incoming connections on port 80
ufw allow 80/tcp
# allow limited connections on port 22 (max. 6 connections in 30 seconds)
ufw limit 22/tcp
# show firewall status
ufw status verbose
ufw show listening

# block all new connections from IP 192.168.1.66
ufw insert 1 deny from 192.168.1.66

# remove blocking rule for IP 192.168.1.66
ufw delete deny from 192.168.1.66

Note: If your servers are connected by a secure network (e.g. VPN), you can skip --ssl and certificates.

Coming next: ncat with ssl-certificates