Analyze log files from several servers in real-time (update: whois, firewall)
First, we setup a machine to analyze the logs: # install missing packages (e.g. with Ubuntu 12.10) apt-get install netcat logtop # create a ramdisk with 1GB for storing the logs mkdir /ramdisk mount -t tmpfs -o nosuid,noexec,noatime,size=1G none /ramdisk # receive logs on port 8080 ncat --ssl -l -k 8080 > /ramdisk/access.log # open second terminal tail -f /ramdisk/access.log | logtop # clean up the ramdisk from time to time echo >/ramdisk/access.log Second, we setup the web servers: # install missing packages apt-get install netcat # send logs to analyzer-ip:8080 tail -f /var/log/apache2/access.log | ncat --send-only --ssl <analyzer-ip> 8080 Besides access.log, we can also monitor other log files using different port numbers. Let's start watching the requests coming in: Instead of analyzing each line separately, we can also aggregate all requests by client IPs: tail -f /ramdisk/access.log | awk -Winteractive '{print $1}' | logtop Or we can aggregate al...