kelatebaru

First, we setup a machine to analyze the logs: # install missing packages (e.g. with Ubuntu 12.10) apt-get install netcat logtop # create a ramdisk with 1GB for storing the logs mkdir /ramdisk mount -t tmpfs -o nosuid,noexec,noatime,size=1G none /ramdisk # receive logs on port 8080 ncat --ssl -l -k 8080 > /ramdisk/access.log # open second terminal tail -f /ramdisk/access.log | logtop # clean up the ramdisk from time to time echo >/ramdisk/access.log Second, we setup the web servers: # install missing packages apt-get install netcat # send logs to analyzer-ip:8080 tail -f /var/log/apache2/access.log | ncat --send-only --ssl <analyzer-ip> 8080 Besides access.log, we can also monitor other log files using different port numbers. Let's start watching the requests coming in: Instead of analyzing each line separately, we can also aggregate all requests by client IPs: tail -f /ramdisk/access.log | awk -Winteractive '{print $1}' | logtop Or we can aggregate al...

using column stores instead of row based stores can reduce access logs from 10 GB to 130 MB of disk space reading compressed log files is 4 times faster than reading uncompressed files from hard disk column stores can speed up analytical queries by a factor of 18-58 Normally, log files from a web server are stored in a single file. For archiving, log files get compressed with gzip. A typical line in a log file represents one request and looks like this: 173.15.3.XXX - - [30/May/2012:00:37:35 +0200] "GET /cms/ext/files/Sgs01Thumbs/sgs_pmwiki2.jpg HTTP/1.1" 200 14241 "http://www.simple-groupware.de/cms/ManualPrint" "Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20100101 Firefox/12.0" Compression speeds up reading the log file: $start = microtime(true); $fp = gzopen("httpd.log.gz", "r"); while (!gzeof($fp)) gzread($fp, 8192); gzclose($fp); echo (microtime(true)-$start)."s\n"; // 26s $start = microtime(true); $fp = fopen(...