Scheduling awstats report generation

We've looked at running awstats reports, but only manually. Let's automate report generation so all you need to worry about is looking at those sweet, sweet numbers.


Automating awstats

In the previous article in this series we set up awstats for your site and ran an update of the reports manually. That's all well and good, but the command to update is big and ugly, and it would be kind of a pain to have to run it every time you want to view updated stats.

Fortunately Linux is chock full of ways to automate stuff. One way is to create a cron script to do all the updating, but since we're shooting for simple we should use a tool that's already processing your web logs on a regular basis. We can piggyback our updates onto logrotate's regular rotation tasks.

Scheduling reports with logrotate

With this approach we'll take advantage of the fact that logrotate is already performing regular log rotation for your domain. I mean, you do have your logs rotating automatically, right?

If not, visit this article series on logrotate and follow the directions there to set up log rotation for your virtual host. Log rotation keeps those logs from becoming giant disk-space-eating behemoths, so it's a very good idea. Really, do it now. I'll wait.

Now that we're certain you have logrotate managing your web logs, let's add a step to what logrotate does when it performs the rotation.

Editing the logrotate entry

Let's look at the logrotate.d file for a virtual host that's just a modification of the default entry for apache on Ubuntu:

/home/demo/public_html/example.com/logs/*.log {
    weekly
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 640 root adm
    sharedscripts
    postrotate
        if [ -f "`. /etc/apache2/envvars ; echo ${APACHE_PID_FILE:-/var/run/apache2.pid}`" ]; then
            /etc/init.d/apache2 reload > /dev/null
        fi
    endscript
}

Don't worry too much about most of those entries if you haven't seen them before. There are only a couple important bits to note here.

First, take a look at the "weekly" directive in that logrotate file. That's okay for simple log rotation, but you probably want your web traffic stats updating daily. In that case you'd want to change that to "daily" so the rotate script runs more frequently, and possibly modify the "rotate 52" entry to keep more archived log files.

Next we look at what we want to add to the logrotate process. See that "postrotate" block up there? Don't worry about what's inside, just note that it's there. What that does is run some stuff after the log rotation is done (in this case, tell apache to reload if it's running). The reason we're looking at it is because there's also a "prerotate" directive that we can use to have awstats run through a log file before it's rotated.

The "prerotate" directive should run the stats update and generate the reports. And it should run just like the command we ran to get our reports. We would create a prerotate block like:

prerotate
    /usr/local/awstats/tools/awstats_buildstaticpages.pl -update -config=www.example.com -dir=/home/demo/public_html/example.com/webstats -awstatsprog=/usr/local/awstats/wwwroot/cgi-bin/awstats.pl > /dev/null
endscript

The "> /dev/null" bit redirects the normal output of the command so it doesn't get sent to the console or emailed to root.

Inserted into the existing logrotate file for our virtual host, the whole thing would look like:

/home/demo/public_html/example.com/logs/*.log {
    daily
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 640 root adm
    sharedscripts
    prerotate
        /usr/local/awstats/tools/awstats_buildstaticpages.pl -update -config=www.example.com -dir=/home/demo/public_html/example.com/public/webstats -awstatsprog=/usr/local/awstats/wwwroot/cgi-bin/awstats.pl > /dev/null
    endscript
    postrotate
        if [ -f "`. /etc/apache2/envvars ; echo ${APACHE_PID_FILE:-/var/run/apache2.pid}`" ]; then
            /etc/init.d/apache2 reload > /dev/null
        fi
    endscript
}

And with that, every time the logs get rotated, the web stats get updated too.

Why use logrotate?

Instead of using logrotate to run the web stats update we could just schedule the stats to update through cron. So why insert the commands into logrotate?

The main reason is accuracy. By sticking a web stats update into the log rotation process we make sure that awstats is looking at log entries up until the last possible moment, when the log is rotated. If you have the web server reloading instead of restarting there might be a couple log entries missed (since the old log file would still be used as the old connections finish). The information lost in that case is negligible and not worth the downtime required to fully restart the web server.

If you want the web traffic reports to update more often than the web logs are rotated, you can use cron to run the update and report-building scripts on a more frequent schedule (like hourly). Awstats will recognize log entries that have already been processed and skip them when analyzing the data.

Monthly reports

Something you'll notice if you leave awstats running the way we've set it up is that the reports being generated only go into detail for the current month. Dividing the information up by month is a decent interval, but you may want to look at previous months in detail instead of only the current one.

So it's entirely optional, but if you'd like to keep monthly reports around it's really just a matter of tailoring a shell script to fit your site.

A cron script

We'll name the script after the awstats config file for the site in question and put it in the cron monthly directory. Using "www.example.com", we would create the file:

/etc/cron.monthly/awstats.www.example.com

Inside the file put the following script:

#!/bin/sh
#
# Run the awstats build script to generate a report for last month.
#

# Modify these 3 variables for your environment:
#
# The location of the awstats installation
AWSTATSDIR=/usr/local/awstats

# Your main domain, as you reported it to awstats
DOMAIN=www.example.com

# The directory where you're storing the reports for this domain
REPORTDIR=/home/demo/public_html/example.com/public/webstats

LASTMONTH=`date -d "last month" +%B`
LASTMONTHNUM=`date -d "last month" +%m`
LASTMONTHDIR=$REPORTDIR/$LASTMONTH

mkdir -p $LASTMONTHDIR
cp -Rf $AWSTATSDIR/wwwroot/icon $LASTMONTHDIR/awstatsicons

$AWSTATSDIR/tools/awstats_buildstaticpages.pl -month=$LASTMONTHNUM -update -config=$DOMAIN -dir=$LASTMONTHDIR -awstatsprog=$AWSTATSDIR/wwwroot/cgi-bin/awstats.pl > /dev/null

Change the values of "AWSTATSDIR", "DOMAIN", and "REPORTDIR" to match your environment and site.

After saving the file make your new script executable:

sudo chmod a+x /etc/cron.monthly/awstats.www.example.com

You can even run that script now to test it (and to get a detailed report for last month's data, if you have some). If you do, be sure to run it using sudo.

What it does

When the script runs it creates (if it doesn't already exist) a directory named after last month. If the current month is September, the directory will be named "August" (or the equivalent for your machine's locale setting). The awstats icons directory will be copied there, then a report will be generated for last month and put in the month's directory.

To look at a monthly report you'll use a similar URL to viewing the current report, but you'll insert the name of the month you want to view in front of the page name. The first letter of the monthly directory names is capitalized, so it will have to be capitalized in the URL used to visit a monthly report as well.

To take our example URL from earlier and use it to look at the August statistics we would change it to:

http://www.example.com/webstats/August/awstats.www.example.com.html

Keeping more monthly reports

Note that the way the script is written the monthly reports get replaced every year (since they're only made available by month name, not by year).

If you want monthly reports to never get overwritten you can modify the script to add the year to the directory names. Change the line that defines "LASTMONTH" above to something like:

LASTMONTH=`date -d "last month" +%B-%Y`

If the month were September, the above line would cause the script to use the directory name "August-2010".

Further reading

You have a solid but basic installation of awstats now. If you want to get more out of awstats there are a few advanced features you can look into.

For starters, you can browse the awstats documentation online.

If you want to generate reports on the fly you can do so by setting up the main awstats.pl script to run as a CGI script from a web browser. It would be a good idea to still run the stats update through a schedule. You'll want to be familiar with using CGI with a web server, and be aware of the risks that can be involved (for both performance and security). Then check the awstats install docs and take a look at the script they provide for automating aspects of that configuration.

If you want to extend awstats there are plugins available on the project's web site. A couple of the more interesting ones let awstats determine the country of origin of visitor IP addresses without launching a lot of cumbersome DNS lookups.

With a little tweaking and help from the documentation, awstats can also be used to build reports for mail and FTP servers.

Digging through all the options in the config file will give you an idea of what sorts of changes you can make to reports and their formatting.

Summary

You should have a nice setup now, with awstats generating reports automatically. Remember that you can run reports for more than one domain, you'll just need to set up separate configs and automated schedules for them.

  • -- Jered

Article Comments:

kevin commented Wed Jan 26 20:06:13 UTC 2011:

Thanks for the great articles, The link in this one needs update. "article series on logrotate" is linking to itself. I think you wanted it to goto http://www.slicehost.com/articles/2010/6/30/logs-not-the-big-and-heavy-and-wood-kind

Jered commented Thu Jan 27 15:30:20 UTC 2011:

Thanks Kevin! Fixed the link to point to the logrotate articles. I appreciate you pointing that out.

sysadmin commented Tue Oct 16 13:43:11 UTC 2012:

Awstats is really cool script. I'm using it for several years already. Just don't understand why are you doing that monthly stats tricks if awstats itself is capable of showing any previous month stats (there are drop-down lists with year and month at the top of the page).

debian cccam server commented Sat May 04 12:57:41 UTC 2013:

Greetings from Idaho! I'm bored at work so I decided to browse your website on my iphone during lunch break. I love the knowledge you provide here and can't wait to take a look when I get home. I'm amazed at how quick your blog loaded on my mobile .. I'm not even using WIFI, just 3G .. Anyhow, superb site!

my website :: debian cccam server

garage door repair commented Wed Jul 23 09:07:58 UTC 2014:

I have read so many content on the topic of the blogger lovers except this piece of writing is genuinely a good paragraph, keep it up.

my blog post - garage door repair

wayne dalton garage door opener manual commented Wed Jul 23 09:39:29 UTC 2014:

It's really a great and useful piece of info. I am satisfied that you shared this useful info with us. Please keep us up to date like this. Thank you for sharing.

Feel free to visit my site; wayne dalton garage door opener manual

garage door repair commented Wed Jul 23 09:42:01 UTC 2014:

Good post. I learn something new and challenging on sites I stumbleupon everyday. It will always be exciting to read content from other authors and use something from their websites.

Here is my web blog - garage door repair

wireless garage door opener commented Wed Jul 23 09:43:31 UTC 2014:

Write more, thats all I have to say. Literally, it seems as though you relied on the video to make your point. You clearly know what youre talking about, why waste your intelligence on just posting videos to your weblog when you could be giving us something informative to read?

Here is my web site; wireless garage door opener

garage door repair commented Wed Jul 23 09:53:27 UTC 2014:

Hello there! This is my first visit to your blog! We are a team of volunteers and starting a new initiative in a community in the same niche. Your blog provided us valuable information to work on. You have done a outstanding job!

Feel free to visit my blog - garage door repair

craftsman garage door opener parts commented Wed Jul 23 09:56:15 UTC 2014:

Hi there friends, its wonderful post concerning tutoringand completely defined, keep it up all the time.

my website; craftsman garage door opener parts

garage door repair commented Wed Jul 23 09:59:39 UTC 2014:

Great items from you, man. I've consider your stuff previous to and you're just extremely great. I actually like what you've bought here, really like what you are stating and the best way through which you are saying it. You're making it entertaining and you still care for to stay it smart. I can't wait to read far more from you. This is actually a tremendous web site.

My blog post garage door repair

garage door repair commented Wed Jul 23 12:36:30 UTC 2014:

It's hard to find well-informed people on this subject, but you seem like you know what you're talking about!

Thanks

Feel free to surf to my blog :: garage door repair

garage door opener commented Wed Jul 23 12:37:13 UTC 2014:

Thanks in support of sharing such a fastidious opinion, post is pleasant, thats why i have read it completely

Check out my site ... garage door opener

wayne dalton garage door opener manual commented Wed Jul 23 12:44:32 UTC 2014:

You should take part in a contest for one of the greatest sites on the web. I am going to highly recommend this site!

Feel free to visit my blog: wayne dalton garage door opener manual

garage door repair commented Wed Jul 23 12:44:58 UTC 2014:

I do not know whether it's just me or if everybody else experiencing issues with your blog. It looks like some of the text within your posts are running off the screen. Can someone else please comment and let me know if this is happening to them as well?

This could be a issue with my web browser because I've had this happen before.

Many thanks

Here is my weblog ... garage door repair

replacement garage door opener remote commented Wed Jul 23 12:49:56 UTC 2014:

What's up to every body, it's my first pay a quick visit of this website; this website consists of remarkable and really excellent data in favor of readers.

Feel free to surf to my web page replacement garage door opener remote

wayne dalton garage door opener manual commented Wed Jul 23 16:35:45 UTC 2014:

Incredible quest there. What happened after? Take care!

my web blog: wayne dalton garage door opener manual

lift weights commented Sun Aug 17 01:03:53 UTC 2014:

I really like reading through an article that can make men and women think. Also, thank you for allowing for me to comment!

Look at my web page; lift weights

garage door repair commented Wed Aug 20 06:32:39 UTC 2014:

Great delivery. Outstanding arguments. Keep up the amazing effort.

My weblog; garage door repair

donbt commented Wed Aug 27 09:11:12 UTC 2014:

Excellent Thank you very much !

http://google.com commented Thu Aug 28 06:22:55 UTC 2014:

What i do not understood is in reality how you are no longer actually much more well-favored than you might be now.

You are so intelligent. You already know thus considerably in relation to this topic, produced me in my view believe it from numerous various angles. Its like men and women don't seem to be interested except it is something to accomplish with Woman gaga! Your individual stuffs nice. All the time care for it up!

My blog google.com (http://google.com)

Want to comment?


(not made public)

(optional)

(use plain text or Markdown syntax)