Official goaccess container with cron deamon running goaccess at intervals of your choice

[grasdk]

Background

My usecase for goaccess is primarily to satisfy my curiousity about who tries to access my self hosted services. As I am an enthusiastic docker user, I started using the goaccess docker image with the --real-time-html parameter.

After getting everything setup and studying the report (really looking forward to the filtering options btw.), I realized that --real-time-html is actually overkill for my occasional curiousity. A daily update or perhaps 4 updates per day would be enough.

That's where the idea below came from.

Periodic update of static report

Using goaccess as a pure CLI tool, one would create a cron-job to generate a periodic html-report. This can be done both with goaccess installed and with goaccess in a container. The crontabs entry could look like this:

0 */6 * * * /bin/goaccess $GOACCESS_ARGS

or (as docker cli - WARNING, NOT TESTED!)

0 */6 * * * docker run --rm -i -v ./report.html:/report.html -e LANG=$LANG allinurl/goaccess $GOACCESS_ARGS

with GOACCESS_ARGS containing the proper arguments for goaccess.

---

Goaccess container with cron daemon

If your philosophy is 'a container for the job', and you want 'the job' to be 'generating a report at regular intervals', the container needs to do this, not the docker host.

Luckily crond and a few other tools exist in the official goaccess container image. Utilizing it becomes a matter of overriding the entrypoint, replacing it with a shell, and then run a command that sets up a cron job to run goaccess and then run crond in the foreground.

Technical solution

All is done in one (rather long) docker run command:

docker run \
--name goaccess \
--hostname goaccess \
--restart unless-stopped \
-v /path/to/webfolder:/srv/report/ \
-v /path/to/config/goaccess.conf:/etc/goaccess/goaccess.conf:ro \
-v /path/to/crontabs/root:/var/spool/cron/crontabs/root:ro \
-v /path/to/logs:/logs:ro \
-e TZ="Europe/Copenhagen" \
-e GOACCESS_ARGS="--config-file=/etc/goaccess/goaccess.conf /logs/*.json.log" \
--entrypoint sh \
allinurl/goaccess \
-c "/bin/goaccess \$GOACCESS_ARGS && crond -f -l 8"

Breakdown in words:
Run the container, providing paths for the html-report, the config-file, the crontab file and the log files. Set the arguments for goaccess in an environment variable (makes it available both in shell in and cron). Override the entrypoint and run both goaccess and crond at container start.

Breakdown per line:

• Line 2: Container name
• Line 3: Hostname from the container's viewpoint
• Line 4: Restart the container unless stoppet (so when the host restarts, the container will start when the docker daemon starts)
• Line 5: host path for html-report folder:in-container path for html-report folder (must match goaccess.conf output value)
• Line 6: host path for goaccess.conf:in-container path for goaccess.conf folder
• Line 7: host path for crontab-file:in-container path for crontab file (NB: MUST be owned by root and have mode 600)
• Line 8: host path for log-file folder: in-container path for log-file folder
• Line 9: Timezone environment variables
• Line 10: Arguments for goaccess, specifying the config file and the logs files in the in-condtinaer log folder to parse (see line 8)
• Line 11: Override entrypoint to be the bourne shell (sh) baked into the container image instead of /bin/goaccess
• Line 12: The container image
• Line 13: The command to start when the container starts (which runs the sh shell - see line 11)
• Line 14: First run goaccess with the given args (see line 10), then (after &&) run crond the cron daemon in the foreground with low logging

The content of /path/to/crontabs/root decides what the cron daemon runs. The content to run goaccess with the arguments every 6 hours looks like this:

0 */6 * * * /bin/goaccess $GOACCESS_ARGS

That's how I use the official goaccess docker image to host and run a recurring generation of the html report.

As a side note, it parses *.json.log in my case every time, so if log files are added or removed, they will be included or left out at each invocation of goaccess that happens via the cron daemon. For me, this is a workaround for the problem described here: #2465 (comment)

[grasdk]

docker-compose, corresponding to the CLI above:

goaccess-with-cron.yaml:

version: "3.8"

services:
  goaccess:
    image: allinurl/goaccess
    container_name: goaccess
    hostname: goaccess
    restart: unless-stopped
    volumes:
      - /path/to/webfolder:/srv/report/
      - /path/to/config/goaccess.conf:/etc/goaccess/goaccess.conf:ro
      - /path/to/crontabs/root:/var/spool/cron/crontabs/root:ro
      - /path/to/logs:/logs:ro
    environment:
      TZ: Europe/Copenhagen
      GOACCESS_ARGS: "--config-file=/etc/goaccess/goaccess.conf /logs/*.json.log"
    entrypoint: sh
    command: >
      -c "/bin/goaccess --config-file=/etc/goaccess/goaccess.conf /logs/*.json.log &&
          crond -f -l 8"

run with docker-compose -f goaccess-with-cron.yaml up

[grasdk]

Auto downloading geo database mmdb file.

Now that cron is there, why not automate the mmdb geo database download inside the container?

The attached shell script will download the latest mmdb file from www.maxmind.com. It requires that you have a maxmind account (free tier is fine) and that you set the two environment variables MAXMIND_ACCOUNT_ID and MAXMIND_LICENSE_KEY with the info from your profile page on maxmind.com.

The shell script is here (download and rename to .sh)
update-geolite.txt

Short breakdown:

1. It downloads a statically compiled AMD64 version of curl if curl is not present (which it isn't in the official goaccess container)
2. It then checks if there is a new version of the mmdb database of choice (see the DB_TYPE variable in the script)
3. If there is a new version it downloads it, unpacks it and places it in /usr/share/geoip (configure your goaccess.conf accordingly)

Note. If you are running the container on ARM64, you need to switch the BINARY_URL variable in the script.

Now, you can add a weekly download of the mmdb (if there is an update) by mounting the script and adding it to the crontab-file.

Crontab-file (runs goaccess every 6 hours and updates the mmdb every wednesday at 18:30)

0 */6 * * * /bin/goaccess $GOACCESS_ARGS
30 18 * * 3 /usr/share/geoip/update-geolite.sh

Adding the script to the container, can be done with volumes. Running it at container start is prepending it to the startup commands...

CLI

-v /path/to/config/update-geolite.sh:/usr/share/geoip/update-geolite.sh:ro \
...
-e MAXMIND_ACCOUNT_ID=xxx
-e MAXMIND_LICENSE_KEY=yyy
...
-c "/usr/share/geoip/update-geolite.sh && /bin/goaccess \$GOACCESS_ARGS && crond -f -l 8"

docker compose

    ...
    volumes:
      - /path/to/config/update-geolite.sh:/usr/share/geoip/update-geolite.sh:ro
    ...
    environment:
      MAXMIND_ACCOUNT_ID: xxx
      MAXMIND_LICENSE_KEY: yyy
    command: >
      -c "/usr/share/geoip/update-geolite.sh &&
          /bin/goaccess --config-file=/etc/goaccess/goaccess.conf /logs/*.json.log &&
          crond -f -l 8"

So the complete compose file looks like this: goaccess-with-cron-and-geoip.yaml:

version: "3.8"

services:
  goaccess:
    image: allinurl/goaccess
    container_name: goaccess
    hostname: goaccess
    restart: unless-stopped
    volumes:
      - /path/to/webfolder:/srv/report/
      - /path/to/config/goaccess.conf:/etc/goaccess/goaccess.conf:ro
      - /path/to/crontabs/root:/var/spool/cron/crontabs/root:ro
      - /path/to/logs:/logs:ro
      - /path/to/config/update-geolite.sh:/usr/share/geoip/update-geolite.sh:ro
    environment:
      TZ: Europe/Copenhagen
      GOACCESS_ARGS: "--config-file=/etc/goaccess/goaccess.conf /logs/*.json.log"
      MAXMIND_ACCOUNT_ID: xxx
      MAXMIND_LICENSE_KEY: yyy
    entrypoint: sh
    command: >
      -c "/usr/share/geoip/update-geolite.sh &&
          /bin/goaccess --config-file=/etc/goaccess/goaccess.conf /logs/*.json.log &&
          crond -f -l 8"

run with docker-compose -f goaccess-with-cron.yaml up