Automatically parse log files in folder.

[AlexGuld]

Hi

Love your work with GoAccess.

One thing, would it be possible to have it automatically parse log files in a defined folder?

That way log rotating will not mess-up parsing and you would not have to change the config every time you add a new log from a system. you would to have to tell what ever system it may be, to output logs to that folder in what ever format you have configured I GoAccess.

[allinurl]

Thank you! As far as folders, you should be able to run something like:

goaccess /path/folder/*.log --log-format=COMBINED -o /path/output/report.html

Let me know if that answers your question.

[grasdk]

@allinurl:
Thank you! As far as folders, you should be able to run something like:

goaccess /path/folder/*.log --log-format=COMBINED -o /path/output/report.html

Let me know if that answers your question.

Would it be possible to add this functionality to the docker-version also?

When providing /path/folder/*.log as the first "command" parameter to the docker image, the log says it can't find the file:

GoAccess - version 1.9.4 - Apr 18 2025 23:18:58
Config file: /etc/goaccess/goaccess.conf
Fatal error has occurred
Error occurred at: src/parser.c - read_log - 2387
Unable to open the specified log file '/path/to/*.log'. No such file or directory

and it is confirmed that there are multiple log files in /path/to/.

I think it is because when goaccess is used on the command line, the shell expands *.log to all existing .log files. However, if a log is rotated, I think the new log-file is not included, as it did not exist at the time of shell expansion of *.log. So maybe the original request here "monitoring a folder" might still be worth supporting in general and not only for docker?

[allinurl]

@grasdk, are you able to share the steps to replicate this on my end? It works fine with the existing files, but like you mentioned, it might be related to logrotate?

[grasdk]

@allinurl sure

To give you a bit of background first.

I'm an enthusiastic docker user and I view my containers as "running services", I am treating goaccess in the same way. For now, this means running with the argument --real-time-html, though I do not really need updates that frequently (every 1 to 60 seconds). Otherwise goaccess will generate a report and then exit, and the next update will be at next execution / container start.

So in short, what I experience happening is this (using your example above):

running 'goaccess logs/* --log-format=VCOMBINED --real-time-html' will make goaccess monitor the files already present in the 'logs' folder. i.e. log and log.1. However, if log.2 is added, goaccess will not see this file, until restarted.

This is the difference in having goaccess monitoring files, or a folder with files.

---

Were this to be solved, it would solve another problem:

When deploying goaccess in docker using the files as arguments, not pipe you cannot provide logs/* as a parameter. Or... to put it correctly... you can, but it takes it literally and shows this error: Unable to open the specified log file '/logs/*'. No such file or directory.

It is not possible to replicate with just docker run, because there, the shell will expand logs/*, but when you use docker compose or ansible docker_container module, and add the logs/* or similar wildcard, you get the mentioned error.

I hope this helps, otherwise I'll try to create a step by step replication guide.

[grasdk]

Recreation steps even with CLI:

1. Prepare a goaccess.conf file with the settings you need (/tmp/mygoaccesstest.conf). I recommend setting the index.html output to /logs, so it will show up in your mounted logs folder (you can make a sepate volume for it of course too, just trying to minimize the setup needed)
2. Prepare a folder with test logs (/tmp/testlogs)
3. Prepare a folder with extra test logs (/tmp/extralogs/) that can easily be distinguised from the logs from previous point (/tmp/testlogs)
4. Run the container like this:

   docker run -p 7890:7890 --rm -i -e LANG=$LANG \
   -v /tmp/mygoaccesstest.conf:/etc/goaccess/goaccess.conf \
   -v /tmp/testlogs:/logs allinurl/goaccess \
   --config-file=/etc/goaccess/goaccess.conf \
   --real-time-html \
   /logs/*.json.log
   

5. Observe that all your logs are being parsed and presented in the index.html, when you open it in a browser.
6. On your docker host do a cp /tmp/extralogs/* /tmp/testlogs
7. Observer that nothing changes in your index-html or the live data.
8. Kill the container
9. Start the container again with the command from 4
10. Observe that the logs previously places in /tmp/extralogs/ are now parsed and shown

Suggestion: One could pass a "--monitor-dir" that tells goaccess to check for new files automatically when running in --real-time-html mode.

---

My config file mygoaccesstest.conf: below (you will of course need to adjust logformat and maybe other settings):

# GoAccess main configuration
tz Europe/Copenhagen
addr 0.0.0.0
port 7890
ws-url ws://localhost/ws

# Report
html-report-title Full Caddy Web Log Report

# Log parsing
time-format %s
date-format %s

log-format "{\"level\":\"%^\",\"ts\":%x.%^,\"logger\":"\"%^\",\"msg\":\"%^\",\"request\":{\"remote_ip\":\"%^\",\"client_ip\":\"%h\",\"proto\":\"%H\",\"method\":\"%m\",\"host\":\"%v\","uri":\"%U\"},\"bytes_read\":%^,\"user_id\":\"%^\",\"duration\":%T,\"size\":%b,\"status\":%s}

# Realtime HTML
real-time-html true

# Output
output /logs/index.html

[allinurl]

Thanks for sharing these steps. I believe this could be easily solved using inotify. I’ll look into it. By the way, could you please open a new issue so that we can address this specifically related to Docker?

[grasdk]

@allinurl Sure, I can do that.

Before doing so:

1. The challenge in this thread isn't docker only as far as I can see. If you run goaccess /logs/*.log --real-time-html and later add another log-file, I believe it will not be part of the real time surveillance. Do you agree?
2. I have a workaround for docker, which also solves another challenge I had. I have posted it as "show and tell" here:
   Official goaccess container with cron deamon running goaccess at intervals of your choice #2837

So technically I have all that I need, but I will open a ticket, if you'd like :)