Working with Nginx logs from bash
Earlier today on the IndieWeb IRC Channel Aaron was talking about writing some code to parse his Nginx logs to get a list of 404s and I commented that it could be done easily enough by setting Nginx to emit JSON formatted logs.
Now that is definitely an option, but then I remembered that I had written a quick-n-dirty parser to spit out a JSON blob per logline and that with the amazingly handy jq tool the JSON could quickly be filtered or acted on!
So I took a break from work to refactor the quick-n-dirty parser to use apache-log-parser to increase it's flexibility (the old version was hardcoded for a specific log line format) and added some helpful comments.
logs.py will generate via stdout a stream of JSON for each line of the log file:
./logs.py < /var/log/nginx/bear.im.log | jq '. | { method, status, request_first_line }'