Shipping your logs to a central server is usually a good thing to do. For a large number of servers it provides a better overview, and no matter the numbers of servers a secondary log location can be helpful in figuring out why something bad happened to a server.
My two VPS nodes are now using loggly as a remote (TLS) syslog server. I’m even allowed do that for free, as long as I don’t upload more than 200 MB of logs per day, nor want the log data to be retained for more than a week. Not that I would mind paying a bit for a longer retention period. It’s just that their pricing feels a bit steep, given that I currently log less than a megabyte per day.
(Yes, I do realize that I’m not their obvious target audience.)
Already running rsyslog I decided to follow loggly’s rsyslog instructions, which did a pretty good job of explaining the additional configuration needed. The one thing I did miss in those instruction were a discussion on queue setting, which very much will matter when loggly’s servers for one reason or another becomes unavailable. By default rsyslog only queues a limited number of entries in memory, so for additional resilience I explicitly enabled a disk assisted queue, based on the rsyslog reliable forwarding guide.
Want to test the queuing? Just put appropriate iptables rules in place, and then speed up time by using logger(1) to pipe lots and lots of entries into syslog.
All in all, the following loggly specific configuration seems to do the trick for me.
# /etc/rsyslog.d/loggly.conf $DefaultNetstreamDriverCAFile /etc/ssl/loggly/loggly_full.crt $DefaultNetstreamDriverCertFile /etc/ssl/loggly/dummy-halleck.crt $DefaultNetstreamDriverKeyFile /etc/ssl/loggly/dummy-halleck.key $ActionSendStreamDriver gtls $ActionSendStreamDriverMode 1 $ActionSendStreamDriverAuthMode x509/name $ActionSendStreamDriverPermittedPeer *.loggly.com $ActionQueueType LinkedList $ActionQueueFileName loggly $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on *.* @@logs.loggly.com:<assigned port>
If you are running Ubuntu, and Launchpad bug #1075901 have yet to be fully fixed, you might manually need to chown syslog:syslog /var/spool/rsyslog/. While you are at it, also install the needed rsyslog-gnutls package.
Worth mentioning is that loggly provides the option of archiving your logs to a S3 bucket. Given my modest log volumes the cost of doing that ought to be pleasantly close to zero.
Will be very interesting to see how reliable this solution turns out to be. The plan is to setup some semi-automated testing, and hopefully have some results to share in a follow-up post, say in a month or two.