Boto, Loggly, and rsyslog

Recently, I discovered an interesting new "Cloud Logging" service provided by a small start-up company called Loggly. This service promised to offer a nice web interface to a logging server that would essentially comprise of a set of syslog servers that could read in all of my standard logs that I would normally get on the vast amounts of cloud-based servers I use, and keep them in one central searchable location.

UPDATE:


Loggly has recently had a horrible track record of downtime, although they're by far the cheapest and most feature-filled service that we've found, the record of about 50% uptime is really a killer. After a 3 day outage, we've switched to papertrailapp, and are looking around for any other alternatives.

Do you have an alternative solution to Loggly? Please let me know in the comments!

Installing rsyslog


Obviously this sparked my attention, as I'd love to not have to log into 5 different servers just to find out what was going on in my boto.log file. I began researching and discovered that I'd have to switch to using rsyslog, which proved to be a slightly more difficult task then you'd expect when running on the Amazon Linux AMI based on RedHat, but with a bit of googling I found an easy solution:

yum shell
install rsyslog
remove sysklogd
run

All this eventually results in replacing the built-in syslog server with rsyslog, which not only handles multiple threads of execution at once, it also handles sending logs to a remote syslog server.

Setting up boto logging

The next thing I tried was to use rsyslogs built-in method of simply polling a file and piping that through its logging mechanisms. There were several problems with this, but the most important and deadly issue was that no matter what I did, the syslog server seemed to crash overnight. As it turned out, this was a problem with rsyslog not handling the logrotation of that file, which caused it to lose place in the file and freak out.



As a solution, I decided to look into simply piping the logs directly to syslog, bypassing the intermediary step of using a file at all. This proved to be relatively easy once I found the right configuration directives. If you recall, boto has the ability to assign logging configuration automatically from the boto.cfg, so simply replace your existing logging configurations with this:

[loggers]
keys=root,boto

[handlers]
keys=syslog

[formatters]
keys=syslog

[logger_boto]
level=INFO
handlers=syslog

[logger_root]
level=INFO
handlers=syslog

[handler_syslog]
formatter = syslog
class = handlers.SysLogHandler
args = ('/dev/log',handlers.SysLogHandler.LOG_USER)
level = INFO

[formatter_syslog]
format = [%(name)s] %(levelname)s %(message)s

The most important thing to note is that you should NOT use any date formatting in the formatter, I've noticed this causing bizarre issues that make the logging not work.


Piping all logs to loggly

The last step is provided to you with a configuration line given by loggly itself. When you set up an input, you get a configuration line to add to your rsyslog.conf file, something like:



*.*    @@logs.loggly.com:123456

Using this, you'll send all your logs directly to the loggly servers, including your newly added boto logs!


Test this using:
import boto
boto.log.infO("Test log from BOTO!")

Allow about a minute for the log to appear in loggly, although it may appear much quicker then that.
1