Boto, Loggly, and rsyslog

Recently, I discovered an interesting new "Cloud Logging" service provided by a small start-up company called Loggly. This service promised to offer a nice web interface to a logging server that would essentially comprise of a set of syslog servers that could read in all of my standard logs that I would normally get on the vast amounts of cloud-based servers I use, and keep them in one central searchable location.

UPDATE:


Loggly has recently had a horrible track record of downtime, although they're by far the cheapest and most feature-filled service that we've found, the record of about 50% uptime is really a killer. After a 3 day outage, we've switched to papertrailapp, and are looking around for any other alternatives.

Do you have an alternative solution to Loggly? Please let me know in the comments!

Installing rsyslog


Obviously this sparked my attention, as I'd love to not have to log into 5 different servers just to find out what was going on in my boto.log file. I began researching and discovered that I'd have to switch to using rsyslog, which proved to be a slightly more difficult task then you'd expect when running on the Amazon Linux AMI based on RedHat, but with a bit of googling I found an easy solution:

yum shell
install rsyslog
remove sysklogd
run

All this eventually results in replacing the built-in syslog server with rsyslog, which not only handles multiple threads of execution at once, it also handles sending logs to a remote syslog server.

Setting up boto logging

The next thing I tried was to use rsyslogs built-in method of simply polling a file and piping that through its logging mechanisms. There were several problems with this, but the most important and deadly issue was that no matter what I did, the syslog server seemed to crash overnight. As it turned out, this was a problem with rsyslog not handling the logrotation of that file, which caused it to lose place in the file and freak out.



As a solution, I decided to look into simply piping the logs directly to syslog, bypassing the intermediary step of using a file at all. This proved to be relatively easy once I found the right configuration directives. If you recall, boto has the ability to assign logging configuration automatically from the boto.cfg, so simply replace your existing logging configurations with this:

[loggers]
keys=root,boto

[handlers]
keys=syslog

[formatters]
keys=syslog

[logger_boto]
level=INFO
handlers=syslog

[logger_root]
level=INFO
handlers=syslog

[handler_syslog]
formatter = syslog
class = handlers.SysLogHandler
args = ('/dev/log',handlers.SysLogHandler.LOG_USER)
level = INFO

[formatter_syslog]
format = [%(name)s] %(levelname)s %(message)s

The most important thing to note is that you should NOT use any date formatting in the formatter, I've noticed this causing bizarre issues that make the logging not work.


Piping all logs to loggly

The last step is provided to you with a configuration line given by loggly itself. When you set up an input, you get a configuration line to add to your rsyslog.conf file, something like:



*.*    @@logs.loggly.com:123456

Using this, you'll send all your logs directly to the loggly servers, including your newly added boto logs!


Test this using:
import boto
boto.log.infO("Test log from BOTO!")

Allow about a minute for the log to appear in loggly, although it may appear much quicker then that.

Comments

Anonymous said…
It's been a year, so you may have already solved this issue, but what I am about to embark on is the following:

Similar to yourself, rsyslogd running on most of our production Linux servers (spread around Linode, Rackspace Cloud, EC2, and in our own offices), all pushing logs up to an aggregating rsyslogd server somewhere in the cloud, which in turn writes out to a MySQL back-end.

This comprises the consolidation layer, and on top of that you place some indexing and analysis tool, such as LogAnalyser or SPlunk, depending on the log volume (we're not replete with cash!)

This should be very resilient, as at each stage rsyslog can spool gigabytes of data to the filesystem, waiting for connectivity to be restored, in the event of any outage.

The final rsyslog cloud server and MySQL could be managed by us, or it could use Amazon RDS out of the box, which seems fairly neat and offloads some management work. Alternatively, with some work I think the final persistence mechanism could be Amazon DynamoDB, which could be more performant and scalabale, and open the door to using some Hadoop/Hive goodness at the analysis end of the process.

I didn't know of loggly, and you peaked my interest, but the mention of 50% downtime puts me right off!
I just checked their prices and they seem quite steep if you want to retain log info for 3 months.

I hope this is useful!

cheers
Chris Jomaron
Columba Systems