In December, I posted an article on My Great Search for Syslog services. Since then, We've started using Papertrail to log and manage our numerous cloud-based servers. Papertrail so far seems great, it gives us a live tail of our logs, as well as search ability and even the ability to archive all of our log content to S3. (I love when services offer exporting to S3).
One major drawback of Papertrail, however, is that it's really only useful to tech-savy users. You can only spot issues if you know exactly what you're looking at. It doesn't really have a way to map what your logs say into any meaningful segments. It does do a decent job most of the time trying to pick some common things out, but my logs are very structured, it would be nice if I could explain my structure to Papertrail.
I recently received a Beta invite to the SplunkStorm project (I can only assume it was in relation to my blog post). I never looked at Splunk before, since it really is an "in-house" solution, not a service. This new model, however, is just what I'm looking for: A true Syslog service offering a nice pay-as-you-go structure (or so they're planning). It's still in beta so no prices are out yet, but from what I can tell the pricing structure will be based on the data that's searchable. In Papertrail, you pay for how much log data you send into the system, whereas SplunkStorm meters by how much data you retain in the system. Once there's too much info in your Project, old data is removed to make way for the new stuff. This means you always can send data into your project and continue to search it. They don't simply start dropping logs after a certain period of time.
Ok, so that's the pricing structure (or so it appears), but what about functionality?
Well, they offer a very complex search system, which includes being able to graph (something Papertrail folks are still working on), and the ability to do other very complicated behaviors.
But it gets better...
They have a unique "Dashboard" system, which lets you set up custom graphs based on specific events, splitting on any field in your logs.
Yes... Field, not hostname, not application name, field.
So what is a field?
They have several "templates" that pull out common information (for example, one template parses an Apache log), but if yours doesn't quite fit? no problem, just use their Template system and you can give it any regular expression and mapping out to custom fields.
First you set up a thing called a Field Transformation. Under Manage->Fields->Field Transformations you can clone an existing one, or start from scratch. Setting up a Field Transformation is relatively simple, you give it a regex and a mapping. My boto config specifies this logging format for my logs that go to syslog:
%(name)s[%(process)d]: %(levelname)s %(message)s
Unfortunately, when using multiple processes/threads with the logging module, you'll often get crap in your log messages:
<11>Feb 2 18:06:09 hostname <14>botoweb.url_mapper[3000]: INFO [username] GET: /
these <14> and often <11> appear in the logs just before the date and log name in most (but not all) of my log lines.
Ok, so lets parse this out. We'll start off with a Transformation named syslog-boto.
Enter my massive regex:
^(<\d+>)?([A-Z][a-z]+\s+\d+\s\d+:\d+:\d+)\s([a-zA-Z0-9\.\_]+)\s+(<\d+>)?([a-zA-Z\.\_\-]+)\[(\d+)\]:\s+([A-Z]+)\s+(\[([a-zA-Z0-9_]+)\]\s+)?(.*)$
And my "format":
date::$2 host::$3 process::5 pid::$6 level::$7 username::$9 message::$10
And tada! We now have a proper way to map these logs to their different field names.
But wait, it's still not quite completed! You have to then set up a Field Extraction that tells your project when to use this template. I set mine up like this:
Name: syslog-boto
Apply to: sourcetype
Named: syslog
Type: Uses transform
Extraction/transform: syslog-boto
Save, and now check your logs, you'll see they're parsed out now. You can even search by field name such as this:
level=info AND username=cmoyer
There's so much more you can do with SplunkStorm. You can create dashboards now based on these fields. For example to chart out activity on your botoweb server by username:
level=INFO | timechart count(_raw) by username
What else are you using SplunkStorm to do?
One major drawback of Papertrail, however, is that it's really only useful to tech-savy users. You can only spot issues if you know exactly what you're looking at. It doesn't really have a way to map what your logs say into any meaningful segments. It does do a decent job most of the time trying to pick some common things out, but my logs are very structured, it would be nice if I could explain my structure to Papertrail.
I recently received a Beta invite to the SplunkStorm project (I can only assume it was in relation to my blog post). I never looked at Splunk before, since it really is an "in-house" solution, not a service. This new model, however, is just what I'm looking for: A true Syslog service offering a nice pay-as-you-go structure (or so they're planning). It's still in beta so no prices are out yet, but from what I can tell the pricing structure will be based on the data that's searchable. In Papertrail, you pay for how much log data you send into the system, whereas SplunkStorm meters by how much data you retain in the system. Once there's too much info in your Project, old data is removed to make way for the new stuff. This means you always can send data into your project and continue to search it. They don't simply start dropping logs after a certain period of time.
Ok, so that's the pricing structure (or so it appears), but what about functionality?
Well, they offer a very complex search system, which includes being able to graph (something Papertrail folks are still working on), and the ability to do other very complicated behaviors.
But it gets better...
They have a unique "Dashboard" system, which lets you set up custom graphs based on specific events, splitting on any field in your logs.
Yes... Field, not hostname, not application name, field.
So what is a field?
They have several "templates" that pull out common information (for example, one template parses an Apache log), but if yours doesn't quite fit? no problem, just use their Template system and you can give it any regular expression and mapping out to custom fields.
First you set up a thing called a Field Transformation. Under Manage->Fields->Field Transformations you can clone an existing one, or start from scratch. Setting up a Field Transformation is relatively simple, you give it a regex and a mapping. My boto config specifies this logging format for my logs that go to syslog:
%(name)s[%(process)d]: %(levelname)s %(message)s
Unfortunately, when using multiple processes/threads with the logging module, you'll often get crap in your log messages:
<11>Feb 2 18:06:09 hostname <14>botoweb.url_mapper[3000]: INFO [username] GET: /
these <14> and often <11> appear in the logs just before the date and log name in most (but not all) of my log lines.
Ok, so lets parse this out. We'll start off with a Transformation named syslog-boto.
Enter my massive regex:
^(<\d+>)?([A-Z][a-z]+\s+\d+\s\d+:\d+:\d+)\s([a-zA-Z0-9\.\_]+)\s+(<\d+>)?([a-zA-Z\.\_\-]+)\[(\d+)\]:\s+([A-Z]+)\s+(\[([a-zA-Z0-9_]+)\]\s+)?(.*)$
And my "format":
date::$2 host::$3 process::5 pid::$6 level::$7 username::$9 message::$10
And tada! We now have a proper way to map these logs to their different field names.
But wait, it's still not quite completed! You have to then set up a Field Extraction that tells your project when to use this template. I set mine up like this:
Name: syslog-boto
Apply to: sourcetype
Named: syslog
Type: Uses transform
Extraction/transform: syslog-boto
Save, and now check your logs, you'll see they're parsed out now. You can even search by field name such as this:
level=info AND username=cmoyer
There's so much more you can do with SplunkStorm. You can create dashboards now based on these fields. For example to chart out activity on your botoweb server by username:
level=INFO | timechart count(_raw) by username
What else are you using SplunkStorm to do?
Comments
ryan vong