Monday, February 21, 2011

It's a dangerous time to read Gizmodo

Gizmodo Recently decided to publicly attack Apple about it's recent change in policy, designed to protect mobile users as well as earn a little more cash for Apple.

Gizmodo's been pretty famous recently for changing their entire website to use a new frame-like UI with fixed-width formatting that pretty much makes everyone upset. They've also added massive amounts of advertisements all over the top half of their stories, making all of their readers scroll through a page of ads before they can even read the content.

Now they're attacking Apple publicly, something that's been known not to work, and have very negative repercussions.

And why, after all, do they claim you shouldn't buy Apple products right now? They're coming out with them too quickly. That's right, Apple is releasing too many products now... Wait... how many new Android phones came out this month alone? Even Best Buy knows that their the big culprits, advertising that they will buy back your old technology (note the primary devices are Androids on their ad), for roughly 50% if it's before 6 months.

Seriously Gizmodo, take a clue, Apple isn't going to let you back into the fan club.

Wednesday, February 9, 2011

Amazon T1 Micro instance With Apache Primer

Although there's been a lot of buzz about the new Amazon t1.micro instances and the "Free Tier" for amazon, I haven't seen a good tutorial on how to just get started using the Amazon Dashboard console, so I decided to make this quick-and-dirty "get your site up and running in 15 minutes" bit. For those of you who don't know, Amazon launched a micro instance type which is perfect for most "simple" websites that don't generate a ton of traffic. If you don't need MySQL, but do need some server-side processing like PHP, this may be the solution for you.

Signing Up for the Services

First things first, head on over to and click the big "Sign Up Now" button.

You'll want to sign up for at least EC2 and EBS, but most other services aren't used in this tutorial.

Sign into the AWS Management Console

Everything really happens under the AWS management console, so after you've gone through the riffraft of verifying your account, you'll want to click the "sign in to the AWS Management Console" link at the top of the page.

Once here, you'll want to click on the "EC2" tab:

Launching your instance

Click on the "Launch Instance" button.

This wizard will guide you through launching a new instance. The key things to keep in mind here are that you want to make sure you're launching a "t1.micro" instance and using Amazon's 64 bit image.

On the next page, make sure you choose the "t1.micro" instance:

The next page you can ignore, just click continue:

The next page just lets you set some labels for your instance, in general you probably just want to at least set the "Name" tag to something in the event you launch multiple instances, you can use this to identify which is which:

For example, I chose to name mine "ApacheServer"

Setting up your Keypair

Click continue to the next page, where you'll be prompted to create a keypair. If you've done this before, you can use an existing key pair, but if you're reading this tutorial I'll assume you haven't set one up yet.

KeyPairs are how you log into your server, using SSH. It's important to note that you'll need to download this key pair and keep it in a safe place. For security reasons, you can never download your keypair again after this setup procedure.

Enter a name for your keypair and click the "Create and Download your Key Pair" link:

Setting up your Security Group

Click continue and you'll be taken to a page allowing you to set up your security group. It's generally a good idea to set up a new security group for each new "group" of servers you want to create. For example, this is for apache, so lets name it "apache" and fill out a description:

Now we see by default that SSH is authorized, but we also want users to be able to access this server via HTTP, and probably HTTPS as well. We add those in by choosing the appropriate fields from the row at the bottom:
Make sure to click the "Add Rule" button after adding each rule:

Before moving on, make sure you see something like this:

Review and launch

Now we look over our information, and make sure everything's ok before launching the instance:

After launching, you should see a page like this:

Setting up your Elastic IP

Of course, Amazon is very volitile, so we'll want to make sure we set up an elastic IP for our instance before we change over our DNS name.

On the left side of your console, you should see a link to "Elastic IPs":

Once here, you can allocate a new Address:

This requires you to confirm that you wish to allocate a new EIP.

You're currently limited to 5 EIPs per account, and you pay for any EIPs which are not associated to instances. If your EIP is associated to an instance, you don't pay for it. From here, you'll see your new EIP:

Right-click on it and choose "Associate Address":

Choose your instance:

Assign your DNS to your EIP

This all depends on who you bought your domain from. You can simply associate your domain with this IP address and your users will be forwarded along to this server automatically.

Logging in

Of course, now you'll need to log in and set up your software, specifically Apache. From a terminal or shell, use the keypair you downloaded to SSH into your instance by your elastic IP address:

$ ssh -i myKeyPair.pem

You'll be greeted with a friendly banner:

       __|  __|_  )  Amazon Linux AMI
       _|  (     /     Beta

See /usr/share/doc/amzn-ami/image-release-notes for latest release notes. :-)

If instead you see something like this:


Then you need to first change your keypair permissions:

$ chmod 0600 myKeyPair.pem

Installing Apache

The instance you've just logged into is based on Redhat, so you've got the Yum package manager. Install apache using the following command:

$ yum install httpd
Confirm all the prompts with a "y", and continue on to setting up your apache configuration

Configuring Apache with WebDav

The easiest thing to do next is to configure your Apache server to allow you to connect via webdav. Do so by first creating an .htpasswd file for your user:

sudo htdigest -c /var/www/.htpasswd webdav myAwesomeUserName

It'll prompt you for a password, make sure this is secure since you'll be using it to gain full access to your website files.

Next you'll want to change the ownership of the /var/www folder to the apache user:
sudo chown apache:apache /var/www

Finally, set up the DAV server in a /etc/httpd/conf.d/000-default.conf file. You can use whatever basic editor you want here, but I use vim:

vim /etc/httpd/conf.d/000-default.conf

Make sure this file looks like the following:
        DocumentRoot /var/www
        <Directory /var/www>
                Options Indexes MultiViews
                AllowOverride All
                Order allow,deny
                allow from all

        Alias /webdav /var/www
        <Location /webdav>
           DAV On
           AuthType Digest
           AuthName "webdav"
           AuthUserFile  /var/www/.htpasswd
           Require valid-user

Starting Apache

You can start apache by simply running:
$ sudo service httpd start

Setting up your WebFiles

You can now connect to your WebServer via WebDav. Simply use the following URL, replacing with your hostname:

And connect using your username and password you set up with the htaccess command. Note that this will take you directly into your web root, so feel free to deposit your index.html file directly here.

Friday, February 4, 2011

Boto and Amazon Route53

For a while now, I've been working on making the Route53 module for boto much more simple and easy to use. While Route53 doesn't yet provide anything special that you can't find in other DNS providers, it's still quite useful to be able to programmatically change DNS records from within your instances. For example, you could set up an instance to boot and automatically associate itself as another server on a given domain for the purposes of Round Robin DNS, or for setting up Memcache servers where you need to have your servers contact all of the possible memcache servers to determine if there's a cached version of your data available.

Creating your first hosted Zone

Before you begin, you'll need to make sure you sign up for Route53 from Amazon Web Services. It's free to sign up, and you only pay for what you use, so if you have no hosted zones you'll pay nothing for being signed up. It only costs $1/month per hosted zone, plus a minimal charge for each hit against your domain.

To create your domain, use the route53 command line client included with boto.

% ./bin/route53 
Usage: route53 [command]
 add_record - Add a new record to a zone
 cmd - Prints this help message
 create - Create a hosted zone, returning the nameservers
 delete_zone - Delete a hosted zone by ID
 get - Get all the records for a single zone
 help - Prints this help message
 ls - List all hosted zones

We want to use the create command to make your first domain.

% ./bin/route53 create 
Pending, please add the following Name Servers:

After you create the zone, the script will tell you where to point your domain. This must be done from wherever you purchased your domain.

As you can see, Amazon's DNS servers are located around the globe so that no matter where your end-user is, they can have a DNS server close by.

Adding a record to a Zone

Of course, without any records, this zone configuration is relatively useless, so lets set up our records. Before we can add a new record, we need to look up the ID for the zone we just added:

% ./bin/route53 ls         
| Name:
| Ref:  xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx

Next we use that to add our A record, using the Command line function:

% ./bin/route53 add_record ZXXXXXXXXXXXXXX A    
{u'ChangeResourceRecordSetsResponse': {u'ChangeInfo': {u'Status': u'PENDING', u'SubmittedAt': u'2011-02-05T01:34:20.912Z', u'Id': u'/change/XXXXXXXXXXXXXXXXX'}}

This change request will take a while to actually occur, so just sit back and wait for the change to happen, or continue on adding records. You can add any type of record, A, AAAA, CNAME, MX, TXT... etc..

Adding records through boto

Of course, the real advantage is probably not through the command line functionality, but through the ability to change your records programmatically. For example, you could create the following script to automatically add the current instance to your zone if it's a standard boto-based image:

import boto
from boto.route53.record import ResourceRecordSets
conn = boto.connect_route53()
changes = ResourceRecordSets(conn, "ZXXXXXXXXXXXXXX")
change = changes.add_change("CREATE", boto.config.get("dns", "name"),"CNAME")
change.add_value(boto.config.get("Instance", "public-hostname"))

What's next?

Eventually, Amazon plans to add in Geographic support which will direct your end users to different IPs based on where they're located. This means you can have EC2 servers (or load balancers) set up in different Regions, and depending on where your user is sitting, they'll get the server closest to them!

Thursday, February 3, 2011

Boto, Loggly, and rsyslog

Recently, I discovered an interesting new "Cloud Logging" service provided by a small start-up company called Loggly. This service promised to offer a nice web interface to a logging server that would essentially comprise of a set of syslog servers that could read in all of my standard logs that I would normally get on the vast amounts of cloud-based servers I use, and keep them in one central searchable location.


Loggly has recently had a horrible track record of downtime, although they're by far the cheapest and most feature-filled service that we've found, the record of about 50% uptime is really a killer. After a 3 day outage, we've switched to papertrailapp, and are looking around for any other alternatives.

Do you have an alternative solution to Loggly? Please let me know in the comments!

Installing rsyslog

Obviously this sparked my attention, as I'd love to not have to log into 5 different servers just to find out what was going on in my boto.log file. I began researching and discovered that I'd have to switch to using rsyslog, which proved to be a slightly more difficult task then you'd expect when running on the Amazon Linux AMI based on RedHat, but with a bit of googling I found an easy solution:

yum shell
install rsyslog
remove sysklogd

All this eventually results in replacing the built-in syslog server with rsyslog, which not only handles multiple threads of execution at once, it also handles sending logs to a remote syslog server.

Setting up boto logging

The next thing I tried was to use rsyslogs built-in method of simply polling a file and piping that through its logging mechanisms. There were several problems with this, but the most important and deadly issue was that no matter what I did, the syslog server seemed to crash overnight. As it turned out, this was a problem with rsyslog not handling the logrotation of that file, which caused it to lose place in the file and freak out.

As a solution, I decided to look into simply piping the logs directly to syslog, bypassing the intermediary step of using a file at all. This proved to be relatively easy once I found the right configuration directives. If you recall, boto has the ability to assign logging configuration automatically from the boto.cfg, so simply replace your existing logging configurations with this:






formatter = syslog
class = handlers.SysLogHandler
args = ('/dev/log',handlers.SysLogHandler.LOG_USER)
level = INFO

format = [%(name)s] %(levelname)s %(message)s

The most important thing to note is that you should NOT use any date formatting in the formatter, I've noticed this causing bizarre issues that make the logging not work.

Piping all logs to loggly

The last step is provided to you with a configuration line given by loggly itself. When you set up an input, you get a configuration line to add to your rsyslog.conf file, something like:


Using this, you'll send all your logs directly to the loggly servers, including your newly added boto logs!

Test this using:
import boto
boto.log.infO("Test log from BOTO!")

Allow about a minute for the log to appear in loggly, although it may appear much quicker then that.