Friday, July 10, 2015

420: Enhance Your Calm

If you've worked with the Twitter API, especially in the early days, you're probably very familiar with the HTTP error code 420: Enhance Your Calm. It's not an official HTTP error code, but being in the 4xx level it does make it a valid client error, and it has some perfect illusions to what you should do to fix the problem.


Stressing out a system to it's absolute max isn't good. In fact, a DDoS attack is exactly that, issuing too many requests for the server to handle that it has no other viable option then to block legitimate traffic. Stressing out a car by redlining it for hours would have the same very negative effect. It's not good for technology, and it's not good for us.

It's well documented that stress, or our reactions to stressful situations, has very negative reactions on us. Stressing out over a situation does no good, and often leads to bad decisions as well as the physical and mental effects it has on you.

But I strive well under stress.

No, you may think you do well when stressed out, but what really happens is you produce more. The quality of code produced while under a strict deadline is always worse. Remember the early issues of It wasn't just because of a lot of incompetent people, it was also because those people were put under such strict deadlines the only reasonable way to accomplish all that was required of them was to work long, hard hours. That leads to stress, which produces shitty work product.

I've had a lot of personal experience with this in my professional life too. When I first came to Newstex, we had to switch off of an antiquated system that was falling apart at the seams. It wasn't designed to handle the massive volume of content we were processing, and it was dropping content already. I was fresh out of college with a lot of debt and needed to prove myself to the company. I put some undo pressure on myself as well because I wanted to make the system perfect. I was getting requirements thrown at me from every direction and would scatter to try to get things done as they came in since there was no way to keep up with it if I didn't. Every day was like fighting a fire, except there was almost no way to gain ground, the backlog kept on burning up and I just couldn't get ahead of it all.

I took shortcuts, but I got the job done. The system scaled very quickly from only taking in about 50k stories a week to over 200k a week without any issues. Since switching onto that new system we've had no significant downtime, we've always been able to recover quickly and build on top of it as we needed to. I produce a lot of code in a very short period of time all because I was put in a very stressful situation and the only way out of that situation was to do so.

5 years later, I realized there is a better way

Back when I first started, I produced a lot of code, and answered every feature request with either implementing it within a few days, or not doing it because it would take too long. The problem with that is you end up with a house made of duct-tape instead of a solid infrastructure. Sure the system works and it's incredibly flexible, but it could have been built a lot better, and made a lot more adaptable and easy to manage. I recently started a brand new and very large project for my company (now re-branded as ACI). I looked at all the old code I had written and though "Wow, how does this even work?". Yes it worked, but it was ugly and not very maintainable.

Stressful situations get you to build code quickly, but approaching a problem in a much more calm manor, when you don't have tight deadlines or lots of features you need to do right now leads to a lot more stable code. It makes you re-think processes and instead of taking the easy way out, do the right thing. Sometimes the easiest way is the right way, but it's almost never the first way you think about it.

When my boss calls me in a panic, I just say "Ok, no problem"

Early on I would get upset when I got a call the day of a deployment telling me I needed to change everything. I've learned the simple solution to not being stressed out about problems is to just not let them stress you out.... don't react in a way that won't get anything done. What good does getting upset about the changes do? When he does that sort of thing now I simply say "Ok, no problem, but then we have to delay this release and push off these features to get it done". It's a trade-off; if you want new things done something else gets pushed off. 

Take time off, and go off-grid

Another thing that I've discovered that really helps when the stressful situations start to build up is to take a break whenever you can. When there aren't fires to be put out, take a week and go completely off-grid. Make sure there are people in-place to manage things while you're gone, but don't take any calls, texts, or even emails while you're away. (I wasn't quite able to not answer any emails, but I didn't answer any from clients at least). At least once a year, your body needs to completely de-compress and forget about everything. When you come back you should be much more focused and able to think more clearly about the issues that you need to solve. Don't look at that never-ending task list and think "it will never be all done", that's not the point of it.

Inbox zero is a pipe dream for most of us. Take what you can tackle, delegate what you can, and just push off what you can't. Some things will just sit at the bottom of the pile forever. That's just the way things go. Don't expect to always get every task completed. Get what's important done.

Take time for lunch, every day.

This can be one of the hardest things to do as a remote employee. Take time off for lunch, if you aren't heading out of your home-office, at least go away from your computer for a while. You should already be standing about once an hour anyway, so this time just go to the kitchen, make something more then just a sandwich, and then sit down at the table and eat it. Don't just scarf it down and go back to work either, enjoy it, take some time away. 

The bottom line is don't get angry.

Just be realistic. When new requests come in be honest, if you don't think you can get them done let them know. If your boss tells you what he needs you to get done, tell him when you can get that done. If he tells you when the deadline is, give him a list of features to prioritize and tell him how far down the list you can get done. And remember it's always best to under-estimate how much you can get done and how quickly you can get it done, if you end up getting more done or get things done quicker it's way better then getting less done or taking longer.

Just don't get angry when new things happen. Explain what you need to get it done, if it's time or another person. That's easier said then done, but the more you work on it the better you'll get.

And remember the next time that angry customer calls, they aren't angry at you personally. They are angry at the situation. It does no good to fuel their anger.

Thursday, December 4, 2014

Is Microsoft the new Apple?

Think Different.

That's the phrase Apple coined back in 1997. It appealed to the upcoming generation who wanted to "fight against the man". In those days, the "man" was Big Blue, the terrible combination of IBM and Microsoft. They were all there was to have for computers. Microsoft beat Apple to the punch, they had invested more in the GUI then Apple had, and gotten a release out first. They were widely adopted, so they quickly grew in support. Businesses and Personal users both used Microsoft Windows, because it's what all the applications were written for, and people all wrote applications for Windows because it's what everyone used.

Apple wanted to change that, they wanted people to use the better operating system, even if there weren't as many applications for it. They started building their own Office-like products, and trying to convince the world that simpler was better.

The new age of Apple

Simpler is better. And when Apple switched to a Unix-based backend to run OS X, they ushered in a new wave of support. They no longer were just the system for the technically un-savy, they gave every geek what we wanted, a nice GUI on top of an existing platform we already knew how to use.

Apple continued to grow and innovate. Eventually they launched the first iPhone; a smartphone designed for the average person. It wasn't the first Smartphone, but it was the first to include an App Store, and a completely new platform designed to help developers put their software in the hands of users on the go. Apple once again was at the forefront of innovation because they were the underdog. They had to innovate, and produce top-quality products to get people to switch away from existing devices.

Apple as the new Giant

After the iPhone launched, Apple continued to innovate for a few years. It's hard to pinpoint exactly when (although I believe it was directly related to the death of Steve Jobs), but at some point after Google launched the Android OS, Apple started copying instead of innovating. Instead of pushing forward with new ideas, and building everything with their users best interests in mind, they decided to bow to pressure and do what others had been doing. They made phones larger, more unmanageable for the average single-hand usage, and even started selling cheaper versions of the phone.

Apple has been riding on their brand for the past few years. They have been stale, and they have been adding very single-use cases to their OS. You can tie an app into Siri, but only in a very specific way, through Home Automation APIs. You can share data between apps, but only through "Health Kit" which is designed to share health-related data. Everything so far has been piecemeal, instead of a better, more general solution.

Microsoft is the new little guy

Everyone forgot about Microsoft. They bought Nokia in an attempt to enter the Smartphone market. They built an entirely new OS around touch and widgets. They looked at how users want to use their devices, and have even produced a new Microsoft Band smart watch that is specifically advertised as a way to stop technology from massively disrupting your everyday life. They've built a new OS shared between a desktop, tablet, and phone (although only the "new" apps work on all three). The problem with their software right now? Nobody writes apps for them.

Because nobody writes apps for them, nobody wants to buy them. Because nobody wants to buy them, nobody writes apps for them.

Wait, doesn't this sound familiar?

Where do we go from here?

I've always loved Apple products, but I'm not an Apple fanboy. Apple currently has the best PCs on the market. They have the best Smartphones on the market as well. Microsoft is catching up quickly, and they've already developed a superior OS, it's just not quite ready for prime-time.

The concepts are there, the ideas are there, but Microsoft needs to execute as well as Apple did back in the 90s. They need to encourage people to Think Different. The next time you're thinking about dropping $650 on an iPhone or Android (and yes, you're really paying $650 despite what your carrier tells you), take a closer look at the $150 Windows phone instead.

As for me? I have no plans to switch my desktop to Windows unless they also embrace Unix. I'm a coder, and I use the terminal in my every day life. We develop iPhone apps because that's where the money (and users) are at. 

However, my next phone, in about a year or two, will most likely NOT be an iPhone.

Wednesday, December 3, 2014

Logentries - A different kind of Logging service

NOTE: I originally reviewed Logentries in a post I did for In it I had confused this with another logging service I had checked out which was an Application-level logger only. Logentries does accept logs from Syslog.

Logentries was actually a recommendation to me after I wrote a blog post about our search for log services. After checking it out for a little while, I realized pretty quickly that the service was not for us. Log entries has a very unique approach to logging, both for Applications and Systems. Instead of deciding what to SEND to Logentries, you have to decide what to keep Searchable in Logentries.

That makes for a very unique approach to logs. With their new Unlimited logging service, you can send as much data as you can produce, from as many servers as you have, and still only pay for what you actually process and index. That means if you want to send all your debug logs, but you don’t need to make them searchable except for that one period of time where you had an issue, you can do that.

  • Lots of features (live-tail, alerting, etc)
  • Send everything, index what you need
  • “Unlimited Logging”
  • Parse your logs how you want them
  • Longer initial setup to parser your logs
  • Need to tell it what you want to index
  • Expensive depending on how much you want to index

Tuesday, July 29, 2014

Algolia understands Service

The final "S" in SaaS is important. Sometimes the difference between two products can be that Service. In Algolia's case, they screwed up. They made a mistake that caused them to lose all of our documents in one of our indexes. Within a few minutes of contacting them, they had figured out what happened, and were working to correct it. Unfortunately they couldn't restore our lost documents, and they felt very bad for this happening.

The Contract doesn't matter

Despite not having an official SLA with them, they knew that we were a customer, and one they wanted to treat right. They didn't point to a document and say "well the service was only down for 1% of the time, so we're only going to give you back 5% of your bill". They are a small company, with a great product, and a fast-paced innovative team. They didn't hide behind an SLA, they specifically went above and beyond to make sure we were happy.

Algolia owned up to the mistake.

We all make mistakes, and they owned up to it, and reimbursed our entire bill for the month. They admitted it was their fault, and they were willing to do anything and everything to apologize and minimize the impact on us.

Algolia understands customer relations 

Algolia Cares, unlike Splunk. They responded quickly, found the problem, and gave me steps to correct it, as well as their assurances that they have implemented test cases to prevent it from happening again. It wasn't catastrophic, our systems didn't go down because of it, but it did inconvenience us, and they were well aware of it.

Happy customers are repeat customers

I'm incredibly happy to see Algolia spend so much time making sure we feel satisfied with the service. They're constantly improving things and taking our feedback, and giving us features that we said we'd like to see. They're keeping us in the loop, and when something bad happens they fix it.

If you need a distributed search system, I strongly suggest Algolia. It's fast, the service is great, and the system is incredibly reliable. We've had only a few issues over the past few months we've been using it, and none of it was true downtime. They looked at our specific use case and have made improvements to their backend systems to make sure our system runs fast, not told us to re-do how we use the system to make it run fast.

They take the initiative to make the system work the way we want it to. They don't force you to use something they've structured for a specific purpose.

SaaS - The "Service" is important

I really hate when you find a brilliant piece of software, something that will help aid in your daily productivity, and then find out you can't use it because of mismanaged business practices. For a while now at Newstex we've swapped between multiple logging solutions, until we had finally landed on SplunkStorm. SplunkStorm was designed to be the SaaS version of Splunk, all cloud-based, allowing us to push all of our logs directly to it and not worry about managing servers, or upgrades to software. It just worked, and it worked pretty well. When they finally announced pricing, we were quick to jump on board. We picked up 500GB worth of logging to keep our logs around for 30 days. It did graphs, post-mortem analysis, and even allowed us to post-process messages after they had already been sent to the system. We could get Analytics out of it without a problem, and they even added support for Alerting (although in beta).

Then the business team got involved

Unfortunately Splunk is a public company, and they decided that SplunkStorm wasn't going to be supported anymore (I still have several outstanding support emails from months ago that will never be responded to). You can't pay for any more they 50GB of storage (which is now just free). It's a "lite" version, and you also can't get Alerts added to new projects. We quickly exceeded the 50GB/month storage quota, but couldn't get an upgrade without going to "SplunkCloud". 

In theory, this was a great idea, Splunk Cloud was just Splunk run as a Service, not a modified version. That meant we could get an API into the system, and integrate with third party services like Tableau. This was great, we thought, so we started the process to see what it would take to get the service.

Splunk does not want to sell to small companies

They advertise themselves as the "Enterprise" logging solution, and they don't want to sell to "small fish". They say they want new clients, but they aren't willing to work with you. The worst part about it was they have absolutely no idea what a remote company is. They passed us off from salesperson to salesperson because they couldn't figure out "who's region we belong to". The first person we talked to wasn't going to help us because he wouldn't get credit for a sale.

Competition among employee's is good for the company, bad for the customer.

I just wanted to buy

We didn't care who got "credit", we wanted to buy. We were ready to give these people our money, $10k/year (which was higher then we really wanted to spend, but it's the cheapest they would give us). We wanted to know where to sign up, they had no idea what that meant. Their system only worked by sending a P/O, and they could only bill annually.

Over a month later

4 weeks went by, still with very little idea of what was going on. We asked to bill quarterly, they didn't respond for another week or so. Finally at the end of the month, we get a response from someone who was ready to talk about sales, but still hadn't gotten final approval to bill us quarterly, just that they were "working on it".

Too little, too late

At this point we'd decided to switch and check out the new Loggly Gen2 systems. We're still in the process of switching over, but the sign up process was painless. We could fill out a few sliders to configure exactly how much we wanted to spend, and enter a credit card and you're done. That's a SaaS system. We did have a small hiccup in the beginning because we had an old account, but a quick tweet got their response within a few minutes, and a resolution within a few hours.

So we're back to Loggly, at least for now. Why? not because Loggly's software is better, but because the SERVICE they have is better. Loggly started as the service for small companies, and they get it. In a small company every resource is important.

Thursday, July 17, 2014

How NOT to do 2 Factor authentication (MailChimp, this means you!)

UPDATE: You can enable QR code/Authy for AlterEgo

Thanks to a co-worker who discovered how to enable a QRCode based authentication for AlterEgo. After logging into AlterEgo via the website, you can go to "Integrations":

Under "Google Authenticator" choose "Connect":

This will generate a QR code you can attach to Authy, or any other standard Software MFA device!

How NOT to do 2 Factor authentication

Two factor authentication is great. It's the latest craze, but it's also a good idea. In general, the password is obsolete. Anyone can guess or brute force a static password, and making people change a password is lame. They forget, which means you need to have way to let them reset.

If it's something they're typing on mobile devices, it's probably going to be pretty weak, and the more you have to type it, the less secure it will be.

A multi-factor (or Two-factor) authentication token solves much of these problems. People will always make insecure passwords, a second form of authentication is key. There are three main types of authentication:

  • Knows Something (Password)
  • Has Something (Authentication Token)
  • Is Something (Firewall)
Breaking into the "Has Something" is critical, but it's also important to make sure it's not an obstacle. There are standards out there for how to do authentication tokens. Almost everyone generates a QR code that you can scan on your mobile application, and/or just uses SMS.

Yes, this does mean that there's a QR code out there that someone could hijack, but hopefully that QR code is not printed, but instead kept securely on the user's device. If you're like me, you use Authy, which does back up your MFA tokens, but also requires you to input more information when you need to restore, only allows on one device at a time, and requires a secondary form of MFA if you do need to restore (such as an SMS).

Other providers, such as RSA, allow for physical MFA tokens. These are by far the most secure, but also expensive, and a hassle if you have a bunch of them. I have one for my 401k, PayPal, and AWS account. Everything else is a Software Auth Token.

Google's MFA does not do backups, and if you upgrade your phone you lose it all. Not as ideal, but still not as bad as....

Mailchimp You're doing it wrong

MailChimp introduced MFA. Pretty great right? You don't want someone getting ahold of your client list, that could be pretty bad.

But they don't use a standard like a QR code, a physical token, or just SMS. Nope, they use a third-party company called AlterEgo

First off, when you search for "Alter Ego" in the app store, this app isn't what comes up. That's pretty bad itself, but not the worst part.

The worst part? They don't do two factor authentication like anyone else. The app is a mobile-browser package, and you can tell. It is NOT optimized for touch screens, let along small devices. It requires a login of username and password... wait isn't this what the MFA was suppose to be solving for us?

Worse yet, while it DOES have time-based codes, those codes are also one-time use. The interface doesn't have a simple way to let you generate a new code until the old one expires, even if you've already used it. In MailChimp, you often have to re-login all over again (another issue) including when you add new people, or are setting up your account for the first time. This means you're typing in your AlterEgo token multiple times within the 1 minute window that the token takes to "expire". That means you have to wait.... you can't just re-generate a new token, even though the one on the screen no longer works.


It does not make me feel more secure. In fact it breaks your normal workflow, and makes your service difficult to use. There is no reason you can't generate a QR code and support every other type of MFA out there, or even just use SMS. You have SMS as a backup, but you can't set it up that way just with SMS.

Please please please, don't continue to require AlterEgo.

Wednesday, July 2, 2014

Google Cloud, still not ready for the real world

With all of the new buzz around the fancy features that Google has been launching lately, I decided to take a stab at using Google Cloud for one of my new projects at Newstex. The project is simple and isolated, so there was nothing to hinder me or tie me into AWS, and nothing to prevent me from testing out the waters.

What I discovered, however, is the reason why many people are still primary focused on AWS instead of Google Cloud; there's a lot of shiny features, but it's missing the important parts.


Let's start with the most important part of any Cloud Infrastructure, Security. It's absolutely paramount that you can control access to resources, and make sure you contain who (and what) has access to different elements of your cloud environment. For example, you don't want to allow a process to have access to start and stop servers if all it needs is access to read files from your storage system. You also don't want to give access to every storage bucket to your new client that wants access to your files to download, just the specific files they should have access to. You also don't want to give the new DevOp you just hired full write-level access to kill all servers and horrendously screw things up before they know what they're doing.

For all of these things, Amazon developed IAM. You can securely control exactly what access any given set of credentials provides, in some cases down to an incredibly low-level of granularity, such as limiting someone to a specific prefix in an S3 Bucket, or a certain sub-set of items within a DynamoDB table. The granularity level is extreme, and it's very easy to construct access control rules that allow you to prevent abuse, as well as just misuse.

Google Cloud lets you split access apart by Project. That's it.

APIs and Documentation

Here's an interesting test for you, try to figure out how to write a row of data into a BigQuery table from a script in Python. Better yet, try finding out how to do it in Node.js. While Google does consider Python a primary programming language, if you're just trying to access an API through a script, it takes a lot of bootstrapping just to get that to work. There's no simple API like "boto" to handle all the automatic magic required to connect. The API keys don't even work with BigQuery, and the API keys aren't even secure (or really secret since you're sending them along with every request). There's no simple signed url scheme, it all relies on OAuth2.

Have you ever worked with OAuth2? It's a complete pain in the ass. Not to mention it's designed for a web-based workflow, not a server-side script. So when you do finally manage to get that script working, you have to go to a browser to authorize your request, then store the access keys for future use. Oh, and those expire. That requires more manual intervention.

It's not a usable API if it requires manual intervention. Yes, you can use it from within AppEngine, or within Google Cloud (with some work), but what if I don't want to run it within the Google Environment? Why am I so tied down to using their systems? Even then, it's not very obvious how to get things to work.

This is a fault of two aspects, the API is not well designed, and the documentation does not describe it well. A simple REST API with JSON would solve this issue, provided there was also decent documentation. There should be documentation and API wrappers to help with at least the most popular languages out there. I shouldn't have to resort to looking through python code to find out how to authenticate myself.

Oh, and if you offer an API Key, make it work with everything, and make it secure.


Google is in a state of transition right now. It's very obvious if you ever talk to a Google engineer that the teams at Google are in a state of unhealthy competition. The BigQuery folks don't like the Cloud SQL Folks, and the Compute Engine folks are at war with the AppEngine folks. This is very obvious if you look at the design of each system. The "Cloud Console" doesn't have everything in it yet. You have to go out to the old consoles a lot to get the full functionality you want (like certain APIs, or to access certain features of BigQuery).

Yes, this will change, but it is a symptom of a larger problem at Google. There is a lot of internal politics going on that bleeds out and hits the consumer. It's unfortunate that the boys at Google don't know how to cooperate and realize that when one team succeeds, the entire company does (just look at how long it took for Android to adopt Chrome as the default browser).

With Material being released, hopefully this will change soon. This is simply a growing pain issue. Google will solve this eventually, but for now we'll have to wait until they solve their internal disputes before as a consumer we can get a good experience.

Quantity of Services

The quantity of services available at Google is pitiful. Google is very single-minded, and they have specific tools designed around an exact workflow. AppEngine is designed around very specific workflows (Web applications and background processes). BigQuery is designed around a very specific process (write-only datastore). What if you want to queue messages? What if you want to store versioned files? What if you need to store petabites of metadata in an SQL-Like environment? How about DNS with location-based routes?

For those solutions, you're on your own. Yes, there are OpenSource solutions you can run, and you can even run them on Google Cloud, but they don't really give you any simple solutions to do so.

Shiny features, not hard-level power

In conclusion, there are a lot of shiny features that Google Cloud offers (such as hot-VM Migrations), but this is not enough. They did not focus enough on the core requirements, just the differentiating factors. Yes, there are some very nice features of Google Cloud, but it is not a complete solution. There is still too much missing to make it a real competitor to AWS.

What does google need to do?

To get me to switch, there are a few things Google needs:
  • Easier access (Signed API Keys)
  • Granular Access Control (by API Key, by service, by access type)
  • Better documentation
  • Better API Wrappers (Node.js, Python, Go)
  • More services, or easy-access to open-source alternatives such as:
    • Redis
    • Memcache
    • Rabbitmq
  • More tutorials, with different use-cases, such as:
    • Video Upload/Encoding
    • Translation
    • Image Manipulation
    • Background Processing of text files
    • Google+ Sentiment analysis
    • etc...
And don't assume everyone is using Java. Do examples in other languages, even in Go to show off how nice of a language it really can be.

My Plea

Please Google, keep on innovating, and make the developer tools better. BigQuery is an awesome tool, but accessing it is not.

I am not an AWS fanboy, it's just the only service that works. I would gladly use Google Cloud if it offered a competitive alternative. It just doesn't right now.