SmoothSpan Blog

For Executives, Entrepreneurs, and other Digerati who need to know about SaaS and Web 2.0.

Archive for the ‘amazon’ Category

Microsoft: World’s Worst Customer Service? (Walmart, Amazon, GE, BestBuy, MacMall, and Paypal Not Far Behind)

Posted by Bob Warfield on July 28, 2014

microsoft-surface-pro-3I recently tried and failed for the fifth time to buy a Microsoft Surface Pro 3.  It’s been a real comedy of errors, but the latest attempt has been by far the most spectacular failure.

Let me start out by saying I really like the Microsoft Surface Pro 3.  I am a perfect candidate for it as I would like to replace the combination of my Macbook Air and iPad with just one device for travel and for demos of my software away from the office.  The business I’m in is software for the CNC Manufacturing world, and while my own software runs on both Mac and PC, most from that world is PC-only.  Hence a device about the size of an iPad that can run desktop Windows software would be a real boon.  The Surface reviews I’ve read have been largely positive, and I played with one at a Microsoft store for long enough to feel like I would be very productive on it.  The keyboard was great and I had little trouble dealing with the Win 8 differences everyone is complaining so much about.  So I resolved to get one.

In fairness, all of my problems have stemmed from one little wrinkle in how I wanted to buy the device.  I’m looking at about $1500 all in, and I wanted an interest free for 12 months deal–the same kind of deal I used to purchase my Macbook Air.  My business is steadily growing and I like the idea of charging most of the cost to the larger version of the business that will exist down the road.  These offers all involve signing up for a credit card, with my Apple Macbook Air it was really no big deal.  I recently had paid off the Macbook Air and so time to get another device.

Here’s what happened.

Fail #1:  Best Buy

Despite haunting the Microsoft Store since the Surface launched in hopes of their offering a deal, no joy.  So I started Googling and wound up at Best Buy.  Looked great, so I attempted to make the purchase.  The online credit card app simply froze up the browser and would neither confirm nor deny I would be able to do the transaction.  Geez, how can a company the size of Best Buy have IT producing forms like this that flat don’t work?  Seems like they’re wasting a lot of opportunity if it happens to very many.

Fail #2:  Walmart

A little more Googling and I discover that Walmart has the same deal.  Great.  Except, oh oh, same problem–the credit card app just fails.  Takes all the info, hit the button to go for it, and nothing happens.  I’m now starting to wonder if the problem isn’t some common third party?  It doesn’t really matter, both these two retail behemoths have lost a $1500 transaction for a stupid reason–their web site didn’t work.

Fail #3:  Amazon

At this point I am thinking it can’t be that hard, SOMEBODY must do this.  So I tried Amazon.  Aha!  They’re offering the no interest deal I want!

I filled out all the information to apply, the application worked (I guess Amazon knows a lot more about software than Best Buy or Walmart), but it turned out to be bait and switch.  Buried in the fine print is a notice that GE Capital would only finance $500 of my $1500 purchase.  Now I have a GE Credit card that will get shredded and never used.  That has to be sub-optimal for both GE and Amazon–they went to all the trouble and cost but are getting no revenue from me.  Not to mention a $500 limit is insulting.  Amazon knows I spend a fortune with them on all sorts of things including Amazon Web Services and have never missed a payment.  Come on guys, do your computers talk at all?  Why offer this stupid $500 credit card on a $1500 purchase?

Fail #4:  PayPal + BillMeLater + MacMall

I went back to the PayPal site to process some orders for my business, and noticed BillMeLater being advertised.  Wow!  I had seen the ads come up every time I had paid for something with PayPal, but I generally just pay cash and had more or less ignored them.  They have a product search that will plug you into a BillMeLater transaction with some merchant that has what you want.  I promptly searched for “Microsoft Surface Pro 3″ and got vectored onto MacMall.  Hmmm, that’s kind of odd to buy a PC from a company that sounds like a Mac company, but why not?  I was getting pretty tired of the chase by now.  I started down the path and promptly noticed I was only going to get 6 months interest free, but again, I was beaten down and ready to do a transaction, so I went ahead.  Filled out all the forms, yada, yada, and BOOM!  I was back to Fail #1 and Fail#2:  PayPal reported that they couldn’t complete the transaction for unspecified reasons (like those other credit card apps just freezing up) and I should try again later.  WTF?!??

Fail #5:  Microsoft + PayPal

Is this becoming Epic Fail, or what?  It’s almost comical by this point.  But, the best is the final episode (so far) and involves Microsoft and Paypal.  I was still focused on the idea of using BillMeLater and it was a new day.  So I had the idea of just seeing who would sell me a Surface Pro 3 and let me pay with PayPal.  I tried Microsoft first, and sure enough.  Excellent!

So I hopped on, performed the transaction, got to the part where you pay PayPal, and for the first time ever (I have made hundreds of PayPal purchases) I saw almost nothing of PayPal and never got the opportunity to use BillMeLater.  Bloody Hell!

I immediately went to PayPal and cancelled the transaction.  There’s a button right there and they accepted and confirmed the cancellation.  Then I went back to the Microsoft Store.  Not so easy to cancel there, I had to call  the dreaded 800 number and wait.  But eventually I got a Service Agent and after answering many strange questions, she assured me that the transaction was cancelled, and that she couldn’t really help me in any way to purchase a Surface with 12 month no interest financing or even to use BillMeLater to make the purchase.  Gee thanks, Microsoft.

So I’m thinking this is pretty silly.  Microsoft must want to be moving these stupid devices and should be making it easier, right?  Maybe I would just go lob a suggestion in to them and maybe someone would get back in touch with me with the right stuff.  I searched in vain both the Microsoft site and the Microsoft Store site for some place I could make the suggestion.  Apparently they are not at all interest in hearing from customers.  I guess I should’ve expected that after getting this far.

Fail #6:  Microsoft + PayPal, Again

This morning I logged into my computer to find 3 email message from Microsoft–a return authorization, a notice that the cancellation had failed, and another notice telling me I should just refuse deliver on the shipment.  Oh boy.  You would think Microsoft could manage to process a cancellation that happened within minutes of an order to avoid needlessly shipping physical goods to a customer who doesn’t want them.  No joy.  So then I bopped over to PayPal to confirm that my cancellation of the prior day was still in place.  The report had been updated to say they were going ahead and paying Microsoft.  WTF?!??  Really?  After both organizations had confirmed the cancellation the prior day?  Are you kidding me?

Now I’m angry.  Both these behemoths had clear instructions from me and had accepted and confirmed.  So, I called PayPal Customer Service.  A nice lady eventually picked up (yeah, lots of voice menus for THEIR convenience) and she confirmed from her screen that I had indeed cancelled payment.  Why then, does my report show this as a transaction that will be paid and why is the cancellation no longer showing?  Well, it looks like the transaction went through before the cancellation could take effect was the response.  OK, why does my balance still not reflect a deduction for the payment then if it’s too late to cancel 24 hours after the cancellation went in and was accepted?  “I’m sorry sir, but it is too late to cancel.  You’ll have to wait 48 hours to see if the seller has refunded your money and if they haven’t, you could file a dispute at that point.”



I was really pretty excited about getting a Surface Pro 3 when I started this trek.  I’m shocked at just how many organizations screwed up their Customer Experience along the way and at just how low the bar is set for that Customer Experience to be acceptable to them.  It can’t possibly be a good thing for sales of the Surface for there to be this much friction in the process.  I am hopeful that some one of the organizations involved will read this and contact me with a solution I’d like, but at the same time, I don’t think I’ll hold my breath.

Macbook Air and iPad?  You’ve got a solid year ahead of you still.  Maybe I’ll just wait until the Surface Pro 4.

Posted in amazon, apple, business, customer service, gadgets, Marketing, microsoft surface, mobile, strategy | Leave a Comment »

Good Customer Experience Trumps Good Customer Service. Bad CUX Trumps All. A Tale of Chukka Boots and Photoshop.

Posted by Bob Warfield on January 22, 2014

ChukkaBootsGood Customer Experience trumps Good Customer Service, even if you are Zappo’s.  My wife quit buying shoes from Zappo’s after they sent her the wrong pair of shoes for the third time and she had to return them.  They didn’t do it all on the same transaction, it happened over a fairly long period of time.  And yes, the Zappo’s Customer Service people were wonderful as always.  But it didn’t matter–the underlying Customer Experience was giving her the wrong shoes and she only allowed that to happen so many times before she gave up on them.

I had a similar experience with Zappo’s, but I didn’t even get as far as Customer Service.  I have bought shoes from them once–a nice pair of Clark’s Chukka Boots.   Great!

Some time later, I went looking for some tennis shoes.  I have a penchant for bright red shoes of the most exotic design possible that I wear when I go to hear live music.  I went straight to Zappo’s, found a pair of shoes I wanted, and tried to purchase.  I expected to be able to use my Amazon account, given they’re owned by Amazon and all, and it looked like I could do that, but I actually couldn’t quite make it work.  I don’t have an account on Zappo’s, because in a time of data breaches like Target’s, I open as few accounts as I can.  So I moved on.  It came time for me to buy another pair of shoes and I went  back to Zappo’s again, thinking that companies as savvy as Amazon and Zappo’s would surely have fixed the problem.  I found the shoes I wanted and tried once more to buy them.  No joy.  I could find no way to buy on my Amazon account and did not want to spend the time opening a Zappo’s account.

Not only did Zappo’s lose the sale of 2 pairs of shoes, but I just won’t go back there again.  It isn’t clear to me Amazon cares much, because in the end, I did buy those 2 pair from Amazon.  But if there was a good alternative I was familiar with, I would’ve skipped Amazon too, just for annoying me.

Now, how hard would it be for Zappo’s not to send my wife the wrong pair of shoes 3 times?  She doesn’t buy shoes all that often, so it was surprising it happened to her so many times.  And how hard would it be for Amazon to make it easy for me to buy shoes from Zappo’s with my existing Amazon account?  Come on, this can’t be rocket science for a company like Amazon.  If Google can figure out to put a birthday logo on their search page on my birthday because it picked up my birthdate somewhere in their far flung empire, Amazon can let me buy Zappo’s shoes with an Amazon account, right?

Fast forward to this morning.  I was doing something and fired up Adobe Photoshop CS3 (yes, I have had it for a long time!).  It immediately announced I had 2 days left to activate or it would die.  Great, I did remember it asking a few days ago.  I had tried and it kept telling me it had an Internet connection problem.  I knew it wasn’t at my end, nothing else was complaining, so I figured I try again–they surely had fixed their problem by now.

No joy.

I was forced to use their phone activation.  With some trepidation I dialed the toll-free number and waited.  I really hate phone support.  It just isn’t ever a happy thing.  Ever.

Eventually, it had me key in a 24 digit serial number followed by a 32 digit activation code using my phone’s keypad.  Wow, that was a joy–not!  But, Photoshop at least did pop up a box that had the phone number to call plus these two lengthy codes to make it easier.  Unfortunately, the phone robot announced my activation code did not have enough digits.

WTF?!??  This was exactly the same code that Photoshop was telling me was the one to use.  How could it be wrong?

I tried twice, to no avail, at which point it told me to hold for a support representative.  Good, I was ready to let some human being know what I thought about all this after having used the software for several years.  Unfortunately, after a 5 minute wait, the Adobe side announced that they were no longer handling activation problems by telephone and gave me a URL I would have to visit with my browser to fix it.  Of course my blood pressure went up to the next DefCon level.

I went to the page suggested and couldn’t find even a hint of clue about what to do.  It was kind of a haphazard FAQ that only listed a few things, none of which could possibly be at issue.  When I got to the bottom, there was a Chat button with a message that cheerfully informed me I could get on right away with an agent if I would simply click.  So I did.

Of course as soon as the chat window opened, it informed me there were other customers ahead of me in line.  WTF?

Okay, deep cleansing breaths.  After no less than 10 messages informing me I was still waiting (no duh, I know I am waiting), Kumar finally popped up.

Kumar is mostly robot.  He is no doubt based on the old ELIZA simulated psychiatrist program which would always turn your question back around without really ever answering much.  It’s a primitive AI technique that’s been around forever.  Try it if you like, it’s kind of creepy in the same way that Kumar was.  I had to provide a description of my problem up front, and Kumar would ask me questions that were phrased along the lines of what I’d already told it, but that didn’t really add much color to the situation:

“Hi Bob.  You’re here because you can’t activate your Photoshop?”

“Yeah Kumar, that’s what I said in the original description.”

This is where Kumar gets clever.  Every time I respond, I get back a message saying, “Okay Bob, I’ll be back in 2-3 minutes after I check into that and take the necessary actions.”  Literally every single response I made, it would do that.  This is because Kumar, or whatever the real human being is named, is sitting in a giant call center somewhere dealing with probably 100 customers simultaneously.  He doesn’t want to get back to any one of us too quickly lest we monopolize too much of his time and annoy the other customers.  So, he uses all this clever software mostly to stall us customers so he can handle more of us.  Sweet!

He asks me to type in my 24-digit serial number (DOH!), but fortunately, I can just copy and paste it (Hah, outsmarted you bozos!).  Then he goes away for extra long–longer than the 2-3 minutes promised.  When he gets back, he wants to know my email for my Adobe customer account.  Oh boy.  Each piece of information will be asked for at 5 to 10 minute intervals–this is going to be painful and I have an appointment in 10 minutes.  I call the appointment to say I am coming, but I will be late.  It’s taken me 45 minutes with Kumar to get this far.

And then, a bit of magic happens.  Kumar comes back and says it’s all fixed, please try again.  I do, and low and behold, the Internet activation works.  A modicum of happiness ensues and I recall the nuclear bombers my DefCon blood pressure rise had summoned.  Then I started thinking about what had happened. Basically, the only reason online activation, had failed, the only reason I had worried whether I would fail to activate and thereby lose a valuable tool, the only reason I had to spend 45 minutes trying to tell Kumar the two pieces of information needed to fix the problem, the only reason I was getting really ticked off at Adobe, was because they wanted to associate my serial number (Kumar didn’t even ask me for the activation code) with my email.

Remember when I said I didn’t create an account with Zappo’s?  Well I also didn’t bother registering Photoshop.  It used to pop up a box about every 2 weeks asking me to fill out an elaborate form, and I would just tell it to go away.  Eventually it offered me the chance to tell it to never ask again, and I did so, thinking what a relief.  Nowhere did they tell me that eventually some power that be would decide they were going to force me to reactivate software that had already been activated and then put me through a painful experience of apparently having that activation fail, just because they wanted me to register.  A registration they no doubt needed so they can send me better marketing spam.

Can we see by now how to apply the maxim that Good Customer Experience trumps Good Customer Service?  Adobe didn’t really give good customer service, BTW, it was terrible.  I don’t blame Kumar for it.  I blame a Draconian wall and a moat filled with alligators designed to keep costs down on a cost center (Customer Service) that was built by a left and a right hand not knowing each other in a large bureaucratic organization and a marketing organization that only cares about filling its lead hungry maw.  It’s about par for the course with large organizations but it also happens to small organizations that pride themselves on treating customers well.  Tragically, it is so unnecessary and counter-productive too.

Let’s take Adobe’s case.  One could argue they never should’ve resorted to all this to connect my email to a serial number.  Let the man not register.  Or, they could’ve just told me I had to register to activate.  Hell, they could’ve just asked for my email as part of the re-activation and I’d have been happy.  Or they could’ve asked me to login to my Adobe account, also acceptable.  There are endless up front Customer Experience things they could have done to eliminate the need for me to deal with Customer Service at all.  Ironically, it would’ve been cheaper to do that.  45 minutes of Kumar and all those automated voice response systems had to cost something.

I run a one-man SaaS company (actually there are a couple part timers, but I’m making a point).  I do all the Customer Service myself.  Whenever and wherever I can, I try to change the User Experience to eliminate classes of Customer Service I see over and over again.  I have to just to survive.  Best of all, it makes the Customers happier and less frustrated.  The next time you’re gearing up a new release of your software, e-commerce front end, or whatever, ask what you can do to reduce the need for Customer Service.  Find out what the common sources of it are.  Get rid of a few of them every time you ship another release.  It’ll be a Good Thing for all concerned, I promise.

Posted in amazon, customer service, Marketing, service, strategy, user interface | Leave a Comment »

The Two Most Desirable Features of a Platform as a Service

Posted by Bob Warfield on July 5, 2011

Big Data?


Uber DB scaling?

Mad Hadoopishness?

Faster app development?

Universal Social Connectivity?

Not so much.  Some platform or other claims all of those things, but the two most desirable features of a PaaS appear to be revenue generation and commodity pricing, not necessarily in that order.

Let me first say that I’ve come to view AppStores as PaaS offerings of a sort, and their ability to generate revenue for developers drives those developers into their arms.  This also applies to more traditional PaaS offerings such as Salesforce’s  As nearly as I can tell talking to folks and scanning the blogosphere for activity, people write for either because they need to integrate with the Salesforce CRM app or because they want the revenue tailwind you can get via

The commodity pricing piece applies to what many prefer to call IaaS, or Infrastructure as a Service, with  Amazon Web Services being the most widely used example so far.  Let’s go ahead and keep them in our PaaS list for this discussion and ignore the IaaS moniker.  If we start cutting out things like Amazon and for various reasons, there is so little left it’s hard to talk about it.  I would submit we may have divided up the market into too many A-A-S’s a little prematurely without waiting to see what would stick.

This is the gist of my problem, BTW.  There’s been plenty of time for PaaS to get big, but it is kind of lumbering along.  Yes, we get things like Heroku.  Doesn’t that qualify largely under the Commodity category though? we’ve already talked about and Amazon too.  But in this Cloudy-Cloudy-*aaSy-*aaSy world, what else is flying high?  There’s room for a ton of things, but you have to solve the revenue and commodity checkboxes first.

Here is the problem for all you would-be PaaS vendors out there–it’s darned hard to get design wins if your platform involves heavy lock-in, heavy rewriting, too much cost, or isn’t otherwise a requirement for a Minimum Viable Product.   Yup, there’s that darned MVP concept rearing its ugly head again.  The trouble is, it’s no longer just something the Cool Kids bandy about while sipping lattes.  It’s become something of a requirement for survival and an article of How Things Are Done In The Valley.  You can’t get capital to fool around with fancy products any more, so you have to go MVP all the way because you’re likely starving on your own nickel until you do.  In addition to the VC’s, Agile thinking and a general suspicion of Premature Optimizaton have really made it hard to sell Feature Density.  The trouble with that list of things at the top is they’re either not that commonly needed, they involve dealing with scale you should be so lucky as to get to (and are therefore not MVP material), or they solve problems you’re convinced you can easily solve as you’re starting your journey (e.g. multitenant).  Yes, those problems are harder than you think, but it doesn’t matter.  They don’t look harder when you’re eyeing what it takes to get an MVP off the ground.

PaaS vendors, you have a couple of choices available to deal with this.  You can ignore it, argue that it’s early days yet and your time will come.  It’s pretty hard to dispute that because it is early days yet.  But don’t get too target fixated at such an early stage either lest some upstart take the early days away.  You can still be Zucked pretty darned easily precisely because it is early days.  Note: To be “Zucked” is to be treated to the same fait Facebook dished out to MySpace.  Call it Fast Follower on Steroids with a Heavy Case of Rabies.  It’s not a pleasant thing to happen to your life’s dream.

Okay, so what’s the alternative for would-be PaaS Masters of the Universe?

Hey, this is a good time for the Cloud.  You know that.  We hear less and less whining about security and all that.  Clever marketers are even now letting IT get just a little bit infected with “Private Clouds”, a potent Trojan Horse strategy to win over their hearts and minds.  Whatever it takes, resistance is futile.  Once their apps and data are on my servers it’s only a matter of reconfiguring the subnets and voila!  All your Apps are be in my Cloud!

Given that is the case, there may be some first mover, network effect, and momentum issues to think about.  In other words, stop being so darned pure to your vision and line up as many customers as you can as fast as you can.  That’s how we win in this part of the Bubble, um I mean Business, Cycle.

What the heck does that mean?

I’m glad you asked, but it should be obvious:  PaaS vendors need to embrace these two desirable features and nail them before worrying about much else.  There are two simple questions:

1.  Are you directly delivering revenue producing traffic to your customers by virtue of some aspect of your PaaS?

2.  Do you have an offering that lets people buy into some commodity-priced-let’s-get-started-without-boiling-the-ocean version of your PaaS?

If you do, Hallelujah Brothers and Sisters!  You have a shot at the promised land and you can now start looking at more potent differentiation rather than table stakes.  If you don’t, back up and let’s figure something out, because this PaaS stuff can turn into a Darwin Test if you’re not careful.

There is room for innovation on the commodity side, but it’s getting harder.  The days may be gone when you can deliver commodity infrastructure.  Storage and CPU like S3 and EC2, in other words.  If you aren’t already spun up and doing good things, you need to start skating where the puck is going to be.  I have offered up my PaaS-as-bite-sized-pieces strategy before (Sell the Condiments, not the Sandwiches).  Check it out.  Lots to commoditize there.

There is also room for tons of innovation on the “delivering revenue producing traffic” front.  We’ve seen it for mobile platforms, in fact, one could argue the appstores are the defining element there coupled with the relative desirability of the platforms to their users.  The participants seem to be pretty mercenary about those two related dimensions.  I am mystified about why Amazon, which ought to understand App Stores better than anyone, is not doing so great with their Android App Store and doesn’t appear to have one at all for Amazon Web Services.  The latter is inexcusable.  There’s got to be all sorts of opportunity to create an App Store there.

Salesforce keeps wanting to be a major PaaS vendor, but they somehow misunderstand the data they were among the first to collect.  Yes, they do have the revenue producing traffic piece nailed.  But, they seem to be very much in denial about whether that is the main reason people will use, and totally opposed to solving the commodity issue.  Every time I talk to an entrepreneur or investor that wonders whether they should use or one of its offshoots, I ask them to consider a simple thought experiment.  Look up Salesforce’s current cost of service as a percentage of revenue.  Take the cost they will be charged to use Force.comand divide by SFDC’s cost of service.  That number is what they must charge to have the same margins as Salesforce, and that assumes they don’t spend another dime on anything else to deliver their service.  Most of the time that makes for a short conversation.  Oh, I didn’t think about it like that.  There’s no way we can be competitive if we have to charge that much.  And so it goes with a lot of other PaaS offerings too, BTW.  Perhaps Heroku is their way of covering both bases and a sign that they do understand.

For other PaaS vendors, maybe there is a sliver of hope.  If you can change that cost of service to be cost of service plus some cost of marketing, because the PaaS will deliver revenue producing traffic, you can afford to pay more.  Heck, Steve Jobs gets 30% just for delivering the traffic and not helping you in much of any other way.

The revenue producing traffic is by far the hardest thing to do.  You can’t materialize it out of thin air.  You either already have a solid traffic stream you can repurpose (that’s what Salesforce did), or you may have to look at partnering opportunities.  For those that have a stream, now is your chance to enter the PaaS business in an interesting way.  Casting eyes around, Adobe is well positioned for this.  Get an AppStore together, Adobe, and link your dev tools and *aaS efforts to it.  There are bound to be others too.  Open Source vendors in the tools business, maybe you have this sort of opportunity as well.  IBM, Oracle, HP, and whomever else this is a huge opportunity for you.  Maybe Cisco too as I think about it.  Trouble is, your Big Sales Force may think it hampers them in some way.  Ignore their parochial interests and charge ahead.  This is a Silver Bullet for PaaS and Cloud ascendancy.

Give it some thought.  It’s high time for some break out PaaS action.

Posted in amazon, bootstrapping, business, cloud, platforms, saas, venture | 7 Comments »

Amazon: The Hidden Empire

Posted by Bob Warfield on May 13, 2011

An amazing read about Amazon and the strategy of creating a huge success on the web:

Amazon: The Hidden Empire by faberNovel

Why can’t the other Big Seattle Tech Company be remotely this smart?

Posted in amazon, business, cloud, service, strategy | Leave a Comment »

People Using Amazon Cloud: Get Some Cheap Insurance At Least

Posted by Bob Warfield on April 23, 2011

I’m reading through Twitter streams, Amazon Forums, and other news sources trying to get a sense of how users are responding and what their problems are.  It’s pretty appalling out there.  B2B companies admitting they have no recent backups and just have to wait for it to come back online.  A company that claims patient’s lives are at stake as they do cardiac monitoring based in the Amazon Cloud and are desperately seeking assistance.  The list goes on.

There’s some basic insurance any company using the Amazon Cloud needs to take out first chance they get.  It’s not hard, it’s not expensive, it’s not push a button and get hot failover to multiple Clouds, and it won’t fix your problems if you’re caught in the current outage.  But it will at least give you a little more maneuvering room.  Many of the acounts I’m reading boil down to a lack of options other than waiting because they have no accessible backup data.  In other words, they’d love to bring up their sites again on another Amazon Region, but they can’t because they’re missing access to a reasonably current data backup, or the Amazon Machine Instances are all in the affected region or issues along those lines.

Companies need the Cloud equivalent of offsite backup.  At a minimum, you need to be sure you can get access to a backup of your infrastructure–all the AMI’s and Data needed to restart.  Storage is cheap.  Heck, if you’re totally paranoid, turn the tables and backup the Cloud to your own datacenter which consists of just the backup infrastructure.  At least that way you’ve always got the data.  Yes, there will be latency issues and that data will not be up to the minute.  But look at all that’s happened.  Suppose you could’ve spun up in another region having lost 2 hours of data.  Not good, not good at all.  But is it really worse than waiting over 24 hours or would you be feeling blessed about now if you could’ve done it 2 hours into the emergency?  These are the kind of trade offs to be thinking about for disaster recovery.  It’s chewing gum and bailing wire until you get an architecture that’s more resilient, but it sure beats not having any choices and waiting.

Another thing: make sure you test your backups.  Do they restore?  Can you go through the exercise of spinning up in another region to see that it works?  Don’t just test once and forget about it.  Pick an interval and retest.  Make it routine so you know it works.

Staging all the data to other locations is not that expensive compared to continuously running dual failover infrastructure.  That’s one of the beauties of elasticity.

There’s a lot of grumbling about how hard it is to failover to other regions and how expensive.  Nothing is harder than explaining to your customers why your site is down.  But at least get some cheap insurance in place so you have options the next time this happens.  And there will be a next time, no matter whether it is Amazon, some other Cloud provider, or your own datacenter.  There is always a next time.

While you’re at it, consider some other cheap insurance:

- Do you have a way to communicate with your customers when your site is down?  An ops blog that you’re sure is hosted in a different cloud is cheap and cheerful.

- Can you at least get your web site home page showing?  Think about how to get DNS access and a place to host that don’t rely 100% on one Cloud provider.

- Is there something about your app that would make partial access in an outage valuable?  For example, on a customer service app, being able to log trouble tickets as email during an outage or scheduled downtime would be extremely helpful.  Mail is cheap and easy to offer as alternate infrastructure, and it is also easy to imagine piping the email messages through a converter that would file them as tickets when the site came back up.  It’s not hard to imagine being able to queue many kinds of transaction this way in an emergency.  What are the key limited-functionality areas your users will want to have access to in an emergency?

- For some apps, it is easier to provide high availability for reading than for writing.  Can you arrange that in an emergency, reading is still possible, just not writing or creating new objects?  Customers are a lot more tractable if they know they still have access to their data, but just can’t create new data for a while.  For example, a bookmarking site that lets me access my bookmarks but not create new ones during an outage is much less threatening than one that just brings up its Fail Whale equivalent on me.

Welcome to the world of Disaster Recovery.  Disasters have a User Experience too.  Have you planned your customer’s Disaster UX yet?

Posted in amazon, cloud | 12 Comments »

What to Do When Your Cloud is Down

Posted by Bob Warfield on April 21, 2011

Heroku status is down

This post is on  behalf of the Enterprise CIO Forum and HP.  

As I write this, Amazon is having a major East Coast outage that has affected Heroku, Foursquare, Quora, Reddit and others.  Heroku’s status page is just the sound of a lost sheep bleating repeatedly for its mother in heavy fog.  What’s a poor sheep to do about this problem anyway?  After all, isn’t a Cloud-based service dead once it’s Cloud is dead?

Rather than wringing our hands and shaking our heads about “That Darned Cloud, I knew this would happen”, let’s talk about it a bit, because there are some things that can and should be done.  Enterprises wanting to adopt the Cloud will want to have thought through these issues and not just avoided them by avoiding the Cloud.  In the end, they’re issues every IT group faces with their own infrastructure and there are strategies that can be used to minimize the damage.

I remember a conversation with a customer when I was a young Vice President of R&D at Borland, then a half a billion dollar a year software company (I miss it).  This particular customer was waxing eloquent about our Quattro Pro spreadsheet, but they just had one problem they wanted us to solve: they wanted Quattro Pro not to lose any data if the user was editing and there was a power outage.

I was flabbergasted.  “It’s a darned computer, it dies when you shut off the power!” I sputtered in only slightly more professional terms.  Of course I was wrong and hadn’t really thought the problem through.  With suitable checkpoints and logging, this is actually a fairly straightforward problem to solve and most of the software I use today deals with it just fine, thank you very much.

So it is with the Cloud.  Your first reaction may be, “We’re a Cloud Service, of course we go down if our Cloud goes down!”  But, it isn’t that black and white.  I like John Dodge’s thought that the Cloud should be treated just like rubber, sugar, and steel.  When Goodyear first started buying rubber from others, when Ford bought steel, and when Hershey’s bought sugar, do you think they didn’t take steps to ensure their suppliers wouldn’t control them?  Or take Apple.  Reports are that Japan’s recent tragedies aren’t impacting them much at all and that they’re absolutely sticking with their Japanese suppliers.  This has to come down to Apple and their suppliers having had a plan in place that was robust enough to weather even a disaster of these proportions.

What can be done?

First, this particular Amazon outage is apparently a regional outage, limited to the Virginia datacenter.   A look at Amazon’s status as I write this shows the West Coast infrastructure is doing okay:

Amazon: one up one downMost SaaS companies have to get huge before they can afford multiple physical data centers if they own the data centers.  But if you’re using a Cloud that offers multiple physical locations, you have the ability to have the extra security of multiple physical data centers very cheaply.  The trick is, you have to make use of it, but it’s just software.   A service like Heroku could’ve decided to spread the applications it’s hosting evenly over the two regions or gone even further afield to offshore regions.

This is one of the dark sides of multitenancy, and an unnecessary one at that.  Architects should be designing not for one single super apartment for all tenants, but for a relatively few apartments, and the operational flexibility to make it easy via dashboard to automatically allocate their tenants to whatever apartments they like, and then change their minds and seamlessly migrate them to new accommodations as needed.  This is a powerful tool that ultimately will make it easier to scale the software too, assuming its usage is decomposable to minimize communication between the apartments.  Some apps (Twitter!) are not so easily decomposed.

This then, is a pretty basic question to ask of your infrastructure provider: “How easy do you make it for me to access multiple physical data centers with attendant failover and backups?”  In this case, Amazon offers the capability, but Heroku took it back away for those who added it in their stack.  I suspect they’ll address this issue pretty shortly, but it would’ve been a good question to explore earlier, no?  Meanwhile, what about the other vendors you may be using that build on top of Amazon.  Do they make it easy to spread things around and not get taken out if one Amazon region goes down?  If not, why not?

Here’s the answer you’d like to hear:

We take full advantage of Amazon’s multiple regions.  We’ll make it easy if one goes down for your app to be up and running on the other within an SLA of X.

Note that they may charge you extra for that service and it may therefore be optional, but at least you’ve made an informed choice.  Certainly all the necessary underpinnings are available from Amazon to support it.  Note that there are some operational niceties I won’t get into too deeply here, but I do want to mention in passing that it is also possible to offer a continuum of answers to the above question that have to do with the SLA.  For example, at my last startup, we were in the Cloud as a Customer Service app and decided we wanted to be able to bring back the service in another region if the one we were in totally failed within 20 minutes and with no more than 5 minutes of data loss.  That pretty much dictated how we needed to use S3 (which is slow, but automatically ships your data to multiple physical data centers), EBS, and EC2 to deliver those SLA’s.  Smart users and PaaS vendors will look into packaging several options because you should be backed up to S3 regardless, so what you’re basically arguing about and paying extra for is how “warm” the alternate site is and how much has to be spun up from scratch via S3.

Another observation about this outage:  it is largely focused on EBS latency, though there is also talk of difficulty connecting to some EC2 instances.  This is the second time in recent history we’ve heard of some major EBS issues.  We read that Reddit had gone down over EBS latency issues less than a month ago.  Clearly anyone using EBS needs to be thinking about failure as a likely possibility.  In fact, the ReadWriteWeb article I linked to implies Reddit had been seeing EBS problems for quite some time.  One wonders if Heroku has too.

What will you do if you’re using EBS and it fails?  Reddits says they’re rearchitecting to avoid EBS.  That’s certainly one approach, but there may be others.  Amazon provides considerable flexibility in the combination of local disk, EBS, and S3 to fashion alternatives.  The trick is in making your infrastructure sufficiently metadata driven, and having thought throught the scenarios and tried them, sufficiently well-tested, that you can adapt in real-time when problems develop.  In this respect, I have seem Netflix admonish that the only way to test is to keep taking down aspects of your production infrastructure and making sure the system adapts properly.  That’s likely another good question to ask your PaaS and Cloud vendors–“Do you take down production infrastructure to test your failover?”  Of course you’d like to see that and not just take their word for it too.

I haven’t even touched on the possibilities of utilizing multiple Cloud vendors to ensure further redundancy and failover options.  It would be fascinating to see a PaaS service like S3 that is redundant across multiple data centers and multiple cloud vendors.  That seems like a real winner for building the kind of services that will be resilient to these kinds of outages.  It’s early days yet for the Cloud, even though some days it seems like Amazon has won.  There’s plenty of opportunity for innovators to create new solutions that avoid the problems we see today.  Even the experts like Heroku aren’t utilizing the Cloud as well as they should be.

Now is your chance!

This post is on  behalf of the Enterprise CIO Forum and HP.

Related Articles

James Cohen has some good thoughts on how to work around Amazon outages.

I tweeted: “The beauty of Cloud: We can blame Amazon instead of our IT when we’re down. Except we really can’t:

Excellent discussion here about how Netflix has a ton of assets on AWS and was unaffected.  In their words, they run on 3 regions and architected so that losing 1 would leave them running.  As Netflix says, “It’s cheaper than the cost of being down.”  Amen.  I’m seeing some anonymous posts whining about the exact definition of zones versus regions, what’s a poor EU service to do, etc., etc..  Study Netflix.  They’re up.  These other services are down.  Oh, and forget the anonymous comments.  Give your name like a real person and don’t be a lightweight.

Lots of comments here and there also that multi-cloud redundancy is hard.  Aside from the fact that this particular incident today didn’t require multiple clouds, consider that it is fantastically easier to go multi-cloud than it is to build multiple physical data centers. was almost a billion dollar a year company before they built a second data center.  Speaking of which, I bet they want to chat with the folks at Heroku now that they own them.

Clay Loveless gets failover in the Cloud.  JustinB, not so much.  Too ready to take Amazon’s word about their features.  Makes me wonder if folks early to AWS who saw it buggy and got used to dealing with that are better able to deal with problems like today’s?  When you run a service, it’s your problem, even when your Cloud vendor fails.  Gotta figure it out.

Lydia Leong (Gartner) and I don’t always agree, but she’s spot on with her analysis of the Amazon “failure” and what customers should have been doing about it.

EngineYard apparently was set to offer multiple AWS regions in Beta, and accelerated it to mitigate AWS problems for their customers.  Read their Twitter log on it.  Would love to hear from some of their customers that tried it how well it worked.

Posted in amazon, apple, business, cloud, strategy | 9 Comments »

Apple’s Crazy Ecosystem and the Prospects for an Amazon Pad

Posted by Bob Warfield on April 18, 2011

I read with interest Marco Arment’s speculation about an iPad competitor potentially being in the relatively near-term offing by Amazon:

I’d bet on Amazon releasing a true tablet, competing more directly with the iPad than the Kindle currently does, in the possibly near future.

While I don’t doubt Amazon could launch a tablet and very well might, I didn’t find Marco’s logic particularly compelling.  In his view, Amazon needs to launch a tablet to deal with the vagaries of Apple’s iOS ecosystem which may soon be hostile to Kindle and very favorable to iBooks.  I note my new iPhone (had to replace one) was immediately pushing the “assumptive close” on me to take iBooks.  No interest Apple: I’ve got Kindle and love it.

I’m also not in the camp of Robin Sloan who has a Kindle and an iPad and claims to use the Kindle 100x more than the iPad.  I used my Kindle well past the point of paying for it based on savings from my book purchases (I read a LOT of books), got an iPad, and I don’t think I have fired up the Kindle or missed it since.  Other than visibility in direct sunlight, I much prefer the iPad in every way.  And visibility in direct sunlight has not been an issue.  I see so little of it, being decamped in front of this darned LCD screen, that my iPad activity is largely before bed at night and any exposure to direct sunlight I will spend enjoying the direct sunlight and not with my nose stuck in my ‘pad.  After catching myself neglecting family by playing with the iPad or iPhone when we were out, I decided that was a bad idea that needed to stop and basically quit bringing them along unless I was off somewhere by myself.  Trust me, it’s a better way to be social.

The logic I got from Marco’s post was that this is largely about Apple’s 30% share of any sales connected with an app like Kindle together with their requirements of same pricing on their platform and their competitive position vis-a-vis iBooks.  He muses about what Apple’s customers will think if Apple boots Kindle.  I confess, I wonder what it would mean to boot Kindle?  Can Apple retroactively zap a happy healthy Kindle reader off my iOS device when I already paid for it?  Can they simply prevent it from being updated and strike it from the App Store?  Remembering how people reacted to Amazon’s retroactively redacting books from people’s Kindles I can well imagine that customers will not be enchanted to put it mildly.  Watching the actual process of giving the boot to the Kindle app will be fascinating.

I think Marco sees the timing of this event as a golden opportunity to persuade people to switch to a Kindle-Pad.  After all, if your cherished app is being zapped at precisely the time Amazon offers an alternative, why not?

But this all neglects the possibility that Amazon has both strong designs on their own aPad and no alternative but to go to war.  I’m not sure either makes compelling sense.

Let’s start with the “no alternative” question.  I will grant that Apple’s terms probably make little sense for the Kindle app.  By the time we slice another 30% off the top for Apple’s treasury, there’s little left for Amazon to share with book publishers.  But have we forgotten that Amazon has a browser-based solution in the wings?  It’s in Beta test today and is called Kindle for the Web.  It shouldn’t be difficult for Amazon to finish that up and redirect any iOS Kindle users to their browsers to access Kindle for the Web.  It isn’t like the user experience on the Kindle apps is so hard to recreate that they even need to be apps.  HTML5 should be more than up to the task, and no Flash in the iOS browser is needed, thank you very much.   It’s not clear to me why a tablet is even remotely needed to save the Kindle business on iOS unless Kindle for the Web is somehow tragically flawed or late.  That shouldn’t be a problem, or if it is, it’s hard to understand why the same organization wouldn’t fumble the much more complex prospect of building a decent tablet and getting customers to switch there.

What about owning a tablet of their own?  While it isn’t entirely clear, it sure looks like Amazon builds hardware as a necessary evil, not as a goal.  Think of their motivation as being somewhat like Google.  They fear hardware creating walled gardens that prevent access to their money makers, but beyond that, they are agnostic.  In fact, they have Google and their Android empire to keep Apple honest.  So what’s the strategic motivation for them to get involved in a messy ecosystem war?  So far, they been more interested in slashing the prices on Kindles than driving them up market into the tablet world.  It could be they just aren’t ready yet, or it could be they’re very deliberate and haven’t moved for strong strategic reasons.

Look at it this way–if they launch a full on tablet, aren’t they going to war with all the other tablets out there?  Do they really want to be they can get big with their tablet faster than the others can shut them down and off the competing platforms?  That doesn’t seem like a great bet to me, though they have a lot better information on which to make a decision.  I’d summarize the strategic arguments for both sides as follows:

Pro: Amazon should build a killer Android tablet ASAP and jump in swinging for the fence.

-  It’s gutsy and would garner a ton of press.

-  It would let them get their other offerings front and center in their owner’s faces and potentially increase sales.

-  They are gatekeepers for some pretty significant functions that a great tablet needs.  Books and music to name two.  It could be this lets them create a profoundly better user experience that wins quickly, at least to the #2 place, and becomes the predominant Android tablet.

-  They can quit worrying about the crazy strategic niceties and just fight a good clean up front fight.  It’s very focused.

-  The days of specialized readers like Kindle are numbered.  If Amazon doesn’t produce a tablet, it will be out of the reader business in a year or two, just like smart phones killed the Flip video camera.

Con:  Amazon wins biggest by nurturing the best ecosystem and not focusing on a device.

-  Amazon should stick to their knitting and core business.  Maximize its success on all platforms instead of on their own platform.

-  Amazon can ill-afford to polarize everyone against them.  What with telcos wanting to control the billing of everything on their platforms and Apple tightly controlling its ecosystem, Amazon has to move nimbly among the feet of these dancing elephants.

-  Amazon is already flirting with the envelope with respect to the content providers–book publishers and the record labels.  The book publishers got off a solid shot across the bow that forced Amazon to back off discounting their books so much on Kindle.  The record labels are not thrilled with Amazon’s new locker service.  They need to mind their strategic P’s and Q’s very carefully lest those players choose to make an Amazon Tablet a nasty little skirmish to demonstrate they’re still in control.

-  Isn’t the open non-denominational way really the more modern way?  Shouldn’t Amazon stick to making its money in the Cloud, where the maximum leverage and margins exist?  Let the rest of the rabble duke it out in the streets over these client devices.  So long as there are good open platforms available, Amazon need only rattle the occasional saber by making sure everyone knows they could field a tablet in short order.  It’s really better to be perceived as capable of moving in that direction than to actually have to pay the price of doing so.

- Smartphones didn’t kill Flip.  Cisco had no business being in Consumer, they blew their numbers, and Flip was the sacrificial lamb.  Kindle is being priced down where no general purpose tablet can go, and even if specialized readers go by the way side, Amazon owns so much of the book market the tablets will have to deal with it even if only via Kindle for the Web.  eBooks have lock-in due to network effects (I don’t want to lose access to my library by switching from Kindle to iBooks), so Amazon’s early win ensures their continued dominance.


I tried to be even-handed with the arguments so you can decide for yourself.  Which way should Amazon go?

I think it’s too early for them to go tablet, so I’ll stay on record as being in the “Con” camp, strategy-wise.   But, I’ll be interested to see which way Amazon goes.  If Amazon does go the full tablet route, I will be interested to see what moves they make to ameliorate some of the “Con” issues.  I suspect they will have some clever solutions to those problems that could very well shift the strategic balance of my pros and cons list.

Related Articles

eBooks have passed print books in terms of popularity.  Amazon has to control the lion’s share of that.  One more reason Apple had better tread carefully.

Rumor says Amazon tablet a done deal and Samsung will build it.

Posted in amazon, strategy | Leave a Comment »

Virtualization Made Mac What it is Today

Posted by Bob Warfield on February 18, 2011

Sam Diaz is writing about Apple’s latest Draconian App Store subscription policies and how they’re not a bad thing.  Forrester CEO George Colony says Apple is headed for a repeat of their defeat at the hands of Windows with these policies:

We know what happened — the world has had to use a lowest-common denominator PC operating system for decades, with excursions into wonderful places like Vista. This time around, Apple’s hostile position could result in a 2014 App Internet market that looks something like this: 80% Android, 10% Apple, 10% Other.

Colony’s concern is that this is the formative time for app consumption and app markets.  It’s too early to exert a monopolist’s egregious tax on those markets.  People aren’t locked in enough yet.

Diaz has a counter-argument:

Here’s the thing: Colony says that like it’s a bad thing. Say what you will about Apple’s share of the PC market – but the fact is that Apple’s lineup of Mac computers are far superior to anything that’s running Windows. And increasingly, quarter after quarter, the company notes that its share is growing and that about half of the Mac purchases in a single quarter have been by consumers who switched from Windows.

My problem with Sam’s argument is that none of that shift started happening until Virtualization meant you could have your Mac cake and eat some Windows software too.  It isn’t really clear they’re leaving the door open to do that with their App Store policies.  This isn’t about not only having Apple wonderfulness PLUS everything else in the world when Apple doesn’t happen to have the right answer.  It’s about ONLY having the Apple wonderfulness and being glad of it, dammit.

It’s going to be interesting to see what happens come the June 30 deadline for compliance with the new policies.  We will no doubt get hints along the way.  As an iPad user who set aside his Kindle but still constantly reads using the iPad’s Kindle app, I’m keenly interested.

During his last go-round with book publishers and Amazon, Steve Jobs largely managed to get book prices on Kindle raised.  That may turn out to be the result here too.  Kindle charges a “publishing expense” fee back to the book publishers.  So far it covers the wireless costs for Kindle’s built-in Sprint modem.  Perhaps Amazon will decide to roll the iPad 30% into that fee, making books sold there dramatically less profitable for publishers.  There would be a certain poetic justice in that.  The publishers leaned on Jobs to break one walled garden only to see another spring up immediately in its place.  What are they going to do about this one?






Posted in amazon, apple, business, cloud, Marketing, mobile | 1 Comment »

Where’s the Amazon AWS App Store?

Posted by Bob Warfield on February 7, 2011

I was talking to the interesting folks over at DreamFactory the other day (courtesy of Phil Wainewright who introduced us, thanks Phil!), and we wound up on a fascinating topic of conversation.

At a time when many companies are still not in the Cloud, DreamFactory is in 5 different clouds with as many as 8 different applications.  They’re doing some very cool stuff that I want to write more about in a later article, but suffice it to say they’re one of a very few companies who can seriously talk about the differences between Clouds.  For example, Bill Appleton, DreamFactory’s CTO, has published some benchmarks of the different clouds that are fascinating.

One of the topics that really grabbed me during our discussions was the strange dichotomy of who has an app store and who doesn’t.  DreamFactory has gotten tremendous mileage out of being listed on the app stores of various clouds.  CEO Eric Rubin tells me they receive at least 100 leads every day from Salesforce’s AppExchange alone.  Obviously, they’re very interested in Clouds having an app store, and in this day and age where app stores are all the rage, it makes a lot of sense.  At the same time, we have Amazon Web Services, one of the most popular if not the most popular Cloud out there, and they have no app store!  Yeah sure, they’ve put up a store for Android apps, but there’s no store for AWS apps. 

That’s a real oversight for Amazon.  I mean after all, they’re only a frickin’ online retailer for crying out loud–probably the world’s largest!  How strange is it that they have no app store for AWS developers?  As I think about it, there’s quite a lot of opportunity there for them.  Given my belief that Clouds will have strong network effects, I think it is a mistake for Amazon to go very much longer without an app store.

I did a search before writing this article and see that there has been some other speculation along these lines.  James Urquhart asks whether they couldn’t sell business apps in a Tweet, and Krishnan Subramanian also speculates in a blog post.

I’m a big believer in offering bite-sized modules that go together to make up a PaaS (Platform as a Service).  Amazon has a few of these, including the newly introduced bulk email service, but a full-on app store with billing capabilities would be great for a lot of smaller apps and startups to get going with.

Honestly, Amazon, what’s taking so long for this?  What are you guys thinking?  You should have been one of the first non-mobile app stores.  The handwriting’s been on the wall for some time!

Posted in amazon, business, cloud, saas | 4 Comments »

Single Tenant, Multitenant, Private and Public Clouds: Oh My!

Posted by Bob Warfield on August 27, 2010

My head is starting to hurt with all the back and forth among my Enterprise Irregulars buddies about the relationships between the complex concepts of Multitenancy, Private, and Public Clouds.  A set of disjoint conversations and posts came together like the whirlpool in the bottom of a tub when it drains.  I was busy with other things and didn’t get a chance to really respond until I was well and truly sucked into the vortex.  Apologies for the long post, but so many wonderful cans of worms finally got opened that I just have to try to deal with a few of them.  That’s why I love these Irregulars!

To start, let me rehash some of the many memes that had me preparing to respond:

Josh Greenbaum’s assertion that Multitenancy is a Vendor, not a Customer Issue.  This post includes some choice observations like:

While the benefits that multi-tenancy can provide are manifold for the vendor, these rationales don’t hold water on the user side.

That is not to say that customers can’t benefit from multi-tenancy. They can, but the effects of multi-tenancy for users are side-benefits, subordinate to the vendors’ benefits. This means, IMO, that a customer that looks at multi-tenancy as a key criteria for acquiring a new piece of functionality is basing their decision on factors that are not directly relevant to their TCO, all other factors being equal.


Multi-tenancy promises to age gracelessly as this market matures.

Not to mention:

Most of the main benefits of multi-tenancy – every customer is on the same version and is updated simultaneously, in particular – are vendor benefits that don’t intrinsically benefit customers directly.

The implication being that someone somewhere will provide an alternate technology very soon that works just as good or better than multitenancy.  Wow.  Lots to disagree with there.  My ears are still ringing from the sound of the steel gauntlet that was thrown down.

-  Phil Wainewright took a little of the edge of my ire with his response post to Josh, “Single Tenancy, the DEC Rainbow of SaaS.”  Basically, Phil says that any would-be SaaS vendor trying to create an offering without multitenancy is doomed as the DEC Rainbow was.  They have some that sort of walks and quacks like a SaaS offering but that can’t really deliver the goods.

-  Well of course Josh had to respond with a post that ends with:

I think the pricing and services pressure of the multi-tenant vendors will force single-tenant vendors to make their offerings as compatible as possible. But as long as they are compatible with the promises of multi-tenancy, they don’t need to actually be multi-tenant to compete in the market.

That’s kind of like saying, “I’m right so long as nothing happens to make me wrong.”  Where are the facts that show this counter case is anything beyond imagination?  Who has built a SaaS application that does not include multitenancy but that delivers all the benefits?

Meanwhile back at the ranch (we EI’s need a colorful name for our private community where the feathers really start to fly as we chew the bones of some good debates), still more fascinating points and counterpoints were being made as the topic of public vs private clouds came up (paraphrasing):

-  Is there any value in private clouds?

-  Do public clouds result in less lock-in than private clouds?

-  Are private clouds and single tenant (sic) SaaS apps just Old School vendors attempts to hang on while the New Era dawns?  Attempts that will ultimately prove terribly flawed?

-  Can the economics of private clouds ever compete with public?

-  BTW, eBay now uses Amazon for “burst” loads and purchases servers for a few hours at a time on their peak periods.  Cool!

-  Companies like Eucalyptus and Nimbula are trying to make Private Clouds that are completely fungible with Public Clouds.  If you  in private cloud frameworks like these means you have
to believe companies are going to be running / owning their own servers for a long time to come even if the public cloud guys take over a number of compute workloads.  The Nimbula guys built EC2 and they’re no dummies, so if they believe in this, there must be something to it.

-  There are two kinds of clouds – real and virtual.  Real clouds are multi-tenant. Virtual clouds are not. Virtualization is an amazing technology but it can’t compete with bottoms up multi-tenant platforms and apps.

Stop!  Let me off this merry go-round and let’s talk.

What It Is and Why Multitenancy Matters

Sorry Josh, but Multitenancy isn’t marketing like Intel Inside (BTW, do you notice Intel wound up everywhere anyway?  That wasn’t marketing either), and it matters to more than just vendors.  Why?

Push aside all of the partisan definitions of multitenancy (all your customers go in the same table or not).   Let’s look at the fundamental difference between virtualization and multitenancy, since these two seem to be fighting it out.

Virtualization takes multiple copies of your entire software stack and lets them coexist on the same machine.  Whereas before you had one OS, one DB, and one copy of your app, now you may have 10 of each.  Each of the 10 may be a different version entirely.  Each may be a different customer entirely, as they share a machine.  For each of them, life is just like they had their own dedicated server.  Cool.  No wonder VMWare is so successful.  That’s a handy thing to do.

Multitenancy is a little different.  Instead of 10 copies of the OS, 10 copies of the DB, and 10 copies of the app, it has 1 OS, 1 DB, and 1 app on the server.  But, through judicious modifications to the app, it allows those 10 customers to all peacefully coexist within the app just as though they had it entirely to themselves.

Can you see the pros and cons of each?  Let’s start with cost.  Every SaaS vendor that has multitenancy crows about this, because its true.  Don’t believe me?  Plug in your VM software, go install Oracle 10 times across 10 different virtual machines.  Now add up how much disk space that uses, how much RAM it uses when all 10 are running, and so on.  This is before you’ve put a single byte of information into Oracle or even started up an app.  Compare that to having installed 1 copy of Oracle on a machine, but not putting any data into it.  Dang!  That VM has used up a heck of a lot of resources before I even get started!

If you don’t think that the overhead of 10 copies of the stack has an impact on TCO, you either have in mind a very interesting application + customer combination (some do exist, and I have written about them), or you just don’t understand.  10x the hardware to handle the “before you put in data” requirements are not cheap.  Whatever overhead is involved in making that more cumbersome to automate is not cheap.  Heck, 10x more Oracle licenses is very not cheap.  I know SaaS companies who complain their single biggest ops cost is their Oracle licenses. 

However, if all works well, that’s a fixed cost to have all those copies, and you can start adding data by customers to each virtual Oracle, and things will be okay from that point on.  But, take my word for it, there is no free lunch.  The VM world will be slower and less nimble to share resources between the different Virtual Machines than a Multitenant App can be.  The reason is that by the time it knows it even needs to share, it is too late.  Shifting things around to take resource from one VM and give it to another takes time.  By contrast, the Multitenant App knows what is going on inside the App because it is the App.  It can even anticipate needs (e.g. that customer is in UK and they’re going to wake up x hours before my customers in the US, so I will put them on the same machine because they mostly use the machine at different times).

So, no, there is not some magic technology that will make multitenant obsolete.  There may be some new marketing label on some technology that makes multitenancy automatic and implicit, but if it does what I describe, it is multitenant.  It will age gracefully for a long time to come despite the indignities that petty competition and marketing labels will bring to bear on it.

What’s the Relationship of Clouds and Multitenancy?

Must Real Clouds be Multitenant?

Sorry, but Real Clouds are not Multitenant because they’re based on Virtualization not Multitenancy in any sense such as I just defined.  In fact, EC2 doesn’t share a core with multiple virtual machines because it can’t.  If one of the VM’s started sucking up all the cycles, the other would suffer terrible performance and the hypervisors don’t really have a way to deal with that.  Imagine having to shut down one of the virtual machines and move it onto other hardware to load balance.  That’s not a simple or fast operation.  Multi-tasking operating systems expect a context switch to be as fast as possible, and that’s what we’re talking about.  That’s part of what I mean by the VM solution being less nimble.  So instead, cores get allocated to a particular VM.  That doesn’t mean a server can’t have multiple tenants, just that at the granularity of a core, things have to be kept clean and not dynamically moved around. 

Note to rocket scientists and entrepreneurs out there–if you could create a new hardware architecture that was really fast at the Virtual Machine load balancing, you would have a winner.  So far, there is no good hardware architecture to facilitate a tenant swap inside a core at a seamless enough granularity to allow the sharing.  In the Multicore Era, this would be the Killer Architecture for Cloud Computing.  If you get all the right patents, you’ll be rich and Intel will be sad.  OTOH, if Intel and VMWare got their heads together and figured it out, it would be like ole Jack Burton said, “You can go off and rule the universe from beyond the grave.”

But, it isn’t quite so black and white.  While EC2 is not multitenant at the core level, it sort of is at the server level as we discussed.  And, services like S3 are multitenant through and through.  Should we cut them some slack?  In a word, “No.”  Even though an awful lot of the overall stack cost (network, cpu, and storage) is pretty well multitenant, I still wind up installing those 10 copies of Oracle and I still have the same economic disadvantage as the VM scenario.  Multitenancy is an Application characteristic, or at the very least, a deep platform characteristic.  If I build my app on, it is automatically multitenant.  If I build it on Amazon Web Services, it is not automatic.

But isn’t there Any Multitenant-like Advantage to the Cloud?  And how do Public and Private Compare?

Yes, there are tons of benefits to the Cloud, and through an understanding and definition of them, we will tease out the relationship of Public and Private Clouds.  Let me explain…

There are two primary advantages to the Cloud:  it is a Software Service and it is Elastic.  If you don’t have those advantages, you don’t have a Cloud.  Let’s drill down.

The Cloud is a Software Service, first and foremost.  I can spin up and control a server entirely through a set of API’s.  I never have to go into a Data Center cage.  I never have to ask someone at the Data Center to go into the Cage (though that would be a Service, just not a Software Service, an important distinction).  This is powerful for basically the same reasons that SaaS is powerful versus doing it yourself with On-prem software.  Think Cloud = SaaS and Data Center = On Prem and extrapolate and you’ll have it. 

Since Cloud is a standardized service, we expect all the same benefits as SaaS:

- They know their service better than I do since it is their whole business.  So I should expect they will run it better and more efficiently.

- Upgrades to that service are transparent and painless (try that on your own data center, buddy!).

- When one customer has a problem, the Service knows and often fixes it before the others even know it exists.  Yes Josh, there is value in SaaS running everyone on the same release.  I surveyed Tech Support managers one time and asked them one simple question:  How many open problems in your trouble ticketing system are fixed in the current release?  The answers were astounding–40 to 80%.  Imagine a world where your customers see 40 to 80% fewer problems.  It’s a good thing!

- That service has economic buying power that you don’t have because it is aggregated across many customers.  They can get better deals on their hardware and order so much of it that the world will build it precisely to their specs.  They can get stuff you can’t, and they can invest in R&D you can’t.  Again, because it is aggregated across many customers.  A Startup running in the Amazon Cloud can have multipe redundant data centers on multiple continents.  Most SaaS companies don’t get to building multiple data centers until they are way past having gone public. 

-  Because it is a Software Service, you can invest your Ops time in automation, rather than in crawling around Data Center cages.  You don’t need to hire anyone who knows how to hot swap a disk or take a backup.  You need peeps who know how to write automation scripts.  Those scripts are a leveragable asset that will permanently lower your costs in a dramatic way.  You have reallocated your costs from basic Data Center grubbing around (where does this patch cable go, Bruce?), an expense, to actually building an asset.

The list goes on.

The second benefit is Elasticity.  It’s another form of aggregation benefit.  They have spare capacity because everyone doesn’t use all the hardware all the time.  Whatever % isn’t utilized, it is a large amount of hardware, because it is aggregated.  It’s more than you can afford to have sitting around idle in your own data center.  Because of that, they don’t have to sell it to you in perpetuity.  You can rent it as you need it, just like eBay does for bursting.  There are tons of new operational strategies that are suddenly available to you by taking advantage of Elasticity.

Let me give you just one.  For SaaS companies, it is really easy to do Beta Tests.  You don’t have to buy 2x the hardware in perpetuity.  You just need to rent it for the duration of the Beta Test and every single customer can access their instance with their data to their heart’s content.  Trust me, they will like that.

What about Public Versus Private Clouds?

Hang on, we’re almost there, and it seems like it has been a worthwhile journey.

Start with, “What’s a Private Cloud?”  Let’s take all the technology of a Public Cloud (heck, the Nimbulla guys built EC2, so they know how to do this), and create a Private Cloud.  The Private Cloud is one restricted to a single customer.  It’d be kind of like taking a copy of’s software, and installing it at Citibank for their private use.  Multitenant with only one tenant.  Do you hear the sound of one hand clapping yet?  Yep, it hurts my head too, just thinking about it.  But we must.

Pawing through the various advantages we’ve discussed for the Cloud, there are still some that accrue to a Cloud of One Customer:

-  It is still a Software Service that we can control via API’s, so we can invest in Ops Automation.  In a sense, you can spin up a new Virtual Data Center (I like that word better than Private Cloud, because it’s closer to the truth) on 10 minutes notice.  No waiting for servers to be shipped.  No uncrating and testing.  No shoving into racks and connecting cables.  Push a button, get a Data Center.

-  You get the buying power advantages of the Cloud Vendor if they supply your Private Cloud, though not if you buy software and build  your Private Cloud.  Hmmm, wonder what terminology is needed to make that distinction?  Forrester says it’s either a Private Cloud (company owns their own Cloud) or a Hosted Virtual Private Cloud.  Cumbersome.

But, and this is a huge one, the granularity is huge, and there is way less Elasticity.  Sure, you can spin up a Data Center, but depending on its size, it’s a much bigger on/off switch.  You likely will have to commit to buy more capacity for a longer time at a bigger price in order for the Cloud Provider to recoup giving you so much more control.  They have to clear other customers away from a larger security zone before you can occupy it, instead of letting your VM’s commingle with other VM’s on the same box.  You may lose the more multitenant-like advantages of the storage cluster and the network infrastructure (remember, only EC2 was stuck being pure virtual). 

What Does it All Mean, and What Should My Company Do?

Did you see Forrester’s conclusion that most companies are not yet ready to embrace the Cloud and won’t be for a long time?

I love the way Big Organizations think about things (not!).  Since their goal is preservation of wealth and status, it’s all about risk mitigation whether that is risk to the org or to the individual career.  A common strategy is to take some revolutionary thing (like SaaS, Multitenancy, or the Cloud), and break it down into costs and benefits.  Further, there needs to be a phased modular approach that over time, captures all the benefits with as little cost as possible.  And each phase has to have a defined completion so we can stop, evaluate whether we succeeded, celebrate the success, punish those who didn’t play the politics well enough, check in with stakeholders, and sing that Big Company Round of Kumbaya.  Yay!

In this case, we have a 5 year plan for CIO’s.  Do you remember anything else, maybe from the Cold War, that used to work on 5 year plans?  Never mind.

It asserts that before you are ready for the Cloud, you have to cross some of those modular hurdles:

A company will need a standardized operating procedure, fully-automated deployment and management (to avoid human error) and self-service access for developers. It will also need each of its business divisions – finance, HR, engineering, etc – to be sharing the same infrastructure.  In fact, there are four evolutionary stages that it takes to get there, starting with an acclimation stage where users are getting used to and comfortable with online apps, working to convince leaders of the various business divisions to be guinea pigs. Beyond that, there’s the rollout itself and then the optimization to fine-tune it.

Holy CYA, Batman!  Do you think eBay spent 5 years figuring out whether it could benefit from bursting to the Cloud before it just did it?

There’s a part of me that says if your IT org is so behind the times it needs 5 years just to understand it all, then you should quit doing anything on-premise and get it all into the hands of SaaS vendors.  They’re already so far beyond you that they must have a huge advantage.  There is a another part that says, “Gee guys, you don’t have to be able to build an automobile factory as good as Toyota to be able to drive a car.”

But then sanity and Political Correctness prevail, I come back down to Earth, and I realize we are ready to summarize.  There are 4 levels of Cloud Maturity (Hey, I know the Big Co IT Guys are feeling more comfortable already, they can deal with a Capability and Maturity Model, right?):

Level 1:  Dabbling.  You are using some Virtualization or Cloud technology a little bit at your org in order to learn.  You now know what a Machine Image is, and you have at least seen a server that can run them and swapped a few in and out so that you experience the pleasures of doing amazing things without crawling around the Data Center Cage.

Level 2:  Private Cloud.  You were impressed enough by Level 1 that you want the benefits of Cloud Technology for as much of your operation as you can as fast as you can get it.  But, you are not yet ready to relinquish much of any control.  For Early Level 2, you may very well insist on a Private Cloud you own entirely.  Later stage Level 2 and you will seek a Hosted Virtual Private Cloud.

Level 3:  Public Cloud.  This has been cool, but you are ready to embrace Elasticity.  You tripped into it with a little bit of Bursting like eBay, but you are gradually realizing that the latency between your Data Center and the Cloud is really painful.  To fix that, you went to a Hosted Virtual Private Cloud.  Now that your data is in that Cloud and Bursting works well, you are realizing that the data is already stepping outside your Private Cloud pretty often anyway.  And you’ve had to come to terms with it.  So why not go the rest of the way and pick up some Elasticity?

Level 4:  SaaS Multitenant.  Eventually, you conclude that you’re still micromanaging your software too much and it isn’t adding any value unique to your organization.  Plus, most of the software you can buy and run in your Public Cloud world is pretty darned antiquated anyway.  It hasn’t been rearchitected since the late 80’s and early 90’s.  Not really.  What would an app look like if it was built from the ground up to live in the Cloud, to connect Customers the way the Internet has been going, to be Social, to do all the rest?  Welcome to SaaS Multitenant.  Now you can finally get completely out of Software Operations and start delivering value.

BTW, you don’t have to take the levels one at a time.  It will cost you a lot more and be a lot more painful if you do.  That’s my problem with the Forrester analysis.  Pick the level that is as far along as you can possibly stomach, add one to that, and go.  Ironically, not only is it cheaper to go directly to the end game, but each level is cheaper for you on a wide scale usage basis all by itself.  In other words, it’s cheaper for you to do Public Cloud than Private Cloud.  And it’s WAY cheaper to go Public Cloud than to try Private Cloud for a time and then go Public Cloud.  Switching to a SaaS Multitenant app is cheaper still.

Welcome to crazy world of learning how to work and play well together when efficiently sharing your computing resources with friends and strangers!

Posted in amazon, cloud, data center, ec2, enterprise software, grid, multicore, platforms, saas, service | 14 Comments »


Get every new post delivered to your Inbox.

Join 321 other followers

%d bloggers like this: