MySQL and BEA: Oracle and Sun Will Be At Each Other’s Throats!

Big news today is that Sun is buying MySQL and Oracle is buying BEA. This creates a couple of strange bedfellows to say the least. BEA is inextricably wrapped up in Sun’s Java business (is it really a business or just a hobby given the revenues it doesn’t produce?) which gives a reason for the two to get closer together. On the other hand, there is hardly a bigger threat to Oracles core database server business imaginable than MySQL, which has got to push the two companies further apart. What a tangled web!  Is Sun leaving Oracle to its own devices in order to pursue cloud computing?  Sure looks like it!

Let’s analyze these moves a bit. I want to start with BEA and Oracle.

As we all know, Oracle started that courtship dance not long ago and was rebuffed for not offering enough.  Amusingly, they closed almost exactly at the midpoint of the prices the two argued were “fair” at the outset.  Meanwhile, the recession is really setting in, stock prices are falling, and Oracle’s offer went up.  Since Cisco’s John Chambers mused about IT spending will slowing, it has become a widely accepted article that this will happen. So shall it be said, so shall it be written, Mr. Chambers. That’s a very bad thing for BEA, which is primarily selling to that market. The corporate IT market is their bread and butter for a number of reasons. Many ISV’s and web companies will look to Open Source solutions like Tomcat or JBoss with which to reduce costs. Corporate IT wants to superior support of a big player like BEA. The darker truth is that big Java seems to be falling out of favor among the bleeding edge crowd. Java itself gets a lot of criticism, but is strong enough to take it. J2EE is another matter, though there is still a huge amount of it going on. There is also the matter of the steady ascendency of RESTful acrchitecture while BEA is one of the lynchpins of Big SOA.  There is already posturing about the importance of BEA to Oracle Fusion.  If it is so important, Fusion may be born with an obsolete architecture from day one. 

The long and the short is that any competent tea leaf reader (is there any such thing?) would conclude that this was a good move for BEA to let themselves be bought before their curve has crested too much more. For Oracle’s part, its a further opportunity to consolidate their Big Corporate IT Hedgemony and to feed their acquisition-based growth machine. I am not qualified to say whether they paid too much or not, but if I do think the value curve for BEA is falling and will continue to fall post-acquisition. They are way late on the innovation curve, which looks to me like it has already fallen.  In short, BEA is a pure bean counting exercise: milk the revenue tail as efficiently as possible and then move on.  For this Oracle paid $8.5B.  Not surprisingly, even though it is a much bigger transaction, there is much less about it on the blogosphere as I write this than about the other transaction.

Speaking of which, let’s turn to the Sun+MySQL combination.  Jonathan Schwartz gets a bit artsy with his blog post introducing the introduction, which he calls “Teach dolphins to fly.”  The metaphor is apropos.  Schwartz says that MySQL is the biggest database up and comer news in the world of network computing (that’s how we say cloud computing without offending the dolphins that haven’t figured out how to fly yet).  What Sun will bring to the table is credibility, solidity, and support.  He talks about Fortune 500 needing all that in the guise of:

Global Enterprise Support for MySQL - so that traditional enterprises looking for the same mission critical support they’ve come to expect with proprietary databases can have that peace of mind with MySQL, as well.

That business of “proprietary databases” means Oracle.  Jonathan just fired a good sized projectile across your bow Mr. Ellison.  What do you think of that? 

I know what I think.  Getting my tea leaf reading union card back out, I compare these two big acquisitions and walk away with a view that Oracle paid $8.5B to carve up an older steer and have a BBQ while Sun paid $1B to buy the most promising race horse to win the Kentucky Derby.  What a brilliant move for Sun!  Now they’ve united a couple of the big elements out there, Java being one and MySQL the other.  They could stand to add a decent scripting language, but unlike Microsoft’s typical tactics, they’ve learned not to ply a scorched earth policy towards other platforms, so they are peacefully coexisting until a better cohabitation arrangement comes along. 

We talked a little about the Oracle transaction being a good deal for BEA:  it’s a lucrative exit from declining fortunes.  What about mySQL?  Zack Urlocker comments about the rumor everyone knew, that MySQL had been poised to go public.  Let me tell you: this is a far better move.  Savvy private companies get right to the IPO alter, and then they find someone to buy them for a premium over what they would go out at.  What they gain in return is potentially huge.  The best possible example of this was VMWare.  Now look where they are.  I will argue that would not have been possible without the springboard of EMC.  At least not this quickly.   Sun offers the same potential for MySQL.  It is truly the biggest open source deal in history.  It’s also a watershed liquidity event for a highly technical platform based offering from a sea of consumer web offerings.  The VC’s have been pretty tepid about new deals like MySQL.  Perhaps this will help more innovations to get funded.

What do others have to say about the deal?

 - Tim O’Reilly echoes the big open source and importance of database to platform themes.

 - Larry Dignan picks up on my rather combative title theme by pointing out that it puts Sun at war with the major DB vendors:  Microsoft, IBM and Oracle.  Personally, I think any overt combat will hurt those three.  The Open Source movement holds the higher moral ground and it just won’t be good PR to buck that too publicly.  Dignan sounds like he is making a little light of Schwartz’s conference call remark that it is the most important acquisition in Sun’s history, but I think that is no exaggeration on Jonathan’s part.  This is a hugely strategic move that affects every aspect of how Sun interfaces with the world computing ecosystem including its customers, many partners, and its future.  When Dignan asks what else Sun needs, I would argue a decent scripting language.  Since Google already has Python in hand, what about buying a company like Zend to get a leg up on PHP?  Last point from Larry is he asks, “If Sun makes MySQL more enterprise acceptable does that diminish its mojo with startups? Does it matter?”  Bottom line: improvements for the Enterprise in no way diminish what makes MySQL attractive to startups, providing Sun minds its manners.  So far it has been a good citizen.  With regards to, “Does it matter?”  Yes, it matters hugely.  MySQL is tapped into all the megatrends that lead to the future.  Startups are a part of that.  Of course that matters.

One other thought I’ve had:  what if Sun decides to build the ultimate database appliance?  I’m talking about order it, plug your CAT5 cable in, and forget about it.  Do for dabases what disk arrays did for storage.  That seems to me a powerful combination.  Database servers require a painful amount of care and feeding to install and administer properly.  If Sun can convert them to appliances, it kills two birds with one stone.  First, it becomes a powerful incentive to buy more Sun hardware.  This will even help more fully monetize MySQL, which apparently only gets revenue from 1 in 10,000 users.  Second, it could radically simplify and commoditze a piece of the software and cloud computing fabric that is currently expensive and painful.  Such a move would be a radical revolution that would perforce drive a huge revenue opportunity for Sun.  They have enough smart people between Sun and MySQL to pull it off if they have the will. 

Conclusion

Sun has made an uncannily good move in acquiring MySQL.  As Wired points out:

One company that won’t be thrilled by the news is Oracle, makers of the Oracle database which has managed to seduce a large segment of the enterprise market into the proprietary Oracle on the basis that the open source options lacked support.

With Sun backing the free MySQL option (and offering paid support) Oracle suddenly looks a bit expensive.

How else can you simultaneously lay a bet on owning a substantial piece of the computing fabric that all future roads are pointing to and send a big chill down Larry Ellison’s spine for the low low price of just $1B?  Awesome move, Jonathan!

Related Articles

VARGuy says the acquisition means Sun finally matters again.  $1B is cheap to “finally matter again!”

Eventual Consistency Is Not That Scary

Amazon’s new SimpleDB offering, like many other post-modern databases such as CouchDB, offers massive scaling potential if users will accept eventual consistency.  It feels like a weighty decision.  Cast in the worst possible light, eventual consistency means the database will sometimes return the wrong answer in the interests of allowing it to keep scaling.  Gasp!  What good is a database that returns the wrong answer?  Why bother? 

Often waiting for the write answer (sorry, that inadvertant slip makes for a good pun so I’ll leave it in place) returns a different kind of wrong answer.  Specifically, it may not return an answer at all.  The system may simply appear to hang. 

How does all this come about?  Largely, it’s a function of how fast changes in the database can be propogated to the point they’re available to everyone reading from the database.  For small numbers of users (i.e. we’re not scaling at all), this is easy.  There is one copy of the data sitting in a table structure, we lock up the readers so they can’t access it whenever we change that data, and everyone always gets the right answer.  Of course, solving simple problems is always easy.  It’s solving the hard problems that lands us the big bucks.  So how do we scale that out?  When we reach a point where we are delivering that information from that one single place as fast as it can be delivered, we have no choice but to make more places to deliver from.  There are many different mechanisms for replicating the data and making it all look like one big happy (but sometimes inconsistent) database, let’s look at them.

Once again, this problem may be simpler when cast in a certain way.  The most common and easiest approach is to keep one single structure as the source of truth for writing, and then replicate out changes to many other databases for reading.  All the common database software supports this.  If your single database could handle 100 users consistently, you can imagine if those 100 users were each another database you were replication to, suddenly you could handle 100 * 100 users, or 10,000 users.  Now we’re scaling.  There are schemes to replicate the replicated and so on and so forth.  Note that in this scenario, all writing must still be done on the one single database.  This is okay, because for many problems, perhaps even the majority, readers far outnumber writers.  In fact, this works so well, that we may not even use databases for the replication.  Instead, we might consider a vast in-memory cache.  Software such as memcached does this for us quite nicely, with another order of magnitude performance boost since reading things in memory is dramatically faster than trying to read from disk.

Okay, that’s pretty cool, but is it consistent?  This will depend on how fast you can replicate the data.  If you can get every database and cache in the system up to date between consecutive read requests, you are sure to be consistent.  In fact, it just has to get done between read requests for any piece of data that changed, which is a much lower bar to hurdle.  If consistency is critical, the system may be designed to inhibit reading until changes have propogated.  It take some very clever algorithms to do this well without throwing a spanner into the works and bringing the system to its knees performance-wise. 

Still, we can get pretty far.  Suppose your database can service 100 users with reads and writes and keep it all consistent with appropriate performance.  Let’s say we replace those 100 users with 100 copies of your database to get up to 10,000 users.  It’s now going to take twice as long.  During the first half, we’re copying changes from the Mother Server to all of the children.  The second half we’re serving the answers to the readers requesting them.  Let’s say we can keep the overall time the same just by halving how many are served.  So the Mother Server talks to 50 children.  Now we can scale to 50 * 50 = 2500 users.  Not nearly as good, but still much better than not scaling at all.  We can go 3 layers deep and have Mother serve 33 children each serve 33 grand children to get to 33 * 33 * 33 = 35,937 users.  Not bad, but Google’s founders can still sleep soundly at night.  The reality is we probably can handle a lot more than 100 on our Mother Server.  Perhaps she’s good for 1000.  Now the 3-layered scheme will get us all the way to 333*333*333 = 36 million.  That starts to wake up the sound sleepers, or perhaps makes them restless.  Yet, that also means we’re using over 100,000 servers too: 1 Mothers talks to 333 children who each have 333 grandchildren.  It’s a pretty wasteful scheme.

Well, let’s bring in Eventual Consistency to reduce the waste.  Assume you are a startup CEO.  You are having a great day, because you are reading the wonderful review of your service in Techcrunch.  It seems like the IPO will be just around the corner after all that gushing does it’s inevitable work and millions suddenly find their way to your site.  Just at the peak of your bliss, the CTO walks in and says she has good news and bad news.  The bad news is the site is crashing and angry emails are pouring in.  The other bad news is that to fix it “right”, so that the data stays consistent, she needs your immediate approval to purchase 999 servers so she can set up a replicated scheme that runs 1 Mother Server (which you already own) and 999 children.  No way, you say.  What’s the good news?  With a sly smile, she tells you that if you’re willing to tolerate a little eventual consistency, your site could get by on a lot fewer servers than 999.

Suppose you are willing to have it take twice as long as normal for data to be up to date.  The readers will read just as fast, it’s just that if they’re reading something that changed, it won’t be correct until the second consecutive read or page refresh.  So, our old model that had the system able to handle 1,000 users, and replicated to 999 servers to handle 1 million users used to have to go to 3 tiers (333 * 333 * 333) to get to the next level at 36 million and still serve everything consistently and just as fast.  If we relax the “just as fast”, we can let our Mother Server handle 2,000 at half the speed to get to 2000 * 1000 = 2 million users on 3 tiers with 2000 servers instead of 100,000 servers to get to 36 million. If we run 4x slower on writes, we can get 4000*1000 = 4 million users with 4000 servers.  Eventually things will bog down and thrash, but you can see how tolerating Eventual Consistency can radically reduce your machine requirements in this simple architecture.  BTW, we all run into Eventual Consistency all the time on the web, whether or not we know it.  I use Google Reader to read blogs and WordPress to write this blog.  Any time a page refresh shows you a different result when you didn’t change anything, you may be looking at Eventual Consistency.  Even if you suspect others changed something, Google Reader still comes along frequently and says an error occured and asks me to refresh.  It’s telling me they relied on Eventual Consistency and I have an inconsistent result.

As I mention, these approaches can still be wasteful of servers because of all the data copies that are flowing around.  This leads us to wonder, “What’s the next alternative?”  Instead of just using servers to copy data to other servers, which is a prime source of the waste, we could try to employ what’s called a sharded or Federated architecture.  In this approach, there is only one copy of each piece of data, but we’re dividing up that data so that each server is only responsible for a small subset of it.  Let’s say we have a database keeping up with our inventory for a big shopping site.  It’s really important to have it be consistent so that when people buy, they know the item was in stock.  Hey, it’s a contrived example and we know we can cheat on it, but go with it.  Let’s further suppose we have 100,000 SKU’s, or different kinds of items in our inventory.  We can divide this across 100 servers by letting each server be responsible for 1,000 items.  Then we write some code that acts as the go-between with the servers.  It simply checks the query to see what you are looking for, and sends your query to the correct sub-server.  Voila, you have a sharded architecture that scales very efficiently.  Our replicated model would blow out 99 copies from the 1 server, and it could be about 50 times faster (or handle 50x the users as I use a gross 1/2 time estimate for replication time) on reads, but it was no faster at all on writes.  That wouldn’t work for our inventory problem because writes are so common during the Christmas shopping season. 

Now what are the pitfalls of sharding.  First, there is some assembly required.  Actually, there is a lot of assembly required.  It’s complicated to build such architectures.  Second, it may be very hard to load balance the shards.  Just dividing up the product inventory across 100 servers is not necessarily helpful.  You would want to use a knowledge of access patterns to divide the products so the load on each server is about the same.  If all the popular products wound up on one server, you’d have a scaling disaster.  These balances can change over time and have to be updated, which brings more complexity.  Some say you never stop fiddling with the tuning of a sharded architecture, but at least we don’t have Eventual Consistency.  Hmmm, or do we?  If you can ever get into a situation where there is more than one copy of the data and the one you are accessing is not up to date, Eventual Consistency could rear up as a design choice made by the DB owners.  In that case, they just give you the wrong answer and move on. 

How can this happen in the sharded world?  It’s all about that load balancing.  Suppose our load balancer needs to move some data to a different shard.  Suppose the startup just bought 10 more servers and wants to create 10 additional shards.  While that data is in motion, there are still users on the site.  What do we tell them?  Sometimes companies can shut down the service to keep everything consistent while changes are made.  Certainly that is  one answer, but it may annoy your users greatly.  Another answer is to tolerate Eventual Consistency while things are in motion with a promise of a return to full consistency when the shards are done rebalancing.  Here is a case where the Eventual Consistency didn’t last all that long, so maybe that’s better than the case where it happens a lot. 

Note that consistency is often in the eye of the beholder.  If we’re talking Internet users, ask yourself how much harm there would be if a page refresh delivered a different result.  In may applications, the user may even expect or welcome a different result.  An email program that suddenly shows mail after a refresh is not at all unexpected.  That the user didn’t know the mail was already on the server at the time of the first refresh doesn’t really hurt them.  There are cases where absolute consistency is very important.  Go back to the sharded database example.  It is normal to expect every single product in the inventory to have a unique id that lets us find that part.  Those ids have to be unique and consistent across all of the shards.  It is crucially important that any id changes are up to date before anything else is done or the system can get really corrupted.  So, we may create a mechanism to generate consistent ids across shards.  This adds still more architectural complexity.

There are nightmare scenarios where it becomes impossible to shard efficiently.  I will over simplify to make it easy and not necessarily correct, but I hope you will get the idea.  Suppose you’re dealing with operations that affect many different objects.  The objects are divided into shards naturally when examined individually, but the operations between the objects span many shards.  Perhaps the relationships between shards are incompatible to the extent that there is no way to shard them across machines such that every single operation doesn’t hit many shards instead of a single shard.  Hitting many shards will invalidate the sharding approach.  In times like this, we will again be tempted to opt for Eventual Consistency.  We’ll get to hitting all the shards in our sweet time, and any accesses before that update is finished will just live with inconsistent results.  Such scenarios can arise where there is no obvious good sharding algorithm, or where the relationships between the objects (perhaps its some sort of real time collaborative application where people are bouncing around touching objects unpredictably) are changing much too quickly to rebalance the shards.  One really common case of an operation hitting many shards is queries.  You can’t anticipate all queries such that any of them can be processed within a single shard unless you sharply limit the expressiveness of the query tools and languages.

I hope you come away from this discussion with some new insights:

-  Inconsistency derives from having multiple copies of the data that are not all in sync.

-  We need multiple copies to scale.  This is easiest for reads.  Scaling writes is much harder.

-  We can keep copies consistent at the expense of slowing everything down to wait for consistency.  The savings in relaxing this can be quite large.

-  We can somewhat balance that expense with increasingly complex architecture.  Sharding is more efficient than replication, but gets very complex and can still break down, for example. 

-  It’s still cheaper to allow for Eventual Consistency, and in many applications, the user experience is just as good.

Big web sites realized all this long ago.  That’s why sites like Amazon have systems like SimpleDB and Dynamo that are built from the ground up with Eventual Consistency in mind.  You need to look very carefully at your application to know what’s good or bad, and also understand what the performance envelope is for the Eventual Consistency.  Here are some thoughts from the blogosphere:

Dare Obasanjo

The documentation for the PutAttributes method has the following note

Because Amazon SimpleDB makes multiple copies of your data and uses an eventual consistency update model, an immediate GetAttributes or Query request (read) immediately after a DeleteAttributes or PutAttributes request (write) might not return the updated data.

This may or may not be a problem depending on your application. It may be OK for a del.icio.us style application if it took a few minutes before your tag updates were applied to a bookmark but the same can’t be said for an application like Twitter. What would be useful for developers would be if Amazon gave some more information around the delayed propagation such as average latency during peak and off-peak hours.

Here I think Dare’s example of Twitter suffering from Eventual Consistency is interesting.  In Twitter, we follow mico-blog postings.  What would be the impact of Eventual Consistency?  Of course it depends on the exact nature of the consistency, but lets look at our replicated reader approach.  Recall that in the Eventual Consistency version, we simply tolerate that we allow reads to come in so fast that some of the replicated read servers are not up to date.  However, they are up to date with respect to a certain point in time, just not necessarily the present.  In other words, I could read at 10:00 am and get results on one server that are up to date through 10:00 am and on another results only up to date through 9:59 am.  For Twitter, depending on which server my session is connected to, my feeds may update a little behind the times.  Is that the end of the world?  For Twitter users, if they are engaged in a real time conversation, it means the person with the delayed feed may write something that looks out of sequence to the person with the up to date feed whenever the two are in a back and forth chat.  OTOH, if Twitter degraded to that mode rather than taking longer and longer to accept input or do updates, wouldn’t that be better? 

Erik Onnen

Onnen wrote a post called “Socializing Eventual Consistency” that has two important points.  First, many developers are not used to talking about Eventual Consistency.  The knee jerk reaction is that it’s bad, not the right thing, or an unnecessary compromise for anyone but a huge player like Amazon.  It’s almost like a macho thing.  Onnen lacked the right examples and vocabulary to engage his peers when it was time to decide about it.  Hopefully all the chatter about Amazon’s SimpleDB and other massively scalable sites will get more familiarity flowing around these concepts.  I hope this article also makes it easier.

His other point is that when push comes to shove, most business users will prefer availability over consistency.  I think that is a key point.  It’s also a big takeaway from the next blog:

Werner Vogels

Amazon’s CTO posted to try to make Eventual Consistency and it’s trade offs more clear for all.  He lays a lot of good theoretical groundwork that boils down to explaining that there are tradeoffs and you can’t have it all.  This is similar to the message I’ve tried to portray above.  Eventually, you have to keep multiple copies of the data to scale.  Once that happens, it becomes harder and harder to maintain consistency and still scale.  Vogels provides a full taxonomy of concepts (i.e. Monotonic Write Consistency et al) with which to think about all this and evaluate the trade offs.  He also does a good job pointing out how often even conventional RDMS’s wind up dealing with inconsistency.  Some of the best (and least obvious to many) examples include the idea that your mechanism for backups is often not fully consistent.  The right answer for many systems is to require that writes always work, but that reads are only eventually consistent.

Conclusion

I’ve covered a lot of consistency related tradeoffs involved in database systems for large web architectures.  Rest assured, that unless you are pretty unsuccessful, you will have to deal with this stuff.  Get ahead of the curve and understand for your application what the consistency requirements will be.  Do not start out being unnecessarily consistent.  That’s a premature optimization that can bite you in many ways.  Relaxing consistency as much as possible while still delivering a good user experience can lead to radically better scaling as well as making your life simpler.  Eventual Consistency is nothing to be afraid of.  Rather, it’s a key concept and tactic to be aware of.

Personally, I would seriously look into solutions like Amazon’s Simple DB while I was at it. 

Making Enterprise Software Sexier: Repeatable Process Without Endless Boredom

I couldn’t quite get past thinking about the issue of “sexy Enterprise Software” that Scoble started us talking about a few days ago.  The topic was such a Rorschach Test for what people think of when they think Enterprise Software.  That in and of itself was quite interesting.  Seeing lots of new perspectives on a thing I thought was familiar is always food for thought.  After a few days of almost unconscious cogititation, an interesting new perspective or two often starts to take root.  My initial response to it all was to suggest that what was interesting about the software was perhaps not so much the immediate UI as the potential for business transformation that Enterprise Software offers, but there were wildly differing perspectives.  Michael Krigsman and Nick Carr got into a real snit over the whole thing.  Nick Carr, who often writes that IT is becoming irrelevant, was strangely passionate about the idea that there is nothing precluding business software from being engaging, while Krigsman was determined that it’s a complete non sequitor to even ponder the sexiness of Enterprise Software because it’s all about repeating a crucial business process:

I demand the system work every time without fail.

While I disagree with much of what Krigsman had to say (including his parting shot at Carr which asks where’s the revenue: compare Google and Oracle and you’ll see the revenue is very much there), the idea that it was all about perfecting process got me thinking. 

Stowe Boyd tickled my thinking further, when he says that Enterprise Software is about Groups not Individuals:

Enterprise software starts with the premise that the user is an employee, or member of the marketing department, or a minion in the IT department. The users rights and capabilities are tied to membership, not to individual identity.

Stowe gets a lot closer to the way I am thinking about making Enterprise software sexier, but not all the way.

My new thinking is focused around the idea that a lot of Enterprise Software is about implementing Repeatable Process.  Perhaps the ultimate incarnation and priesthood for Repeatable Process is Six Sigma.  Practitioners will point out that there is a lot more to it than repeatability, in the sense that one is supposed to continually improve the process.  Any defect that is identified results in a process change that eliminates the root cause of the defect so it can’t happen again.  It can be a powerful methodology: much of the success for companies like General Electric is laid at the alter of Six Sigmas. 

This is all fine and well, but I think it can be taken too far and applied in a manner such that the endlessly repeated process results in endless boredom.  That in and of itself can be a problem, because boredom leads to mistakes and lack of attention.  Boredom costs productivity, which is not what we’re about with Enterprise Software.  There is a further danger to Six Sigmas that has started to be heard more recently.  According to Wikipedia:

A Fortune article stated that “of 58 large companies that have announced Six Sigma programs, 91 percent have trailed the S&P 500 since.” The statement is attributed to “an analysis by Charles Holland of consulting firm Qualpro (which espouses a competing quality-improvement process).”^[15] The gist of the article is that Six Sigma is effective at what it is intended to do, but that it is “narrowly designed to fix an existing process” and does not help in “coming up with new products or disruptive technologies.” Many of these claims have been argued as being in error or ill-informed.^[16][17]

When Six Sigma is used as a cost cutting program, it has been shown to stifle new product innovation.^[18]

Of course this gets the High Priesthood into a fine frothy fit, but it sure seems to fit with everything I know and have read about innovation.  Even Deming advocates something shockingly similar to the choices I’m suggesting for our Enterprise Users.  According to Wikipedia, one of his principles is Semi-Automated, not Fully Automated:

Dr. Deming lamented the problem of automation gone awry (”robots painting robots”): instead, he advocated human-assisted semi-automation, which allows people to change the semi-automated or computer-assisted processes, based on new knowledge. Compare to Japanese term ‘jidoka’ (which can be loosely translated as “automation with a human touch”).

I like this concept of semi-automation, where everyone understands the ultimate goals and destination, but there is personal freedom about how best to get there.  If Deming, the Grand Pajandrum of consistent process execution can advocate choice, why doesn’t it make sense here?

What then to do about it?  In part, I think there is a problem in that we continue to carry over a lot of the old 3270 Green Screen approach to Enterprise Software.  We view that the job of the computer is to rigidly control the people using it and that this is the only way to ensure our perfectly concieved process will be accurately carried out.  Much of it has extremely rigid workflow.  Such user interfaces are extremely modal, and frankly, they feel awful.  They force their users to act in a pre-defined play like marionettes.  There are endless “screens” that have to be dealt with, and sometimes a key piece of information is on a screen that we can’t quite remember how to get to.  The track is extremely rigid, and usually make no sense at all to a newcomer.  It might also be less than ideal for achieving the best possible results.  After all, people are not marionettes.

Jason Kolb says this about Krigsman’s posts:

…his point is that enterprise software is to be stable, above all else, sacrificing the user if need be. 

Jason is not advocating this, in fact quite the opposite, but it is another way of saying what Enterprise Software is in many cases.  I take exception to the idea that we may need to sacrifice the user in favor of the process.  People are not at their best and most productive when treated this way.  There is a tremendous amount of literature about the value of offering people choices as they go about their tasks, but the rigid process brand of enterprise software doesn’t hold with that.

An awful lot of Enterprise Software is nothing more than moving database records and fields from one form to the next while some poor user is expected to do some menial task to the form that the computer can’t quite manage for itself.  Shades of Amazon’s Mechanical Turk.

What if there is a better way?  What might it be?

There are tantalizing glimpses out there.  One of the best comes from James Governor, who made a long post about some SAP demos he has seen.  It’s worth a read.  The essence of it is enabling the ad hoc to become a part of the process.  I think of it as self-modifying process.  It is modified if and when the users of the process need to change it to do better.  Such an idea places individuals back into control, at least a little bit.  After all, part of Six Sigma involves continuously improving the process to make it better.   When was the last time your Enterprise Software was malleable enough to improve on the fly?  I thought so.  Yet here is an example that shows how to give at least some of the users choices rather than locking them into an immutable process.

Perhaps it is simply a matter of providing choice that matters to the users about what to do next.  Choice that goes beyond the green screen menu.  It blinks at you and says smugly, “Figure out how to fit what you want into my world or you’ll never get there.”  No, we want software that fades into the background.  It has choices that are natural, make more sense to people than machines, and that any reasonable human being can fit together in the way that people are great at and computers are lousy at to make anything they need to have happen get done.  One of the maddening things about “workflow” and “business process” is that it’s so synchronous.  You have to follow a task to its completion or lose everything.  It’s like those annoying web storefronts where if you make one tiny little mistake on a form late in the process, it kicks you back and makes you reenter all of the information. 

Such choices might be more like Google’s system of project management.  They’re a very textual company, so they use a system based on email.  Folks receive emails asking them what they expect to accomplish next.  They respond via email.  The system then comes back at a later date and asks whether they got the tasks done.  Reporting based on these results is available to managers.  Wouldn’t such a system be tremendously more friendly than trying to fight through Microsoft Project or some other highly stylized project management system?  Given all the complaints about Sales Force Automation (salespeople often hate filling in the data), it made me wonder why you couldn’t create a CRM system around the same principles.  Maybe that would be the ultimate killer CRM that salespeople would finally accept as being worthwhile.

I wonder when the last time a workflow designer (Business Process Architect or whatever high-faluting title they use) actually thought about giving users more choices was?  Hmmm, probably depressing to consider.

Maybe users would be happier if the tasks were much more granular and the users could bounce around as they saw fit.  How do you maintain the integrity of a process in a world like that?  Well, it’s an asynchronous world, so you have to keep track of what’s left to do and let the users decide what to do next.  The email inbox. RSS feed, and Social Network newsfeed are such paradigms.  They’re worlds where users can open as many cans of worms as they like and work on them in any order.  It recognizes that people are people.  They get interrupted.  They change their minds.  They may forget or need to go back to something.  They get tired of doing one thing and need to do another for a while as a respite.  But despite all of that, they don’t want to be led around by the nose.  They want choice.

With choice comes responsibility.  Folks are accountable for their choices.  That’s okay.  Responsibility is a good thing.  People rise to the challenge more often than not and everyone is better for it.  But we need to help people make the best choices.  Give them feedback.  Give them a way to keep score on their choices.  Some form of analytics.  To choice and analytics, let’s add another valuable spice: collaboration. Consider a case that seems completely unrelated:  in Agile programming, a lot of productivity is gained by having programmers work in pairs instead of as individuals.  Collaboration is not only fostered, it is forced.  And it turns out to be a lot of fun once you get used to it.  Does anyone do something like that with Enterprise Software?  Is there no way to pair up users and benefit?  I’ll bet there must be.

Here is another crazy thought: what if we need to turn Enterprise Software inside-out to make it sexy?  Old school Enterprise has various user categories that are largely clerks locked into precise workflows based on how a particular company likes to operate.   HR and Finance are the categories where most of this stuff lives.  Enterprise Software has succeeded in making the clerks wildly more efficient, so we need fewer of them, but there are still many many clerks out there.  How can we turn Enterprise Software inside-out and eliminate the need for clerks?  Remember (or perhaps you don’t, there are fewer and fewer who do) how it used to be before the PC was common and there was such a thing as a Knowledge Worker?  Managers all had assistants who did things like taking memos and writing letters.  Go further back and we learn that CC meant “Carbon Copy”, which was literally the process used to send a memo to more than one person.  Word Processors on PC’s turned that task inside out in a relatively short period of time and there was suddenly much less need for everyone to have an assistant.  Now an assistant could take care of a relatively large department, and he could do so with day-to-day tasks that were at a much more interesting level. 

And then I had the craziest idea yet.  What if the software was designed in a way that its users actually wanted to follow the best process?  What if they actually enjoyed it, or perhaps at the least overlooked that it was repetitive and boring for some reason?  Think about it for a minute.  Isn’t most of what goes on with Social Networks repetitive and boring if you strip it down and just look at what you can do?  All that newsfeed reading and poking and messaging, it isn’t as if this is the world’s greatest game.  Yet people seem to stay engaged.  Some would say addicted even.  What’s needed here?  An incentive would be my first thought.  A reason to move forward and do the next thing well.  A reason that is more compelling, more immediate, and more under the user’s control than just the promise of their next paycheck.  Incentives are a powerful force, and often, they need not even be monetary to be effective.  I mentioned the idea of keeping score above.  I’ve used score keeping many times to get competitive people working above and beyond the call of duty.  It’s a way of life for salespeople, and they love it.  It’s tantamount to turning work into a game, albeit with more serious consequences.

Let me give one last example that ties all of this together.  There is more choice, there is scorekeeping, there is self-service, and at the end of the day everyone benefits.  A friend recently suggested a great application for businesses that have lots of part-time help.  My brother manages a retail store, and his situation is not unique.  He has available 90 part-time hours for the store, and 5 or 6 employees to spread those hours around to.  Nobody is ever happy about it.  They all fight constantly for more hours, and this places my brother as manager into the role of the “heavy” who has to say “no” more often than “yes.”  You can just imagine some heavy-handed Enterprise Software arbitrating this fight with an obscure HR-derived algorithm that is deemed fair to all but annoys everyone and is almost opaque to understanding.

Enter this other friend’s idea.  He wants to turn the whole thing into a Dutch Auction.  People would hop onto a web application and bid what hourly rate they’re willing to take for a certain number of hours.  Lower rates get more hours.  He sees this as a win-win.  Management is no longer cast in the role as heavy.  If you an employee really wants a lot of hours, they can bid to get them.  Chances are that a slightly lower hourly rate is more than offset in gross dollars by gaining more hours.  The company was going to pay for 90 hours (in my brother’s case) anyway, so it wins by getting a lower hourly rate over those hours.  Management is happy because they’re no longer in the middle of a no-win dispute.

It’s a case of self-service, offerings choices, and loosening up a process until people can participate and imporve the situation for everyone.  By completely rethinking the problem instead of paving the cowpaths (i.e. automating existing manual processes), it’s possible to provide radically better solutions.  Enterprise Software can be sexy, but only if you think of the people more than the process.

The IT Perspective on Bloated Unfriendly Enterprise Software: More Reason to Buy SaaS

I recently responded to the free-for-all Robert Scoble started on unfriendly Enterprise Software.  Now George Ou writes about IT’s perspective, and had this to say:

• Enterprise software generally has lower usability and more bugs than commercial software.  That’s sort of counter intuitive to the word “enterprise” but the name is a joke in IT circles since enterprise software is typically painful.
• Enterprise software is designed for and sold to IT departments so the expectation is that you have trained people supporting the software whereas commercial off-the-shelf software has to more or less be self explanatory.  Enterprise isn’t sold to the end user and the end user doesn’t sign the check so their considerations are secondary to enterprise software makers.
• Enterprise software requires a lot more interaction between multiple systems which makes it fundamentally more complex to develop, deploy, and support.
• Enterprise software also typically addresses a much smaller user base than off-the-shelf software like Microsoft Office so the development budget to user ratio is smaller.  This means programming shortcuts like Java are often taken which makes the software horrendously bloated and inefficient.  You’re not going to see enterprise software developed in light-weight C++ like MS Office any time soon because that level of skill is too rare and difficult and expensive to acquire.

George sure sounds like an IT guy to me, and a card carrying member of the Old School.  That’s not necessarily a good thing, I’m afraid.  He’s written a doozy filled with misperceptions and I had to respond point by point to clear some of the fog away: 

• Enterprise software generally has lower usability and more bugs than commercial software.  That’s sort of counter intuitive to the word “enterprise” but the name is a joke in IT circles since enterprise software is typically painful.

I presume George is trying to compare Enteprise Software to packaged software like MS Office (which he refers to in another bullet), but he really doesn’t say.  Having been on both sides of the fence (I built Quattro Pro for Borland and have done several Enterprise gigs), I don’t see any reason to agree with George’s broad brush statement.  I found the recent release of MS Office 2007 to be surprisingly buggy, very unfriendly to someone that already new the prior version, and Vista is not exactly setting new standards for quality or end user satisfaction.  OTOH, Vista was designed primarily to appeal to what Old School IT cares about, so maybe that makes up for it with people like George.

• Enterprise software is designed for and sold to IT departments so the expectation is that you have trained people supporting the software whereas commercial off-the-shelf software has to more or less be self explanatory.  Enterprise isn’t sold to the end user and the end user doesn’t sign the check so their considerations are secondary to enterprise software makers.

George, you could not be more wrong in my experience.  I’ve sold multi-million dollar Enterprise Software to a variety of Global 2000 companies.  In the majority cases (nut not all), we found that the business users were completely driving the need to purchase, not IT.  In terms of training, yes there was some, but a part of every sales cycle was sitting the likely trainee down at the keyboard without said training and trying to see whether they could get comfortable enough to endorse purchasing the package.  This doesn’t even consider the raft of Enterprise Products that people use without training such as Expense Report software or HR Focal Review software.  Very few of our seats sold were occupied by trained users.   There were certainly a few guru admins that needed the training, but the majority had to be able to walk up and use the software unaided.  In the case of my last company, this included salespeople who wouldn’t sit still for a lot of training anyway.

There is no end of categories for Enterprise Software that require usage with minimal training.  I think the real problem is that IT is often very isolated from the user expderience.  Their focused on tasks like Database Administration of the newly installed Enterprise Software and often have little idea what their end users are going through except by anecdote.  This doesn’t have to be the case, and shouldn’t be the case if IT will be supporting the software, my only observation is that it often is the case in many real organizations I’ve dealt with.  Further, IT often doesn’t really understand the business needs or requirements.  I have frequently had business users come to me very concerned that IT is solely focused on their own needs (which have to do with minimizing the package’s impact on their IT organization) and wanting to make sure the Business side is heard.  When I hear IT folks like George say things like, “Enterprise isn’t sold to the end user and the end user doesn’t sign the check so their considerations are secondary to enterprise software makers,”  I really begin to feel these user’s pain.

• Enterprise software requires a lot more interaction between multiple systems which makes it fundamentally more complex to develop, deploy, and support.

This is true, but it’s an IT consideration that has little to do with end user friendliness or with whether the code is “bloatware”.  If the latter is a problem, it’s usually because IT has a bunch of poorly maintained Legacy systems that are hard to integrate with.  Whatever the case, this is very much secondary to the discussion.

• Enterprise software also typically addresses a much smaller user base than off-the-shelf software like Microsoft Office so the development budget to user ratio is smaller.  This means programming shortcuts like Java are often taken which makes the software horrendously bloated and inefficient.  You’re not going to see enterprise software developed in light-weight C++ like MS Office any time soon because that level of skill is too rare and difficult and expensive to acquire.

This is some very bad Old School thinking, and many comments on George’s column call him to task over it.  I’ve got several reactions.  First the obvious one for anyone who actually develops software:  the myth that C++ is lightweight and fast compared to Java was laid to rest a LONG time ago.  George may be technical, but this reflects a profound lack of understanding of the tools involved.  Moreover, it makes no sense to think that “programming shortcuts” lead to bloat.  If we’re building software with less budget (this is another myth I’ll get to), less people, and less time, who had the time to write the bloat? 

These interpreted languages lead to less code, not more.  In fact, it’s tempting to advocate ditching Java for many projects and moving even further towards the scripting language world.  George’s views on this raise one important commercial issue in doing so:  IT won’t necessarily understand what it means and may think less of the software if it is written in PHP or Ruby On Rails.  They’d be wrong for the most part, but it could factor into sales. 

As regards smaller development budgets, I have two reactions.  First, SAP and similar companies spend as much developing their modules as Microsoft does building MS Office.  Second, having these huge teams and spending so much is the wrong way to build great software no matter what area you’re focused on.  I’ve talked before about the virtues of small teams.  They can make a huge difference in the final software and in the overall productivity of the team.

It’s been fun to get all this off my chest.  There are some classic old chestnuts of IT thinking here, evidently still doing harm in various ways.  What this really brings home to me is that the SaaS model is fundamentally better in the face of these kinds of attitudes.  Rather than try to fight through these entrenched viewpoints, SaaS sidesteps most of it entirely.  The Business does make the decision on SaaS software in most of the cases.  They do so with minimal IT intervention.  Evaluation cycles are shorter as a result, and a lot of practical IT interests just don’t matter.  The details of the care and feeding of the software, for example, go away.  IT is often very concerned about platforms because they have to live with the care and feeding issues and their people are trained on particular platforms.  With SaaS, IT is not responsible for care and feeding, so it need not concern itself. 

Lastly, the SaaS vendor is on the hook month-by-month for keeping the customer happy.  This is probably the most important part of the whole equation.  SaaS is just a better way to do business with you software vendor.

Will MS Office Or Oracle Be Slaughtered First By The Cloud?

There’s a spate of articles about the new Live Documents service, a cloud-based SaaS challenger to Microsoft Office.  Like any good entrepreneur, CEO and founder Sabia Bhatia claims his new baby and it’s relatives will displace MS Office by 2010.  That’s right around the corner, but wildy optimistic.  ZDNet’s Dan Farber calls these cloud Office wannabes “ankle biters”, and with good reason.  They don’t come close to posing a threat to Microsoft yet.  It’s not for lack of trying.  There’s a pretty good crowd of them out there now.  Live Documents has joined a cast that includes Google, Zimbra (now owned by Yahoo), Zoho, ThinkFree, Adobe, and others.

Why can’t these guys take over by 2010?  Because every story you read is largely a man bites dog story.  There’s little to no discussion of amazing new features these products offer that would give a reason to switch.  The mere fact that products calling themselves Office equivalents can exist in the cloud without needing to be installed seems to be newsworthy enough.  There are no great roundup reviews that are getting big attention in the blogosphere that play them off against each other.  What one does find are articles telling us about the introduction of features that seem painfully elementary.  It wasn’t so long ago that the Google Spreadsheet learned how to hide columns, for example.  Even as TechCrunch writes “While Live Documents Yaps, Zoho Delivers,” Stowe Boyd writes that he can’t get Zoho to play nicely with Google Gears, even though the ability to work disconnected is the big newly announced feature for Zoho.  Apparently the Live Docs messaging annoys Michael Arrington, who writes:

New product press releases unencumbered by the complexities of releasing actual software set off alarm bells. And when those press releases are so boastful as to suggest that the (unlaunched) product can hurt a competitor’s $20 billion revenue stream, the alarm bells get much louder…

So far Live Documents is nothing more than bullshit and smokescreens. That may have been the way to do business when Bhatia co-founded Hotmail in 1996, but his software is going to have to survive on its own in a hyper competitive marketplace when it actually launches. Hubris alone won’t do it.

From my perspective, this matter-of-fact let’s paste together a bunch of things out on the web and not worry too much if they work well is a problem for an Office Killer.  The Microsoft Office represents basic literacy in the business world.  Give it up and you may find you’re unable to speak the lingua franca of others you must communicate with day to day.  Real challengers have to keep this in mind.  I was a General in the last Office Suite Wars, having fathered Quattro Pro at Borland.  We made considerable inroads (Quattro Pro sold on the order of $100 million its first year) but ultimately fell by the wayside because we lacked a word processor to go with our suite.  The one thing I can tell you is that absolute 100% compatibility with the market leader at the time together with significant innovation over that market leader and significant economic advantage (we were much cheaper at the outset) were key to the success we did achieve.  I don’t have a sense these upstarts have achieved any of these ideals.

There is a market that I think is more interesting when we look at who might become a Cloud Casualty sooner rather than later.  I’m speaking of Oracle, of course, and specifically of the database server business.  MySQL appears poised for an IPO, but beyond that there is a raft of contenders out there who have achieved a lot relative to Oracle.  There are fairly Oracle compatible products like Enterprise DB.  There are products that have serious innovations such as the column-based DB’s.  And the economic advantages are undeniable.  Unlike the Office Suite arena, it gets harder and harder to find significant killer features that Oracle offers that don’t exist in the Open Source DB world.  We’ve seen extremely large web sites powered by MySQL and some of these others, for example.  It would be hard to claim Oracle is dramatically more scalable in the face of the evidence, although one can likely conclude it remains easier to scale and more performant. 

The problem is that the costs associated with Oracle licenses are positively usary.  A friend who runs a SaaS company says his Oracle costs are bigger than his hardware costs.  I asked him why he continues to use it and he indicated his CTO was convinced he had to for scalability.  At my last company, Callidus, we ran some tests and were surprised at how performant these solutions were compared to Oracle.  I believe that currently, it’s an inertia thing.  People are sure they can get Oracle to scale, they have people on board who know how, and unless cost is a serious issue from the get go (as it can be with startups focused on ad revenues), the tried and true is chosen.  When enough people have experience scaling MySQL and its relatives, that inertia will have gone away.  Venture Capitalists tell me most of their portfolio companies are there.  As companies built on technologies like MySQL mature and people move on to other jobs, the word spreads.  Businesses running old-school on-premises software will be the last bastions for that inertia, but realistically, I’ve still talked to members from that elite group who are keeping COBOL CICS and IBM AS400 software alive. 

It won’t take an awful lot of shift before Oracle feels the pain.  The problem is that Oracle depends on this business as a cash cow to finance it’s other expansion.  Knock 20-25% off the top through erosion to these upstarts and it will dramatically change the Oracle profit picture for the worse.  Here’s another interesting strategic point to consider: utility computing ties together larger tectonic plates that can result in greater and more sudden market shifts.  Imagine companies like Amazon deciding to offer database dial-tone in their rent-a-clouds.  The database is the most labor intensive and problematic piece of software in the whole suite.  If someone automates those problems away, promises scalability on a utility computing grid, and handles normal SQL, many will rush at the opportunity.  Such aggregation of users can drive a lot of license fees away in a hurry.  They also take away a lot of the intertia issue in a couple of ways.  First, the cloud service has to deal with the operational knowledge required for server care and feeding.  Customers won’t have to.  Second, these cloud vendors are no small shakes.  Names like Amazon, Google, and Microsoft are bandied about.  The fear of dealing with some Open Source vendor that is viewed as too small and flakey is greatly ameliorated.  An additional sweetener is that the competitive strength of users of such services would be much greater versus their competitors who still use expensive Oracle and have to manage the servers themselves.

Despite interest by folks like Nick Car in LiveDocuments, breaking Oracle’s strangehold on the database world is a more likely spot for the disruption of the old school to show up first.  It would mark quite a change.

Multitenancy Can Have a 16:1 Cost Advantage Over Single-Tenant

Multitenancy is one of those things that has been more qualitative than quantitative.  True blue SaaS believers put it down as a must-have to even call an offering SaaS.  Those less devout are understandably somewhat skeptical.  The heretics will say that liberal use of modern virtualization technologies is good enough.  Everyone seems to agree that the purpose behind multitenancy is to lower costs for the SaaS vendor.  These savings are translated into lower costs to their customers and a better shot at profitability.

In this article, I wanted to try to quantify those savings as well as put forward a definition for multitenancy that I think is clearer than a lot of what I’ve seen out there.  You’ve got the short answer in the title of this post, but I’ll walk you through how I got to the 16:1 value. 

First, let’s try to get to a useful definition of multitenancy.  Wikipedia defines multitenancy as follows:

Multitenancy refers to the architectural principle, where a single instance of the software runs on a software-as-a-service (SaaS) vendor’s servers, serving multiple client organizations (tenants). Multitenancy is contrasted with a multi-instance architecture where separate software instances (or hardware systems) are set up for different client organizations. With a multitenant architecture, a software application is designed to virtually partition its data and configuration so that each client organization works with a customized virtual application instance.

This definition is fine up to the point where the word “partition” comes up with a link back to a discussion of LPARs which are defined as follows:

In computing, a logical partition, commonly called an LPAR, is a subset of computer’s hardware resources, virtualized as a separate computer. In effect, a physical machine can be partitioned into multiple LPARs, each housing a separate operating system.

There is a lot of confusion here that comes from attaching a particular notion of how Multitenancy might be implemented (LPARs) with the concept itself.  I found the same thing reading SaaS-Man’s “Myths of Multitenancy” where the “myths” are based on particular implementation assumptions.  Let’s get one thing absolutely crystal clear before going further: there is no single implementation or architectural design pattern that can be called multitenancy.  Rather, multitenancy is an abstract concept largely rooted in benefits more than features.

To get away from implementation specific definitions (which are more examples than definitions), I am defining multitenancy as software that has the following properties from the perspective of three audiences seeking benefits:

For the SaaS Vendor:  Multitenancy is the ability to run multiple customers on a single software instance installed on multiple servers.  For the vendor,  operations can be performed at the level of instance, tenants, and users within a tenant.  This is done to increase resource utilization by allowing load balancing among tenants, and to reduce operational complexity and cost in managing the software to deliver the service.  For example, a patch can be easily rolled out to all tenants by patching a single instance instead of many.  Everyone’s data can be backed up in one operation by backing up a single instance.  The operations costs are then lower due to economies of scale and increased opportunities for automation.

For the SaaS Customer (a Tenant):  Multitenancy is transparent.  The customer seems to have an instance of the software entirely to themselves.  Most importantly, the customer’s data is secure relative to other customer’s data and customization can be employed to the degree the application supports it without regard to what other tenants are doing.  The only tangible manifestations of multitenancy are lower costs for the service, and better service levels because its easier for the service provider to deliver those levels.  The tenant will also want the ability to manage the system from their perspective (for example to manage security access among their users) and will want that to be as seamless as possible and certainly to never be impacted by other tenants.

For the SaaS User (one seat on the Tenant’s account):  Multitenancy is transparent.  They just see the application as a user of any application would. 

Further definiton around multitenancy gets into the realm of being too specific about implementations.  For example, there are folks who say multitenancy means an application won’t be highly customizable because it’s too hard to build.  It makes no sense to generalize about the properties of multitenancy on such a basis, so it should be left aside as something that relates to specific examples of multitenancy.  There are implementations of customizability that are incompatible with multitenancy, but I know it is entirely possible to build multitenant systems that are highly customizable if the proper metadata approaches and other customization tools are provided.  Eventually, I will write a post about how this can be done.

The details of a particular implementation of multitenancy can vary greatly from one vendor to the next.  It makes sense the details should vary just as Enterprise apps vary a lot from one domain to the next.  For example, most web software vendors like Twitter probably don’t think of themselves as Multitenant architectures, but they are with the degenerate case that a User and a Tenant are one and the same.

I bring up the odd example of Twitter because it is the start of one dimension that matters a lot for implementation choices:  number of users per tenant.  A single user per tenant is a nice fine grained unit.  In many ways it may be the easiest to create because load balancing and partitioning are so much simpler, although I don’t want to say they are simple by any means.  This is a meaningful class of SaaS, BTW, because most desktop productivity applications can be modeled in this way, so it’s relevant to Google’s view of what SaaS is, for example.

Next up would be to have a fairly small number of users per tenant.  Salesforce.com had an average of 21 seats per tenant when I recently did the calculation.  That’s very small.  If you are sure you’ll never have many more than that based on the nature of your application, you can largely treat this the same way as one per tenant with a few tools to make it easy to manage small groups of users as a unit.  Of course, Salesforce can’t do that, because they also have much larger customers than just 21 seats!

Let’s jump over to the opposite end of the scale, because it is an interesting perspective too.  Suppose the application will average more like 1,000 or even more users per tenant.  Suddenly the server utilization and automation arguments are a little bit less valuable.  We have 50x larger tenants than in the Salesforce example.  We have 50x fewer customers to reach the same level of seats.  Whatever repetitive tasks we may perform for every tenant have to be done 50x less often even if we do them manually on a tenant-by-tenant basis.  Is this an argument not to go multitenant for such applications?  Possibly, because multitenancy offers a lot less benefit to such a domain.  One could envision that virtualization and a lot of automated scripting could get us nearly all the way there, at least until we had so many of these large customers that it made sense to do something more sophisticated.  Interestingly, several SaaS vendors I talked to have mentioned that the management overhead for a customer is nearly the same almost regardless of size.  There are some operations that are size dependent, but a properly implemented SaaS app pushes as much of that sort of thing as possible back to the customer as self-service (for example, to manage accounts and passwords).

Okay, we have a working definition that’s very simple:

Multitenancy is the ability to run multiple customers on a single software instance installed on multiple servers to increase resource utilization by allowing load balancing among tenants, and to reduce operational complexity and cost in managing the software to deliver the service.  Tenants on a multitenant system can operate as though they have an instance of the software entirely to themselves which is completely secure and insulated from any impact by other tenants.

Let’s turn now to the question of estimating the cost benefits of a well constructed multitenant architecture.  I’ve collected statistics on the relative cost to provide service for a number of public SaaS offerings for which such numbers are available:

 

We can see a mix of business SaaS like Salesforce.com as well as web software companies like Google.  The first takeaway is that costs can vary quite a lot from one company to the next.  This is a function of pricing (selling too cheaply relative to the costs of delivery raises the %) and of internal operating efficiencies.  It’s hard to draw much conclusion, so let’s just work from the average of 26%.  An average web software company can deliver $1 of revenue for about 25 cents spent on hosting and managing their software for customers.

Lest you be thinking that all of the Cost of Service is hardware, let’s consider the case of Google.  Gartner reports that runs on the order of 1 million commodity class machines at this time.  Even if we assume they’re replaced annually, which they aren’t, that places cost per machine at $4,225 and the same report says they spend about $1,800 on a machine, storage, and all hardware.  The rest of the cost is for network connectivity, datacenter physical space, and personnel to run all those little boxes.

I want to emphasize that we absolutely can’t be too cavalier about even the 1,000 seat per tenant case when it comes to operational efficiency.  At $50/seat, that customer is paying $600K of revenue per year.  If the SaaS vendor wants to spend 25% to deliver the service, they can afford to pay no more than $150K to keep the lights on for that large customer.  Much of that will have to go for the hosting (hardware, network charges, and datacenter).  What’s left is a fraction of an IT person for operations on this tenant.  Even a vendor with such gross numbers of seats per tenant must therefore be able to deliver the service very efficiently.  Considerable automation of manual tasks will be necessary.

Now let’s consider the costs of running traditional Enterprise on-premises software.  Talking to SaaS vendors about how they came by their pricing, there is a rule of thumb that says the annual contract cost of a SaaS offering is roughly equal to the perpetual license most vendors sell.  Oracle’s Timothy Chou says the industry averages are that it costs 4x the license per year to run traditional enterprise software.  Can you see how I got to a 16x operational efficiency advantage for SaaS over conventional enterprise on-premises?  Take Chou’s 4x and divide by 26% and you’re there.

I walk away from this analysis with two big takeaways.

First, SaaS vendors need to manage by the numbers pretty tightly when it comes to the cost of service delivery.  They’re very unlikely to hit the 26% average on their first release, and I’ve been told this by a number of young SaaS companies and their venture capitalists.  That means being on a continuous program of improvement.  If it was me, I’d want a dashboard of live metrics around these costs that really kept it top of mind together with trending so I could be sure costs were moving in the right direction.  Achieving a 16x advantage over conventional thinking is not a seat of the pants project.  It’s going to take some real effort and some real smarts to get there.   

Second, this advantage is an interesting concept for SaaS customers to contemplate.  Delivering a service efficiently means not screwing it up.  Escalations and problems will drive such costs through the roof.  The SaaS vendor really only has the choice of making the customer happy, and doing so in a highly automated way that ensures the happiness is no accident.  Isn’t this really the lesson Japanese car manufacturers used to become so successful?  Focusing on quality actually lowers costs.  SaaS vendors live and breath exactly this because they’re financially incented to do so.  On-premises vendors, by contrast, get paid a lump sum up front and don’t have this operational efficiency monkey on their backs.  The customer bears that cost, and apparently, it is a cost that’s 16x greater.

Is it any wonder SaaS vendors see multitenancy as their Holy Grail?

What is SaaS? Can it Exist Without the On-Premises Comparison?

I had two different conversations recently where the people I was talking to were reluctant to use the term “SaaS”.  To them it equated to something a non-SaaS ISV was trying to get to, but if you didn’t have that non-SaaS Enterprise ISV background, you thought more in terms of calling it “Web 2.0″ or just “web software.”  Interestingly, they were kind of uncomfortable with the whole “Service” component of SaaS as well.  We’ve all heard the dramatic artifice that says there is no Good without Evil, because Good is only a relative term and it needs an opposite to give it meaning.  These folks seemed to imply SaaS was the same.

On a somewhat related note, a lot of folks who will answer to the name “SaaS” actual prefer “On-demand.”  I can’t quite get to the bottom of why, but the feeling I get is that SaaS is a little too caught up in Marc Benioff and Salesforce.com, or maybe its more a question of which term they grew up with.  Note that the “service” piece is once again missing from On-demand.

After more back and forth, I was able to boil out two issues that relate to this. 

First let’s talk about Service.  While a SaaS (or On-demand or web software) provider wants to deliver to the customer the benefits of service, at the same time they want to relentlessly eliminate the cost of service delivery.  They do this through economies of scale, but more powerfully, they do it by writing software that eliminates the human component of the service.  This is the prime function of multitenancy.  It’s possible to manage a bunch of instances by hand, but it would be very expensive to do so.  The first step is to write a bunch of scripts and perhaps use virtualization to reduce those costs.  That still is not cheap.  The next step is to actually build the necessary automation into the fabric of the application itself.  This dramatically lowers the cost of the service by replacing manual activity with software.

The result is that companies in the SaaS business often do not think of themselves as Service companies.  They’re actively trying to eliminate service by replacing it with software so that they can scale more efficiently and pass the savings along to customers.  These folks want to be known as software companies and not just companies that added service to software.  It’s an interesting nuance.

The second issue is simpler, and has to do with the customer.  Companies selling B2B are much more likely to consider themselves as SaaS, while companies selling to consumers think of themselves as having web software.

This business of putting computing in the cloud has a ways to go before we’re all on the same page about exactly how to talk about it.  The good news is I’ve talked to many companies using the model and they’re nearly all doing quite well without having to spend a fortune on marketing.  There’s a tremendous appetite among potential customers and everyone agrees it’s gotten a lot easier to sell over time.

Wookey Leaves, Where Does That Leave Oracle Fusion?

John Wookey, who headed up Oracle’s Fusion project, has moved on.  I have good news and bad news on that for Fusion fans.  The good news is that I doubt it changes much with respect to Fusion.  The bad news is it is an admission that Fusion is badly off track.  I’ve been talking with various Oracle folks at various levels for some time, and this news has been obvious for quite a while. 

It takes a very long time to rearchitect enterprise software.  SAP’s ByDesign was in the labs for five or six years, it is only recently available to “early customers“, and the target profile it is suited to is businesses smaller than SAP’s current power alley due to various limitations such as 4x slower performance.  Fusion has a slightly easier job as it “only” amounts to an SOA integration of all the components in their suite.  The rewrites, at least, need not be all the way to core.  Nevertheless, they are chasing a moving target in the form of an ongoing acquisition strategy.  They’re also having to retrofit a variety of different architectures with this capability, and they have to do that in the wake of having scaled back the various acquired entities to enhance profitability.  Can you guess what the pecking order is between financial results, customer satisfaction, and a new technical SOA architecture?  If you’re like me, you’ll see that Fusion is likely dead last in that pecking order.  What this means is that Wookey may have moved on from a nearly impossible job.

As Vinnie Marchandani put it when he broke the story, Fusion in 2008 was always a moonshot.  He says 2009 earliest, and I would not be surprised to see 2010.  Oracle has a difficult set of numbers to keep going with, ongoing merger digestion, and frankly, great difficulty investing and mobilizing around software innovations of the kind Fusion represents.  They traded in their innovation hats when they brought in a couple of M&A experts in the form of Chuck Phillipps from Morgan Stanley and Safra Catz to run the place under Larry Ellison.

Anyone who has worked at Oracle, as I have, will know that projects often proceed with considerable investment all the way to beta test and then get shunted into a new direction or cancelled entirely.  I’m not suggesting that Fusion will be cancelled, rather, I’m suggesting there is a lot going on below the water level on the Oracle iceberg that’s completely invisible.  Folks I talked to suggest that Fusion is isolated, at least until pretty recently, and that part of the problem is all the other products are moving on without it.  If that’s true, and it’s not out of character for the culture, Oracle can demo Fusion all they like, but the lion’s share of the work will be below the surface getting these apps properly configured to talk to it.  That’s a huge undertaking.  To make matters worse, they’re targeting BEA now, which is theoretically a competitor to Fusion.  Rationalizing that against work already done on Fusion is apt to set the schedule back further and could even prompt a restart.  And, all that effort can’t start until BEA is acquired, a fate they seem to be resisting, at least for the moment.

This is not the best news for Oracle.  SAP has their new toy out on the table.  SaaS and Open Source are coming up from below.  If the mission continues to be growth through acquisition, as it seems it must, Oracle will have to keep stepping up the pace there too.  Meanwhile customers are going to see some reasons to wonder about buying anything new from Oracle.  They’ll likely pay their maintenance and finish ongoing projects, but why add modules with such an uncertain future?  At the least they should look out in the market and wonder whether a better module from a competitor today won’t wind up in Oracle’s stable tomorrow.

If the financial markets see a hint that Oracle’s installed base and businesses are cooling, it will not be a happy time for Oracle shareholders or management.  Fortunately, that has yet to happen.  Still, it makes me nervous when a software company ceases to be a software company and becomes an M&A aggregator.  Those games eventually come to an end.  We’ve seen it before several times in the industry with players like CA.