Is Twitter Not Multitenant or What?

So here I am, 5 days after the big announcement, and still no new Twitter UI.  We just finished a weekend, which would seem to me like a logical time to roll it out to the remainder of the audience.  No joy.  WTF, over?

Is Twitter not multitenant, or what?  I sure they look with disdain at hearing that term usually reserved for business SaaS software, but I mean really, what’s up with this, guys?  Did you not get the memo about keeping all users on the same code line?  Did we forget to Tweet that somewhere along the line so you never read it?  Is your architecture so fragile that you can’t afford to let everyone have the new UI at once?  WTF is up with this?

I don’t know what’s up with me on this.  I mean, some guys who “get it” are all “Meh” about it, while others think it’s a whole friggin’ platform.  Basically, Twitter needs to be a richer medium for me to like it better, so it sounds cool to me and I want to see it. .

OK, I can see the advisability of not rolling it out to everyone in one fell swoop.  I’m trying to calm down, and sure, I’ve written about release feathering myself.  That’s kewl and all, but how long is this going to take and why isn’t there more transparency into when I as a user can expect to get the UI?  You know, like there must be an algorithm or some such.  If my handle starts with an “A” I get to go first, unless I’m a Tech Crunch reporter with the handle “Zelda” in which case I get to go first too.  Hey, at least I could figure out what to expect.  Or maybe you could even make my expected upgrade date accessible to me in the UI somewhere.

Here is the thing.  If you are going to make the biggest update ever.  If you’re going to have PR about it and all.  You’ve got to have more transparency and a shorter release feathering cycle so people know what to expect and can get access sooner.  I mean Google is much bigger and managed to get the Priority Inbox to me a lot sooner.  After all, you’re pissing off guys like Anil giving it to his wife before him and all.  Anil is right by the way in that post about it being a platform that doesn’t act like Switzerland (if you want to call that pane a platform).  But Twitter has been acting more and more walled garden-ish all the time.

Do you know what I mean?

Android is so Open, it got Flash Back on the iPhone

How ironic.  On the same day that MG Seigler was penning one of his characteristically snarky posts (snark is one of the ways Techcrunch pursues its Follower Economy) about how Android isn’t really open, Apple announces the return of Flash to the iWorld.

Adobe’s stock price is cooking this morning as a result (I wonder if 10% of Adobe’s revenue is even traceable to Mobile Flash? Maybe), and I have to admit, being the Adobe Flex fanboy that I am, I’m chortling over my morning cafe mocha too!

A couple of things to talk about on this news.

First, why do I award credit for this to Android?  Peeps, come on–Steve Jobs didn’t just wake up from his hissy fit about Flash (thanks for that link, Larry!) and decide, “Oh darn, I was wrong.”  No, no, no, not happening.  Something grabbed Steve Jobs by the throat and shook him right out of his reality distortion field.  That something was competition.  Not only has Android been selling extremely well, but the Bastiches have even been using Apple’s, “Hey, we’re not the Man, we’re Cool“, marketing tactics.  Dude, what’s next, Android thinking differently (I can’t see Google having the populist moxie to get that grammar wrong the way Apple did, all those PhD nerds just couldn’t handle it).

Competition is what forced this change, MG Seigler.  Android’s “We’re more open than open can be” pitch was working.  In fact, this Apple announcement sweeps away most of what one would’ve argued were barriers to openness.  It isn’t just Flash that benefits.  Besides which, a heck of a lot of people love their iDevices (3 out of 4 of our whole family have iPhones and we share an iPad) but want Flash access.  Forget movies, they can change.  I can’t even see the stock graphs on Yahoo because they’re Flash.

Gordon Gecko was wrong when he said, “Greed is good.”  It pains me as a person often described as being slightly to the right of Attila the Hun to reach that realization.  What Gordon should have said was this:

Competition, for lack of a better word, is good. Competition is right, Competition works. Competition clarifies, cuts through, and captures the essence of the evolutionary spirit. Competition, in all of its forms; Competition for life, for money, for love, knowledge has marked the upward surge of mankind.

And here, it has let Flash back into the iWorld.  Good job, Android!

Second, I want to take a few lines to talk about why I think Flash is so important.  Much has been made of how the LAMP stack has transformed web development.  Companies can now create products without requiring millions of dollars in R&D expense.  But the LAMP stack primarily benefit the back-end, in other words the server-side.  What about the client?  What is the equivalent of a LAMP stack for client development?  Unless you’re a tremendous fan of 3270 green screen UI, and I know some are even in this day and age, you need the equivalent of the LAMP stack to efficiently product great clients.  Here is the secret:  Flash is the LAMP stack of UI development!  Yes, there are some Flash wannabes out there, specifically Silverlight.  So what?  Microsoft would love for .NET to be the LAMP alternative too.  Flash is it for these simple reasons:

-  It is ubiquitious.  Aside from the iWorld, and that changes today, the vast majority of the world has already installed Flash and it runs in their browsers.  That’s 99.3% penetration in mature markets.  Essentially, everyone has it.

-  It is a write once run anywhere tool that really works.  You write your Flash code and all 99.3% of users can run it without you needing to give it another thought. 

-  AJAX and Javascript, the alternative, is riddled with browser dependencies.  Any dev team that has tried to get even simple rich UI like a Wiki-style rich text editor to work understands this.  It’s a huge overhead to keep up with all of these browser dependencies, and a constantly moving target.  With Flash, Adobe does that work for you so you can get on with building your app.

-  Flash has amazing graphical power and Job’s protestations to the contrary is actually quite fast.  People are writing 3D games in it, for example.

-  Flash can transcend the browser to create apps via AIR, and as we’ve discovered in the mobile world, apps are where it’s at.  Google used to offer Gears for this purpose, but pulled the plug on it.  Silverlight?  Hey, competition is good, bring it.  But they aren’t really there yet.

This all adds up to something that’s very important to the ongoing evolution of User Experience across all devices.  Simply put, Flash deserves to keep going and to be available on all those devices.  Thank you, Steve Jobs, for relenting, even if it took a gun to your head.  Adobe: now is not the time to rest on your Laurels.  There are some issues here and there in the House of Flash that you still need to attend to.

Once You Have Bad Programmers, You’re Doomed!

I love that line from Paul Graham’s post about what went wrong with Yahoo:

In technology, once you have bad programmers, you’re doomed. I can’t think of an instance where a company has sunk into technical mediocrity and recovered. Good programmers want to work with other good programmers. So once the quality of programmers at your company starts to drop, you enter a death spiral from which there is no recovery.

That observation is so true, as is this one:

Theirs was not to reason why; theirs was to build what product managers spec’d.

That’s in reference to Yahoo’s totally Product Managment-driven decision process.  It has seemed to me at times like Microsoft errs dangerously on this side too.  Product Managers are essential, and can be great, but if you place in a role where their job is to hand down stone tablets for the developers to slavishly implement, it doesn’t work.  It is not empowering to the developers who have tons of great ideas, insights, and vision themselves.  But ultimately, it’s just not a good idea to have guys in Ivory Towers thinking great thoughts they have no responsibility or accountability to deliver on.  It’s certainly not much of a team effort when things work that way.  I see great Product Managers as being the ultimate Customer Advocates.  Who else can say that they spend 100% of their time understanding the company’s customers and translating that into product requirements?  The trick is to balance those requirements against other sources of requirements, for example the need to innovate and differentiate, which far more often comes with vision rather than customer feedback.  It takes both as even the brilliant Steve Jobs discovers when his visions get too much at odds with what customers want.

The final great point from the post for me was:

In the software business, you can’t afford not to have a hacker-centric culture.  So which companies need to have a hacker-centric culture? Which companies are “in the software business” in this respect? As Yahoo discovered, the area covered by this rule is bigger than most people realize. The answer is: any company that needs to have good software.

Hacker culture often seems kind of irresponsible. That’s why people proposing to destroy it use phrases like “adult supervision.” That was the phrase they used at Yahoo. But there are worse things than seeming irresponsible. Losing, for example.

Yes indeed, there are worse things than seeming irresponsible.  Though I think the appearance of irresponsibility is only an appearance and comes from suits not being able to understand the hackers.

So let’s say you’re running a company that has Bad Programmers.  Are you really doomed? 

It’s an interesting problem to think about fixing it.  How did you get there to start with?  Here are some thoughts:

-  Like Yahoo, you had a culture that didn’t value Technology.  Great programmers can smell that a mile away and will general avoid it.  I have seen cases where most of a company didn’t value the developers much, but there was an elite core group where great developers could flourish.  It’s often hard for overly sales-driven cultures to value developers as much as they should.  As one friend put it, sales-driven cultures see the Customer as God and Sales as the Church.  There were certainly elements of this at work for Yahoo.

-  You outsourced or isolated your Developers.  This is not a guarantee you have Bad Programmers, but it is a virtual guarantee they’re not really plugged into your corporate culture the way they should be.  Developers can smell this too.  Sometimes they like being off by themselves, but its too easy for it to turn into a Stone Tablet scenario at which point they’re unhappy.  The opposite extreme is the Dev Group that has no customer contact and is just implementing whatever they feel like.  Also bad!

-  The fish rotted from the head.  Bad Leadership is a problem for Developers.  They have little patience for a lot of politics or a weak leader that’s easily snowed.  Dilbert is not a recipe for a great Hacker Culture.

-  Lazy hiring.  You started out with some great developers, but you did a poor job hiring and building an organization.

-  Lousy process.  Smart people abound, but the processes in use are not productive, or worse are destructive.  This is usually a function of poor change control.  If left to fester, you can get spaghetti code from the brightest of teams.

So back to the question of fixing it, and while we’re at it, let’s also look at avoiding it in the first place.  Consider these proactive steps:

-  Make sure your corporate culture values what Developers do.  They don’t call them Software Companies for nothing.  They’re not Sales Companies, Marketing Companies, or Finance Companies.  Make sure you act like it, while also recognizing the huge value these other groups bring. 

-  Keep the Developers well plugged into your corporate culture.  If they’re remote, work overtime looking for ways to make sure they’re still plugged in.  Rotate them through corporate.  Send the right people out to the remote locations frequently.  Never let it turn into a case of micromanagement from afar (Stone Tablets) or no management at all.  Make sure there is strong leadership in the remote locations and make sure the leaders there most of all are plugged back into corporate.  Let me tell you, this will be a lot of work.  You outsourced and offshored to save money.  Welcome to the first of many hidden costs that will make it seem less attractive, though not fatally so.

-  Get great Engineering Leadership.  You need leaders who are inspired, visionary, extroverted, technical, and downright fun to hang around.  They need to be great not just with the developers, but with their peers in the other functional areas.  The leadership, more than anything, has to move the cultural values both ways across the blood brain barrier that often separates Developers from the rest of the culture.  They’d better speak both languages (Geek and Business) to do that well.

-  Realize that Hiring is critical from day one, and it never stops.  I could write many posts on hiring, but I will leave it to one simple observation.   99% of the time people seem to regard hiring as a temporary distraction from what’s really important, such as delivering a product.  But hiring the wrong people will cause you to deal with more kinds of Hell for longer than any architectural mistake you can possibly make.

-  Keep tuning the process.   Start out with the obvious Best Practices, but don’t rest on your laurels from there.  Development organizations have two responsibilities: delivering Great Products and building a superior Software Factory.  They spend most of their time focused on the Products, but like Hiring, the Software Factory is not to be overlooked.  Any given product release is just a battle in the war.  If your Software Factory releases faster than the competition, if it builds better product every cycle, and if it does this more cheaply than the competition, it’s a huge advantage.

One last piece of advice.  If you already have a lot of Bad Programmers runnning around, find a way to isolate them.  Don’t just beg Good Programmers to come in and drop them piecemeal into a pit of Bad Programmers.  They won’t be effective and they won’t hang around.  Start out making sure you have great leadership, and build new groups with Good Programmers.  Preferably do this product by product, and through acquisition of Teams that are known good entities who know how to work together to accomplish great thinks.  Don’t hire Good Programmers and term them into Bad Programmers inadvertently, or drive them away altogether.

Is Your Child a Computer Prodigy?

I recently was asked (along with others), “What language a budding computer scientist should try to study in school?

Fundamentally, it’s the wrong question.  This will sound harsh, but alas, it is only reality.  Or, if you like quotes:

“I never did give anybody hell. I just told the truth and they thought it was hell.”
Harry S. Truman

The language doesn’t matter, and no self-respecting computer science curriculum should be letting you choose languages.  They may expose you to a variety, but you’re not there to learn computer languages.

Real developers are born and not made, no matter what the quality of the curriculum.  You can’t teach it, you can only help it to blossom.  If you’re one of these people, you’ll learn many languages quickly as you become interested in whatever requires a particular language.  If you’re not, you can still have a fine career in IT or Prof Svcs.  Most of the people who are really going to get it were programming before they set foot in a University.  In fact, a really good one will be tempted to skip school if you’re not careful.  Computers call to them with a Siren’s Song that cannot be ignored. 

Find a person like this a quality program, preferably one that is under the Math Science moreso than Engineering auspices.  The abstract and theoretical sides will be more nurtured there and it’s harder to pick them up by osmosis.

If you wonder whether your child has this talent, Python is a great place to start.  Get them a book called Python Programming for the Absolute Beginner and see what they can do.  If you’re one of these people, don’t push your child to it.  Watch from afar.  Be interested, and responsive, but don’t force it, and whatever you do, don’t make it a competition. 

Like all careers, you have to love it to be really good at it.

Tim Bray’s Schtick: He Likes 3270 Green Screens as UI

So I’m reading Bray’s blog as usual, and I come across his argument against Flash that I see occasionally–namely, that all Flash UI sucks.

Why?  Here are his words:

What’s not to like, then? Well, the user experience, which in my experience is fourth-rate for anything but games; No “Back” button, feaugh. And of the course the fact that it remains essentially proprietary.

So, I use a Flash-blocker every day, and I am not a friend of Flash inside Google, but none of my arguments have anything to do with being part of the Web, or not.

The closest thing to an explanation is that there is no “back” button.  Hmmm, is that really the issue?  Needing to understand more, I did a search for “Tim Bray RIA”.   Sure enough, I find this article which says:

One of his points is that Rich Internet Applications aren’t worth the hype. He says that web applications are generally better than desktop applications, because they enforce simplicity and support a back button, and that users prefer them.

WTF?  Really?  All this for a frickin’ back button?

When you get down to it, Web UI sans AJAX, Flash, or some other RIA (Rich Internet Application) engine is just like the 3270 green screens of old.  It’s fill in the form and press the “Enter” button (BTW, that’s why it’s called the “Enter” button).  Yeah sure its simple.  Darned right, it doesn’t get much simpler.  And for many things, it’s just exactly the right thing.  But heck, if its the only way to do everything we’re really selling the web short.  It’s the old when you have a hammer everything is a nail.  I just can’t get my head around the idea that “web applications are generally better than desktop applications” if for no other reason than there’s so many types of user interaction that just don’t make sense for a 3270 Green Screen.

Maybe its a Learning Style thing, I dunno, but it’s not for me.  Give me a simple UI where simple makes sense and a Rich UI where that makes sense, which is a lot more places than just games.

Minimizing the Cost of SaaS Operations

SaaS software is much more dependent on being run by the numbers than conventional on-premises software because the expenses are front loaded and the costs are back loaded.  SAP learned this the hard way with its Business By Design product, for example.  If you run the numbers, there is a high degree of correlation between low-cost of delivering the service and high growth rates among public SaaS companies.  It isn’t hard to understand–every dollar spent delivering the service is a dollar that can’t be spent to find new customers or improve the service.

So how do you lower your cost to deliver a SaaS service? 

At my last gig, Helpstream, we got our cost down to 5 cents per seat per year.  I’ve talked to a lot of SaaS folks and nobody I’ve yet met got even close.  In fact, they largely don’t believe me when I tell them what the figures were.  The camp that is willing to believe immediately wants to know how we did it.  That’s the subject of this “Learnings” blog post.  The formula is relatively complex, so I’ll break it down section by section, and I’ll apologize up front for the long post.

Attitude Matters:  Be Obsessed with Lowering Cost of Service

You get what you expect and inspect.  Never a truer thing said than in this case.  It was a deep-seated part of the Helpstream culture and strategy that Cost of Service had to be incredibly low.  So low that we could exist on an advertising model if we had to.  While we never did, a lot was invested in the critical up front time when it mattered to get the job done.  Does your organization have the religion about cutting service cost, or are there 5 or 6 other things that you consider more important?

Go Multi-tenant, and Probably go Coarse-grained Multi-tenant

Are you betting you can do SaaS well enough with a bunch of virtual machines, or did you build a multi-tenant architecture?  I’m skeptical about your chances if you are in the former camp unless your customers are very very big.  Even so, the peculiar requirements of very big customers (they will insist on doing things their way and you will cave) will drive your costs up.

Multi-tenancy lets you amortize a lot of costs so that they’re paid once and benefit a lot of customers.  It helps smooth loads so that as one customer has a peak load others probably don’t.  It clears the way to massive operations automation which is much harder in a virtual machine scenario.

Multi-tenancy comes in a lot of flavors.  For this discussion, let’s consider fine-grained versus coarse-grained.  Fine grain is the Salesforce model.  You put all the customers together in each table and use a field to extract them out again.  Lots of folks love that model, even to a religious degree that decrees only this model is true multi-tenancy.  I don’t agree.  Fine grained is less efficient.  Whoa!  Sacrilege!  But true, because you’re constantly doing the work of separating one tenant’s records from another.  Even if developers are protected from worrying about it by clever layering of code, it can’t help but require more machine resources to constantly sift records.

Coarse-grained means every customer gets their own database, but these many databases are all on the same instance of the database server.  This is the model we used at Helpstream.  It turns out that a relatively vanilla MySQL architecture can support thousands of tenants per server.  That’s plenty!  Moreover, it requires less machine resources and it scales better.  A thread associated with a tenant gets access to the one database right up front and can quit worrying about the other customers right then.  A server knows that the demands on a table only come from one customer and it can allocate cpus table by table.  Good stuff, relatively easy to build, and very efficient.

The one down side of coarse grain I have discovered is that its hard to analyze all the data across customers because it’s all in separate tables.  Perhaps the answer is a data warehouse constructed especially for the purpose of such analysis that’s fed from the individual tenant schemas.

Go Cloud and Get Out of the Datacenter Business

Helpstream ran in the Amazon Cloud using EC2, EBS, and S3.  We had help from OpSource because you can’t run mail servers in the Amazon Cloud–the IP’s are already largely black listed due to spammers using Amazon.  Hey, spammers want a low-cost of ops too!

Being able to spin up new servers and storage incrementally, nearly instantly (usually way less than 10 minutes for us to create a  new multi-tenant “pod”), and completely from a set of API’s radically cuts costs.  Knowing Amazon is dealing with a lot of the basics like the network infrastructure and replicating storage to multiple physical locations saves costs.  Not having to crawl around cages, unpack servers, or replace things that go bad is priceless. 

Don’t mess around.  Unless your application requires some very special hardware configuration that is unavailable from any Cloud, get out of the data center business.  This is especially true for small startups who can’t afford things like redundant data centers in multiple locations.  Unfortunately, it is a hard to impossible transition for large SaaS vendors that are already thoroughly embedded in their Ops infrastructure.  Larry Dignan wrote a great post capturing how Helpstream managed the transition to Amazon.

Build a Metadata-Driven Architecture

I failed to include this one in my first go-round because I took it for granted people build Metadata-driven architectures when they build Multi-tenancy.  But that’s only partially true, and a metadata-driven architecture is a very important thing to do.

Metadata literally means data about data.  For much of the Enterprise Software world, data is controlled by code, not data.  Want some custom fields?  Somebody has to go write some custom code to create and access the fields.  Want to change the look and feel of a page?  Go modify the HTML or AJAX directly.

Having all that custom code is anathema, because it can break, it has to be maintained, its brittle and inflexible, and it is expensive to create.  At Helpstream, we were metadata happy, and proud of it.  You could get on the web site and provision a new workspace in less than a minute–it was completely automated.  Upgrades for all customers were automated.  A tremendous amount of customization was available through configuration of our business rules platform.  Metadata gives your operations automation a potent place to tie in as well.

Open Source Only:  No License Fees!

I know of SaaS businesses that say over half their operating costs are Oracle licenses.  That stuff is expensive.  Not for us.  Helpstream had not a single license fee to pay anywhere.  Java, MySQL, Lucene, and a host of other components were there to do the job.

This mentality extends to using commodity hardware and Linux versus some fancy box and an OS that costs money too.  See for example Salesforce’s switch.

Automate Operations to Death

Whatever your Operations personnel do, let’s hope it is largely automating and not firefighting.  Target key areas of operational flexibility up front.  For us, this was system monitoring, upgrades, new workspace provisioning, and the flexibility to migrate workspaces (our name for a single tenant) to different pods (multi-tenant instances). 

Every time there is a fire to be fought, you have to ask several questions and potentially do more automation:

1.  Did the customer discover the problem and bring it to our attention?  If so, you need more monitoring.  You should always know before your customer does.

2.  Did you know immediately what the problem was, or did you have to do a lot of digging to diagnose?  If you had to do digging, you need to pump up your logging and diagnostics.  BTW, the most common Ops issue is, “Your service is too slow.”  This is painful to diagnose.  It is often an issue with the customer’s own network infrastructure for example.  Make sure to hit this one hard.  You need to know how many milliseconds were needed for each leg of the journey.  We didn’t finish this one, but were actively thinking of implementing capabilities like Google uses to tell with code at the client when a page seems slow.  Our pages all carried a comment that told how long it took at the server side.  By comparing that with a client side measure of time, we would’ve been able to tell whether it was “us” or “them” more easily.

3.  Did you have to perform a manual operation or write code to fix the problem?  If so, you need to automate whatever it was.

This all argues for the skillset needed by your Ops people, BTW.  It also argues to have Ops be a part of Engineering, because you can see how much impact there is on the product’s architecture.

Hit the Highlights of Efficient Architecture

Without going down the rathole of premature optimization, there is a lot of basic stuff that every architecture should have.  Thread pooling.  Good clean multi-threading that isn’t going to deadlock.  Idempotent operations and good use of transactions with rollback in the face of errors.  Idempotency means if the operation fails you can just do it again and everything will be okay.  Smart use of caching, but not too much caching.  How does your client respond to dropped connections?  How many round trips does the client require to do a high traffic page?

We used Java instead of one of the newer slower languages.  Sorry, didn’t mean to be pejorative, and I know this is a religious minefield, but we got value from Java’s innate performance.  PHP or Python are pretty cool, but I’m not sure they are what you want to squeeze every last drop of operations cost out of your system.  The LAMP stack is cheap up front, but SaaS is forever.

Carefully Match Architecture with SLA’s

The Enterprise Software and IT World is obsessed with things like failover.  Can I reach over and unplug this server and automatically failover to another server without the users ever noticing?  That’s the ideal.  But it may be a premature optimization for your particular application.  Donald Knuth says, “97% of the time: premature optimization is the root of all evil.” 

Ask yourself how much is enough?  We settled on 10 minutes with no data loss.  If our system crashed hard and had to be completely restarted, it was good enough if we could do that in less than 10 minutes and no loss of data during that time.  That meant no failover was required, which greatly simplified our architecture. 

To implement this, we ran a second MySQL replicated from the main instance and captured EBS backup snapshots from that second server.  This took the load of snapshotting off the main server and gave us a cheaper alternative to a true full failover.  If the main server died, it could be brought back up again in less than 10 minutes with the EBS volume mounted and away we would go.  The Amazon infrastructures makes this type of architecture easy to build and very successful.  Note that with coarse-grained multi-tenancy, one could even share the backup server across multiple multi-tenant instances.

Don’t Overlook the Tuning!

Tuning is probably the first thing you thought of with respect to cutting costs, right?  Developers love tuning.  It’s so satisfying to make a program run faster or scale better.  That’s probably because it is an abstract measure that doesn’t involve a customer growling about something that’s making them unhappy.

Tuning is important, but it is the last thing we did.  It was almost all MySQL tuning too.  Databases are somewhat the root of all evil in this kind of software, followed closely by networks and the Internet.  We owe a great debt of gratitude to the experts at Percona.  It doesn’t matter how smart you are, if the other guys already know the answer through experience, they win.  Percona has a LOT of experience here, folks.

Conclusion

Long-winded, I know.  Sorry about that, but you have to fit a lot of pieces together to really keep the costs down.  The good news is that a lot of these pieces (metadata-driven architecture, cloud computing, and so on) deliver benefits in all sorts of other ways besides lowering the cost to deliver the service.  Probably the thing I am most proud of about Helpstream was just how much software we delivered with very few developers.  We never had more than 5 while I was there.  Part of the reason for that is our architecture really was a “whole greater than the sum of its parts” sort of thing.  Of course a large part was also that these developers were absolute rock stars too!

Ironically, Big On-Prem Enterprise Software is the Most Customer Driven Software There Is

Vinnie Merchandani thinks big Enterprise software involves building a lot of features nobody wants.  He riffs about underutilized features and wishes for a market like the iPhone’s AppStore.   He wishes SaaS vendors to use their real time visibility into what their customers are doing to give them that kind of market-driven visibility into what to build.

Vinnie and I are fellow Enterprise Irregulars, and I really enjoy his perspective most of the time, but he has it all wrong on this one.  Backwards in fact.  Sorry Vinnie!

You see, Big On-Premises Enterprise Software is more customer driven than any other kind of software there is.

“Huh, why is that and what are you talking about, Bob?”

As one person described it to me, for these sorts of companies the Customer is God and Sales is the Church.  Sales people are natural arbitrageurs.  Any time a question comes up they are rapidly computing who to say “no” to.  Is it easier to negotiate with the customer about their request, or is it easier to negotiate with the company?  The bigger the deal and the better the salesperson (and the two go hand in hand, don’t they?), the more likely it is they choose to say “yes” to the customer and “no” to the company.

All that feature bloat in those products is as a result of saying “yes” too many times to customers.  “Yes” was easier than educating them about better ways to do things, or on how while it seems like “yes” is a good answer today, tomorrow they will have enough experience with the software to realize “yes” is no longer important.

Many view the essential function of marketing and sales as finding ways to say “yes” as often as possible, no matter what the cost.  But here is the problem.  Many times “yes” is exactly what wrecked the product.  “Yes” led to the kinds of problems Vinnie wants to escape from.

Look at it this way.  Marketers are fond of saying that since everyone consumes marketing, they all think they are good marketers.  The same is true for products.  Every user wants to tell you how to fix it before they have even explained what the problem is that needs fixing.  There is a whole school of product management that holds that the way to build great products is to constantly survey end users, build great big prioritized lists of what they want, and then give it to them.  That’s how Microsoft builds its products, for example.   But just as everyone who watches movies is not Steven Spielberg, most people who think they have a great idea for how to change a product are probably not helping.

When you approach things that way, you have a lot of trees and no sign of a forest.  Microsoft is known for successful products (unfortunately more in that past than present), but not for great products.  How do companies operate that are known for great products?  Well, consider Apple, which is perhaps the diametric opposite of the democratic product design process.  They annoint a very few philosopher kings who have the vision.  They’re spurred on by the philosopher emperor-in-chief Steve Jobs until they get it right.  Insanely Great Products are the end result.  Apple’s brief fling with moving away from that approach with Soda Pop King John Sculley was a total disaster for them.  On a much smaller scale, 37Signals is a similar model.  They actually take features back out of products if they think they were a bad idea.  That’s something that is very hard to get customers to recommend very often, and they take heat for it.  But they stick to their guns and they are generally quite successful.

Operating that way leads to conceptual integrity.  You get a forest instead of just a bunch of trees.

Companies can’t afford to ignore customers, there has to be a balance.  You need people making product decisions who are enlightened enough to connect the two ends together.  They’re going to ensure a forest gets built, but they will let customers position enough of the trees to keep them happy too.  And, where a tree is being positioned that’s in the way of a vital river needed to bring water to the forest, they will object and stop that tree being planted.  Even if it means losing one customer.  Because it ensures a healthy forest that ultimately leads to more customers in the end.

It’s a tricky process.  If you have a company that either ignores customers too much, or gives in too easily, you’ll have a product that suffers for it.  Visionaries have to be good listeners, and they have to go forth among the customers to understand their needs.  They have to be good at selling their point of view too.  If you can convince a customer to buy into the vision, they’ll quit trying so hard to plant trees in the wrong places.  Steve Jobs does all that in spades.  Is it any wonder he has been so successful?

AmazonFail Shows Data Matters Too

No software company in their right mind would change code and move it to production without extensive testing to make sure the new code wasn’t broken.  It’s a tried and true business process supported by tools like automated build (gather up all the files, compile as necessary, package, and produce a running version), source code control systems (check in and check out with auditing so all the files that go into a particular version are known and verified), and so on.  That’s all fine and well, and though I do sometimes find a company that hasn’t made it to even that basic stage of operational process, there is a level of complacency associated with having such a process working.  The AmazonFail incident shows us that there is a lot more than just program code at stake here. 

First, what was the AmazonFail incident?  It seems that Amazon suddenly started delisting Gay and Lesbian publications from their sales rankings.  This had the effect of removing the books from search according to the WSJ.  Needless to say, the incident resulted in a great hue and cry since many assumed Amazon had done this intentionally and took it as evil behavior (along the lines of Google’s “do no evil” mandate).  Before long there were charges of  online censorship, and Twitter and the blogosphere were lit up bright talking about it.  Rumors even emerged that the incident was a result of hackers.  Eventually the whole thing became known as “AmazonFail“, and there is a hashcode for that on Twitter if you want to read all about it.  As I write this, “#amazonfail” is the third most popular search term on all of Twitter.

Clearly it’s been a major PR black eye for Amazon, but what caused it?

Amazon calls it, “an embarassing and ham-fisted cataloging error.”  Ultimately it affected over 57,000 titles including books well beyond the Gay and Lesbian themes first reported.  A more detailed internal story comes to us from Andrea James of SeattlePi:

Amazon managers found that an employee who happened to work in France had filled out a field incorrectly and more than 50,000 items got flipped over to be flagged as “adult,” the source said. (Technically, the flag for adult content was flipped from ‘false’ to ‘true.’)

“It’s no big policy change, just some field that’s been around forever filled out incorrectly,” the source said.

Amazon employees worked on the problem well past midnight, and then handed it over to an international team, he said.

Doesn’t this sound just like the sort of problem that can be caused by making a minor code change and then rolling out the changed code to production without adequate testing?  First, it was a minor change.  One field was filled out incorrectly by one person in France.  Second, it created a huge problem as minor changes in code often can.  Third, the problem was not immediately obvious, nor was the cause, so it got pretty far along before it could be fixed.  Yet it wasn’t code, it was just data.

In this day and age of Cloud Computing, SaaS, and web applications, data is becoming increasingly just as critical as code.  Metadata, for example, is the stuff of which customizations to multi-tenant architectures are made of.  In that sense, it is code of a sort.  “Soft” heuristics are common in search applications that have to decide which words to ignore completely, how to weight different words, which words might be profanity (or adult content in this case), which words are positive, negative, might be Spam related, and all the rest.  That’s all critical metadata to a search engine such as Amazon’s.  There’s a lot of other critical data to consider including credit card and other privacy-related information, financial information (what if someone gets the decimal point wrong when entering sales tax for a particular area?), and so on.

Data drives modern applications so completely, that we have to think about what this means for the security and robustness of our applications.  We’re still in our infancy on that front.  Modern software will test all the data entered to be sure it doesn’t contain a malicious payload (the SQL injection attack is one way to hack by entering special data in a field exposed to users) and there are many similar low level tests that are made.  But what about the business process that ensures the integrity of that data?  How can a single individual in France create such a big problem for Amazon by changing a little bit of data?

Let’s assume for the moment that we choose to treat that data as code.  That would mean we do some or all of the following:

-  Archive all the data changes in the equivalent of a source control (also called a “version control“) system.  We’d know every version of the data that was entered, who entered it, when they entered it, and there would be comments about why they entered it.

-  The incorporation of that data into a production environment would happen only at carefully orchestrated times (called “releases”) and only after testing had been done to ensure no problems were created.

-  If a problem manifested itself, an automated system would be available to roll back the changes one or more versions until the problem went away.  This is an important step with extremely serious problems because earlier releases will have been functioning long enough that there are no known serious problems.  Rolling back gives time to find the error without exposing all the customers to it in the meanwhile.  The error is eliminated and then the updated change is tested, and rolled out again.

Does that sound hopelessly painful as a process?  It’s exactly what most software developers go through for even the slightest change to code.  I’ll admit it would be too painful for every data change.  Amazon must add, delete, or change thousands of new listings every day.  Each one can’t be a full development release cycle.  But it does seem that applications should have some safety valves built into their fabric, and into the all-important data that they rely on.  Changing a listing is different than changing some data that almost 60,000 books rely on in search.  That data should be marked as sensitive and handled differently. 

There’s lots of architecture and process work to think about in order to avoid or minimize similar problems in the future for a whole host of applications.

Related Articles

It’s the Data Stupid.  Vinnie Mirchandani offers more examples of how we get into trouble with data.

TechCrunch gets it all wrong in a guest post by Marry Hodder.  Hodder argues unconvincingly that this was all due to an algorithm gone wrong and kept secret.  What happened is precisely NOT an algorithm.

If it had been an algorithm, it would have been an unambiguous set of rules by Hodder’s own Wikipedia-quoted definition.  Algorithms are explicit for their creators, and they are not accidental.  That they may be hidden from users does not detract from their explicitness, purposefulness, or unambiguity.

To assume an algorithm is at fault, is to misunderstand what an algorithm is, or to assume to Amazon purposefully set about creating an explicit and unambiguous set of steps to discriminate against Gay and Lesbian titles.  While the conspiracy theorists may like to natter on about this theme, it ain’t so.

The problem here, and my point in this blog post, is that we assumed we could focus on the algorithm and ignore the data that fed it.  Clearly a bad assumption.  In so doing, we allowed a minor change by an individual to create a major PR problem.  If we really think it was all a conspiracy against all things Gay, why not look for more explicit evidence.  Did Patricia Cornwell’s books about Kay Scarpetta, which include some Gay themes (Scarpetta is Gay) disappear as well?  I don’t think so.

It’s been true since we’ve had computers:  garbage in leads to garbage out.  The data is just as important as the algorithms.

Developers: Be Passionate About What You Build

I’m parsing carefully Seth Godin’s latest post on expertise and passion.  He’s asking questions like:

Should the person who runs the customer service operations at a ski school also be required to love skiing?

Can the CFO of a large church be an atheist?

Does the head of marketing at Kodak have to have a passion for chemicals?

He winds up preferring passion for what you do to passion for the product its associated with, although he freely admits he’d like both.  Isn’t it ironic he talks about Kodak as involving passion for chemicals rather than photography, BTW?

I’m left a little uncomfortable with the whole post.  It seems very un-Godin-like.  He’s usually unequivocal in his views, and yet he seems to allow for the possibility of someone who likes being a CFO but could care less about what the organization does. 

Where startups are concerned, and especially when it comes to software development, I can’t be so unequivocal.  Looking at my own personal experience running development organizations, it would be impossible for me to do that well if I wasn’t passionate about the products we were building.  Perhaps you don’t have to love skiing to make sure customer service operations at a ski school are making customers happy.  But I don’t see how you can build insanely great software if you have no rapport with the software or its users.  Software Engineering is one of those disciplines where it’s all too easy to lose the plot and become infaturated with tools, platforms, languages, patterns, and any other thing that the customers often can’t see at all.  It leads to all the wrong things, and ultimately, to poor software.

So when you’re building your development team, when you’re hiring its members, find a way to tell whether they’re passionate about what they’re building and who they’re building it for.  It’s important.  The oft-repeated mantra to hire people who are “smart and get things done” just isn’t good enough.  In fact, I would take “passionate, and gets things done” first if I couldn’t add smart into the bargain.  For a startup, you need to insist on all three.

Is Computer Science Broken, and Should Computers Be Fractal?

Interesting post via Dare Obasanjo who passes on Joe Gregorio’s post “CS Broke”.

Dare seizes on one part of Gregario’s post as being the important one:

Computer Science, at a deep and fundamental level, is broken, and that applies not only to software but to hardware. One of the reasons that I have this feeling is that after programming for the past 25 years the field hasn’t really changed. The conversations aren’t any different. You could substitute ‘Windows API’ or ‘Borland CGI’ for ‘HTML and CSS’ and you’d be having the same exact conversations I had 15 or 20 years ago. We still struggle with leaks, be it memory, or file handles, or threads, or whatever. We still have race conditions. We still struggle with software that grows linearly in features but exponentially in complexity.

Obasanjo’s response makes sense:

1. If your definition of computer science includes HTML & CSS or Win32 then you’re doing it wrong. In the words of Edsger Dijkstra, computer science is no more about computers than astronomy is about telescopes.
2. Even if you limit the discussion to computer programming, does it also mean that civil engineering is broken because people today still have to discuss and solve the same kinds problems faced by the builders of the great wall of China or the Roman aqueducts?

But I think Dare has missed the more interesting part of Gregorio’s point, the fractal part.  It’s fair to raise the issue that computer science is no more about computers than astronomy about telescopes, but that misses a bigger point.   First, there are other sciences and disciplines that are concerned with telescopes.  If computer scientists aren’t concerned with computers, who is?  I think Dijkstra went to far with that glib dismissal.  Someone has to be concerned.

Second, telescopes are further removed from astronomy than computers are from computer science.  Yes, there are abstract and theoretical corners of computer science such as automata theory and studies of algorithms that can’t exist or make sense on today’s computers (have you got a quantum computer yet?).  But there is a also a lot of computer science that’s engineering and is concerned with what we do day to day. 

Lastly, civil engineers do discuss problems similar to the great wall of China or the Roman aqueducts, but by now those problems are much more well understood than the issues Gregorio is concerned with.  The conversations the civil engineers are having are at a much higher level with less constant reinventing of the mud bricks pulled by donkeys on sleds methods used by their Roman and Chinese predecessors. 

But is it really so bad?  And what does Gregorio mean with his answer to the problem, which is concise but cyptic:

I’ve been struggling to find a way to express this concisely. The best way I’ve found so far is to ask:

What part of the computer in front of you is fractal?

The answer is none of it, yet in nature, which has been at this game of computation for billions of years, everything is fractal. We’re doing it all wrong.

The comments on Gregorio’s post are interesting.  My gut reaction, which is that much of what Gregorio decrys is what happens when computer science is ignored and we push ahead anyway, is reflected in the comments.  HTML itself is an ugly mess as are many languages that grew rather than were designed.  I always liked Wirth’s languages (Pascal et al) much better than C and its derivatives.  One actually reflected good computer science and was in fact easier to write compilers for as a result while the other was a half step beyond a good macro assembler and sure seemed a hack.  Certainly pragmatism wins the day in many cases and C beat Pascal for various reasons though there remains a dedicated following to Delphi to this day and its creator Anders Hejlsberg is at work with Microsoft trying to make C a better place.

But this still misses the fractal point.  What would it mean for the computer in front of you to be fractal?  Wikipedia defines a fractal thus:

A fractal is generally “a rough or fragmented geometric shape that can be split into parts, each of which is (at least approximately) a reduced-size copy of the whole,”^[1] a property called self-similarity.

Some of the comments on the original thread imply the web or nature itself is fractal.  There are aspects which are fractal in both cases, but in neither case is everything fractal.  Fractals are but one way of capturing complexity in an abstract way.  In the same way, I’m not sure it makes sense to insist that computers be fractal.  Fractals I suspect are an appropriate way to view and schedule massively parallel algorithms to many cores, and that may be a good literal application of fractals.  The Thinking Machine of Danny Hillis is an ideal fractal architecture because the notion of CPU and memory merge to a single entity.  I still think Hillis was just ahead of his time and we’ll get there.  When we do, it may take some sort of fractal model to make it work at all easily or well.  Hillis showed what can happen when the astronomers build the telescopes and the Thinking Machine was a fascinating architecture.

Being a software guy, I tend to focus more on software than machine architecture though.  For me, the manifestation of a fractal nature comes from layered architectures such as the network layering discussed by Tannenbaum in his great textbook Computer Networks.  There are similar layered abstractions that exist elsewhere, but we seldom see an opportunity for practitioners to reach across layers or to create their own layers.  That’s a shame because it isn’t hard to do and leads to a lot of power.  I think of Domain Specific Languages as a sort of fractal concept because we create new languages on demand for specific tasks or domains.  I have argued before that creating a language is the only way to realize the ultimate power of the computer because executing a language is really what they do uniquely.  It’s what gives them their power.  Understanding this is definitely a computer science issue.  Read Godel Escher and Bach for a wonderful discussion of the deeper meanings to be found.  You can’t read through “the eternal golden braid” and not think of things at least somewhat fractally.

Sadly, this brings me back to the conclusion that most of the time computer science is ignored and we hack on.  There are few equipped to create a Domain Specific Language that makes solving a problem easy rather than just plunging in and hacking it out.  It’s not clear to me you can teach a person meta-thinking, which is what I call the kind of thinking that boils everything down to a language.  The distant cousin of this approach has us building a framework to solve a problem, rather than a language.  That’s not nearly as good, but often we don’t even do that.

Appologies to my regular audience for having gone off on a Geek tangent.  It happens!

If, OTOH, you liked it, here are some similar writing from this blog:

The X in Software Engineering: Exactly!

Software Handyman?!?? Not at all. Software Itelf Is Math And There Are Engineers And Scientists

Is Programming Like Music or Engineering, and Must it Be Unintuitive?