The X in Software Engineering: Exactly!

Thanks for this from Discipline and Punishment:

The underlying desire — that programming ought to involve rigorous, logical reasoning about clearly defined abstractions — is largely correct. The confusion arrives when the content — this ability reason precisely about very imprecise things (like the real world) — is confused with the form, some magical ‘X’ that instantly elevates any activity to programming. Unfortunately, like all idolatry, it’s not a rational conclusion so disabusing individuals of such notions is terribly difficult. But I think they all come around eventually. All disciplines that don’t participate in metaphor, that revolve around creativity in the face of severe constraints, eventually come to realize that less is more.

Software Handyman?!?? Not at all. Software Itelf Is Math And There Are Engineers And Scientists

Ravi Mohan says the title “Software Engineer” is wrong because real engineers use mathematics every day.  He goes on to clarify this view slightly by saying, “if I am not using “theory” on a day to day basis, I am not *engineering* anything.”  Modeling is also, apparently a big part of what Mohan thinks Engineers do that Software Engineers do not.  He concludes by saying the title, “should something like ‘Software Maintenance Worker’ or ‘Software Handyman’, but I guess it is easier to hire someone if his job title is ‘Rodent Officer’ instead of ‘Ratcatcher’”.

It would be easy to chalk the article up as a linkbaiting exercise more so than a thoughtful discourse, and it is, but the topic deserves a deeper examination.  The issue of whether Software is science or art comes up a lot, and I’ve written about it myself, so I have some views.

Mohan is a bit vague about whether this applies to all software engineers, or the subset engaged in writing business or enterprise software.  He relies heavily on Raganwald’s discourse here, which is largely focused on business software.  It’s important to review the Raganwald article, No Disrespect, because it offers a decidely different view than Mohan portrays.  Mohan takes the view that Reg argued in the article that programmers are clerks.  He’s missed the subtlety.  Some programmers are clerks just as some engineers in other professions are clerks.  The question Reg poses is whether you want to be a clerk?  If not, then you need to rebel a bit about how your professional situation is treating you and how it operates.  Reg points out that if you’re going to act like a clerk, you’ll be treated as one and shouldn’t complain.  In fact, the article Reg suggests if you want to understand the value of a CS Degree is quite interesting.  In it, Brian Hurt talks about an awful lot of things that sound exactly like what Ravi Mohan says real engineers do.

Mohan’s article made me laugh thinking back on my own Computer Science education.  It was under the auspices of the Math Sciences group at Rice University, and if you think we didn’t do math, you’re mistaken.  When I think about the kinds of math we did, that brings me to my next thought on the subject.  The argument can be made that traditional math is of use to Software Engineers.  Reg starts down that path himself in Raganwald’s response to Mohan.  It’s accurate, but I’ll leave that response to others. 

Here is my, somewhat more radical response:

Software itself is math.

I defy the Ravi Mohan’s to give me a formal definition that shoots that down.  Software is math.  Code is math.  Whether you are from the school that says math is symbol pushing, or the school that says math is giving a notation to intuition, there is a home for you in software.  How could you read something like Godel Escher and Bach and not come away knowing that software is math?  Certainly those that get down and dirty with a detailed understanding of algorithms, programming language semantics, or even lambda calculus (gee, its called a calculus for a reason, no?) must see that this is math.

So I think Mohan is, in the end, tragically wrong in his view that Software Engineers (yes, I am entitled to call them Engineers once again) don’t use math.  In fact, whether you understand software as math is perhaps the defining question of whether you are a Computer Scientist or a Hacker (no pejoratives on the latter).  But this is true in the other Engineering professions too.  Many Engineers are cookbook engineers.  They don’t derive the math.  They don’t create new math.  They just apply the math.  These other disciplines have their cookbook approach.  There is a list of the different kinds of bridges: truss bridges, suspension bridges, yada, yada.  Likewise we have lists of algorithms and design patterns.  Not many engineers ever create a fundamentally new bridge and not many Software Engineers create a new algorithm or design pattern.  So where’s the rub?

Getting back somewhat to the original motivation, which was to slam Enterprise software as being a land of clerks: okay, they have a point.  It can be that way.  The clerks do still use math, but it is mostly arithmetic with a little algebra and no calculus or even trig, metaphorically speaking.  However, it doesn’t have to be that way, and there can be significant competitive advantage when an organization moves away from that point of view.  Ironically, it goes back to the math and “oftware itself is math” issues.  One of the things that separates the Enterprise Clerks from Enterprise Software Engineers is an ability to reach both hands deep into the malleable stuff that is software and mold more exciting architectures.  SOA scratches at the surface, but is a step beyond clerkdom.  I don’t want to get off on a tangent defining how to do exciting Enterprise architecture, but I can tell you it can be done and it is worth doing.

The last point I’ll make is to point out:  isn’t it interesting that many of the newest languages and trends are even “mathier”?  Closures, anyone?

Is Programming Like Music or Engineering, and Must it Be Unintuitive?

Two different blog posts hit me today with one of those unconscious knee jerk desires to disagree.  First was Joel Spolsky wishing that undergraduate computer science programs would quit spending so much time on formalisms and become more like a Julliard music program.  Second was Raganwald contending that if a language is really going to push the envelope of “better”, it must by necessity be even less intuitive.  Taken individually, I probably could have swallowed these pills without comment, but wouldn’t you know it: both wound up back to back in my feed reader.  Somehow they tickled the same mysterious place in my subsconscious, but I had to actually write this diatribe to understand how.

Let’s start with Raganwald.  On the face of it, the logic is pretty sound.  It’s not unlike one of my favorite quotations, which goes:

The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself.  Therefore all progress depends on the unreasonable man. 
George Bernard Shaw

The essential thesis that Reg works from is that all programming languages are equally powerful, because they’re Turing complete.  As such, he wants to talk about what makes one language better than another.  He concludes at one point that since all languages are equally as powerful, and since better is subjective, that better languages make you a better programmer, which in and of itself is subjective.  Also, making you a better programmer by definition means moving you out of your comfort zone.  Because of all that, “better” languages have to be unintuitive.  “QED”, as we used to say.

Right at this point I’ve had too many almost-rights go by to be able to sit still.  First, “better” does not have to be subjective.  We could choose to quantify it in some way.  I know that as a profession, programmers have a tremendous love hate relationship with various metrics to the point where we largely throw up our hands and dismiss them all as useless.  Sorry, but I am not in that camp.  I think there are many useless metrics, but not all are useless any more than all evaluations of every programming language are hopelessly subjective personal opinions.  I am even happy to accept fairly simple-minded quantifications such as being able to write similar things in fewer lines of code.  And please, don’t waste our time with silly contrived examples where fewer lines of code result in backwards progress.  In general, and with appropriate subjective oversight to weed out the silly off-point examples, I am convinced that fewer lines of code is a good thing.  Here is a harder one:  how about fewer clearer lines of code?  That’s going to be harder to quantify as a metric, and I can accept that. 

Now here is where I guess Reg and I part company:  is a language that needs fewer clearer lines of code necessarily less intuitive?  Reg seems to say “yes”, that language will be less intuitive.  I say that by the very construction, i.e. the lines are clearer, the answer must be “no.”  Is the only resolution that it is impossible to create a language using fewer clearer lines of code?  Gosh, I hope not, because that seems to say we can’t build better languages unless you want to quarrel with “fewer” or “clearer” not being better.  That’s my “QED”, but let’s tease things apart further, because it gets interesting.

If nothing else, this quantification business reflects a lack of good formalisms being available for a lot of squishy things like clarity of code.  There is another phenomenon at work too with this word formalism.  BTW, you can substitute “model” there if it helps make what I’m saying clearer.  There may be no single formalism or even a small set that is best for all problems solved by all programming languages.  Rather, we may have different fairly independent domains with happy little language ecosystems that work best in one or a few, but that may be terrible in others.  I am surprised and amused at the apparently fractal nature of this formalism by domain concept.  Whenever we think we’re close to finding the universal formalism with which to create the best possible language, we build something like Lisp.  After it’s built, and we use it for a while, what we discover is that such universal formalisms are really good at is creating domain-specific languages, but that they are not so good at many other things.  So in a sense, they’re lousy tools for any particular problem,  but they make it easy to create a special tool that gives the best possible solution to the sub-domain.  See what I mean by fractal?  There’s something very cool about that fractal idea that just feels right.  If you agree, it probably means we’re communicating.

Is it any surprise that it takes many formalisms to make the disparate problems we face soluble with fewer clearer lines of code?  I’m not surprised.  I’ve fiddled with just a few domains and languages in my career, and there are many differences.  Here are some I’ve played with: 

-  Database problems like the inherently parallel and set theoretic languages including the rather homely but oh-so-practical SQL. 

-  Bootstrapping everything from nothing can be done very easily with Forth.

-  Certain kinds of graphics cry out for the purity of mathematical languages with an ability to manipulate matrices and deal with collections of graphical entities.  Some like Fortran for math, but I’ve always like APL and spreadsheets.  I never played with Mathematica enough, but it is likely another alternative.  Tying something like APL and spreadsheets together with graphics in a suitably expressive way would probably make for a wonderful graphics language.  I’ve certainly seen interesting graphics come out of Mathematica, so maybe it’s been done.

-  User interface is a surprisingly parallel problem space dealing with events.  Most frameworks make this painful.  I’ll bet there is an opportunity for a great language in this niche.  I like Adobe’s Flex a lot, but it is far from perfect.  Constraint oriented programming (Thinglab!) is closer, so I am very hopeful about Adobe Thermo raising the bar further.

-  Low-level systems programming of memory allocators, process schedulers, and the like loves languages from the C family.   The less successful branch that included things like Modula-2 and Eifel is not bad either.  There are likely more recent developments I’m not familiar with.

-  Application domain specific programing likes Ruby. 

-  String processing is another world of fascinating special-purpose formalisms such as regular expressions and weird parser generator languages.

-  Creating new languages and pure computer science concepts seems to benefit hugely from Lisp. 

All of the above are sharply affected by the choice of accompanying frameworks and libraries, so they have to be considered as a whole.

This is why I’m so convinced that polyglot systems are the way to go.  Whether the polyglot family has some underlying “assembly language” like Lisp or even Ruby is almost immaterial.  The virtual machine can also be the underlying unifying theme, so long as the various polyglot languages can communicate with one another. 

A last comment on this way of thinking is that I feel Design Patterns are simply ways of imposing or adding those formalisms a particular language doesn’t directly support or doesn’t support well.  There’s been a lot of back and forth on this in the blogosphere, and it seems controversial.  Some say that a perfect language would need no design patterns.  I don’t think so.  I hope I’ve pointed out adequately that the domains require different languages so that there are no “perfect” languages.  From my perspective, you could take any design pattern and come up with a language construct that builds the pattern into the language.  Whether it makes sense or not is another issue.

Languages can become too boroque and obtuse when they are burdened with too many formalisms.  This is particularly true if the formalisms are at war with one another.  Too many ways to do the same thing may not be helpful.  I’d better stop, because I’m rapidly heading back to my polyglot programming soapbox.  Give me a set of very concise and relatively small langauges for well-defined problem domains.  Perhaps I’ll take one “general purpose”  (Jack of all Trades, master of none?) language to fill in the gaps.

Which brings me to Joel Spolsky’s post.  He wants to forget a lot of the formal training and immerse undergrads in large programming projects coordinated by talented teachers.  Joel is responding to a lamentation that schools are too quick to press kids into Java and don’t teach enough of the old formalisms (now you see why I’ve used that peculiar word so much!).  His answer is to create what he views as a “Julliard” style curriculum where the students build relatively complex software in teams under the watchful eye of talented teachers and without so much of the formalisms.  In this model, students would sign up to build a real piece of software of some kind, perhaps a game or social network.

I had one of those formal computer science educations.  I only have the Bachelor’s degree, but I completed all of the work for a PhD as well.  I just felt it would be more fun to write a business plan for my first startup than a thesis (something which maybe Joel and I agree on given his post).   We covered the formalisms in spades.  There was also an opportunity to take on some large software projects, but that was perhaps 1/3 of the curriculum. 

You are probably familiar with the observation many have made that the top 5% of programmers are perhaps 20 times more productive than the average programmer.  I know this to be true, and have seen it many times.  What does this have to do with the discussion?  Simply this: the amount of experience does not seem to matter in determine whether you’re in that 5%.  It does not seem to be possible to teach one who is not a 5% performer how to get there.  I won’t try to prove that here, take my word for it, or at least ask yourself what it means if it is true.

What is more relevant is my observations on watching that 5% group.  What did they do better?  I used to say the thing that really set them apart was a facility with factoring of all kinds.  Those were the words I used, but factoring has so many meanings these days that I feel I should clarify.  The top performers were really good at flipping problems around and viewing them from many angles.  The top players could do it from extreme “meta” positions that were layers of abstraction away from the core problem.  These people lived and breathed isomorphisms, whether or not they had any idea what the word meant.  I’ve described the application of isomorphisms to creativity in an earlier post on “The Medici” effect.  It’s worth a reread.

By contrast, the average performers tended to view everything according to a very limited set of models or formalisms they grasped.  They would try to hammer every problem into one of those few models.  In the worst cases, the models they were able to understand and employ were limited simply to basic structured programming constructs.  With difficulty they could get control flow.  They were not especially good at breaking things into functions or procedures, let alone modules.  Object oriented programming was often a bridge too far. 

We had several classes where this distinction became obvious.  One was a survey of programming languages that included Lisp, APL, SNOBOL, Prolog, and a couple of others I’ve forgotten.  We learned each new language and had to complete a model program that demonstrated we had really grasped the new models and formalisms being offered.  Another was a course wherein we built up an entire language using the fundamental Turing constructs.  The language created was our own, and everybody’s was different, but we had to build it up from the basics of Turing.  This course was done in Lisp.  It was a 300-level undergraduate course, but it was a definite weed-out course that showed who had it and who didn’t.  The last one was the hardcore algorithms course.  Once again, to understand a full range of interesting algorithms required applying many different formalisms.

You’ve probably guessed my problem with Joel’s approach.  It’s probably fine if you are seeking to create a vocational school for factory programmers.  But, the formalisms are the real meat.  It isn’t the lines of code.  The programmers who got the formalisms could grind out more code than any half dozen of the others combined.  They did so almost automatically and unconsciously.  I suspect Julliard doesn’t work quite as Joel envisions either.  Do students there stretch their thinking about musics “formalisms”, or do they spend most of their time perfecting a single arrangement and composition?

People, go out and learn some new languages.  Learn some new design patterns.  Dig out those formalisms.  It is surprising how intuitive many of them will be.  The more you pick up, the more intuitive future ones will be.  Divorce yourself from a particular language.  That’s what’s holding you up and making things unintuitive.  The best news is that because you don’t have to write reams of code ala Joel, you can learn faster and in your spare time.  The emphasis is on variety more so than volume.

QED

Coté’s Excellent Description of the Microsoft Web Rift

Coté’s latest RedMonk post perfectly captures my reservations about Microsoft, which I refer to as their “rift with the web”.  Here are the relevant passages:

Microsoft frameworks are plagued by lock-in fears. That is, you’re either a 100% Microsoft coder or a 0% Microsoft coder. Sure, that’s an exaggeration, but the more nuanced consequences are that something intriguing like Astoria will play best with Microsoft coders, unlike Amazon’s web services which will play well with any coder.

This thing he calles “lock-in fear” and the extreme polarization (encouraged by Microsoft’s rhetoric, tactics, and track record) that you’re either all-Microsoft or no-Microsoft is my “web rift”.  I’ve written about this a couple of times before, and it never fails to raise the ire of some Microsoft fan or other.

This particular RedMonk post is chiefly concerned, it seems to me, with making sure that Microsoft’s Astoria doesn’t disappear under the continuous din of innovation Amazon is putting up for us lately around their cloud computing services.  The essential new thing about Astoria in Coté’s mind is that it is a RESTful framework rather than a .NET framework:

If you’re just coding to a URL, that’s not quit so bad as coding to a .Net library and all the Microsoft baggage and tool-chain needed to support that.

I agree, but I think Coté has obfuscated two issues together that don’t have to be: the issue of how software components communicate versus whether a solution is hosted/SaaS or not.  One can imagine components communicating RESTfully without any need to have them be hosted in SaaS fashion.  My own bias would be to go ahead and host, but it’s not a requirement and to put the two together as one is conflating the issue (don’t you love that word “conflate” that has drifted into common use here in the valley?).  Coté’s contention that hosting itself will reduce fears of lock-in is also pretty hard to swallow.  While I am again a whole-hearted advocate, giving your software over to a hosted environment and all of its attendant API’s (RESTful or not) is a big step towards lock-in, no matter how you look at it.  This is again an area where Microsoft’s old school monopolist behavior won’t serve it well.  There will be fear, perhaps unreasonable, that Microsoft will take unfair advantage if handed the keys to your kingdom by hosting on their cloud infrastructure.  The problem is that they’ve failed to conduct themselves as the Swiss do in matters of banking.  They are voracious competitors and seemingly always will be.  It isn’t enough for them to win, others must lose.

There are several other interesting points made in the post.  For example, on the issue of hosting, Coté wonders why big companies are so slow to launch.  It’s very true, we’ve seen it for all the big players.  The answer is perhaps that they have more to lose and are more likely to reach the win-lose decision point too quickly and with too much momentum to gracefully correct the problems.  Startups have a distinct advantage in this.  It’s not clear to me why more big companies don’t fund startups expressly to deal with the issue with the intention of acquiring them later if things work out.

Coté also makes a hugely important point about the value of self-service:

My sense is that unless it’s all delivered as a URL with dead-simple docs and pricing (check out the page for SimpleDB), any given technology won’t work out at web-scale.

Put another way, these new technologies need to be completely self-service. If a developer has to ever talk with a human from the company or team offering the project, something has gone wrong.

Self-service is a crucial part of viral growth potential in today’s world.  Any company that releases a product without at least a semblance of a plan for how to make it self-service at some point down the road is laying the foundations for failure.

Related Articles

Latest Microsoft Office service packs decommits support for Older File Formats: Especially Competitors

This is typical of Microsoft’s “we don’t have to be nice, we’re the phone company” (appologies to Lily Tomlyn) behavior.  To access your old files requires you to delve into the registry.  Microsoft claims these old files are a security risk.  It’s tacky, it’s more lock-in, and it’s more evidence that MSFT is up to their same old tricks.

Eventual Consistency Is Not That Scary

Amazon’s new SimpleDB offering, like many other post-modern databases such as CouchDB, offers massive scaling potential if users will accept eventual consistency.  It feels like a weighty decision.  Cast in the worst possible light, eventual consistency means the database will sometimes return the wrong answer in the interests of allowing it to keep scaling.  Gasp!  What good is a database that returns the wrong answer?  Why bother? 

Often waiting for the write answer (sorry, that inadvertant slip makes for a good pun so I’ll leave it in place) returns a different kind of wrong answer.  Specifically, it may not return an answer at all.  The system may simply appear to hang. 

How does all this come about?  Largely, it’s a function of how fast changes in the database can be propogated to the point they’re available to everyone reading from the database.  For small numbers of users (i.e. we’re not scaling at all), this is easy.  There is one copy of the data sitting in a table structure, we lock up the readers so they can’t access it whenever we change that data, and everyone always gets the right answer.  Of course, solving simple problems is always easy.  It’s solving the hard problems that lands us the big bucks.  So how do we scale that out?  When we reach a point where we are delivering that information from that one single place as fast as it can be delivered, we have no choice but to make more places to deliver from.  There are many different mechanisms for replicating the data and making it all look like one big happy (but sometimes inconsistent) database, let’s look at them.

Once again, this problem may be simpler when cast in a certain way.  The most common and easiest approach is to keep one single structure as the source of truth for writing, and then replicate out changes to many other databases for reading.  All the common database software supports this.  If your single database could handle 100 users consistently, you can imagine if those 100 users were each another database you were replication to, suddenly you could handle 100 * 100 users, or 10,000 users.  Now we’re scaling.  There are schemes to replicate the replicated and so on and so forth.  Note that in this scenario, all writing must still be done on the one single database.  This is okay, because for many problems, perhaps even the majority, readers far outnumber writers.  In fact, this works so well, that we may not even use databases for the replication.  Instead, we might consider a vast in-memory cache.  Software such as memcached does this for us quite nicely, with another order of magnitude performance boost since reading things in memory is dramatically faster than trying to read from disk.

Okay, that’s pretty cool, but is it consistent?  This will depend on how fast you can replicate the data.  If you can get every database and cache in the system up to date between consecutive read requests, you are sure to be consistent.  In fact, it just has to get done between read requests for any piece of data that changed, which is a much lower bar to hurdle.  If consistency is critical, the system may be designed to inhibit reading until changes have propogated.  It take some very clever algorithms to do this well without throwing a spanner into the works and bringing the system to its knees performance-wise. 

Still, we can get pretty far.  Suppose your database can service 100 users with reads and writes and keep it all consistent with appropriate performance.  Let’s say we replace those 100 users with 100 copies of your database to get up to 10,000 users.  It’s now going to take twice as long.  During the first half, we’re copying changes from the Mother Server to all of the children.  The second half we’re serving the answers to the readers requesting them.  Let’s say we can keep the overall time the same just by halving how many are served.  So the Mother Server talks to 50 children.  Now we can scale to 50 * 50 = 2500 users.  Not nearly as good, but still much better than not scaling at all.  We can go 3 layers deep and have Mother serve 33 children each serve 33 grand children to get to 33 * 33 * 33 = 35,937 users.  Not bad, but Google’s founders can still sleep soundly at night.  The reality is we probably can handle a lot more than 100 on our Mother Server.  Perhaps she’s good for 1000.  Now the 3-layered scheme will get us all the way to 333*333*333 = 36 million.  That starts to wake up the sound sleepers, or perhaps makes them restless.  Yet, that also means we’re using over 100,000 servers too: 1 Mothers talks to 333 children who each have 333 grandchildren.  It’s a pretty wasteful scheme.

Well, let’s bring in Eventual Consistency to reduce the waste.  Assume you are a startup CEO.  You are having a great day, because you are reading the wonderful review of your service in Techcrunch.  It seems like the IPO will be just around the corner after all that gushing does it’s inevitable work and millions suddenly find their way to your site.  Just at the peak of your bliss, the CTO walks in and says she has good news and bad news.  The bad news is the site is crashing and angry emails are pouring in.  The other bad news is that to fix it “right”, so that the data stays consistent, she needs your immediate approval to purchase 999 servers so she can set up a replicated scheme that runs 1 Mother Server (which you already own) and 999 children.  No way, you say.  What’s the good news?  With a sly smile, she tells you that if you’re willing to tolerate a little eventual consistency, your site could get by on a lot fewer servers than 999.

Suppose you are willing to have it take twice as long as normal for data to be up to date.  The readers will read just as fast, it’s just that if they’re reading something that changed, it won’t be correct until the second consecutive read or page refresh.  So, our old model that had the system able to handle 1,000 users, and replicated to 999 servers to handle 1 million users used to have to go to 3 tiers (333 * 333 * 333) to get to the next level at 36 million and still serve everything consistently and just as fast.  If we relax the “just as fast”, we can let our Mother Server handle 2,000 at half the speed to get to 2000 * 1000 = 2 million users on 3 tiers with 2000 servers instead of 100,000 servers to get to 36 million. If we run 4x slower on writes, we can get 4000*1000 = 4 million users with 4000 servers.  Eventually things will bog down and thrash, but you can see how tolerating Eventual Consistency can radically reduce your machine requirements in this simple architecture.  BTW, we all run into Eventual Consistency all the time on the web, whether or not we know it.  I use Google Reader to read blogs and WordPress to write this blog.  Any time a page refresh shows you a different result when you didn’t change anything, you may be looking at Eventual Consistency.  Even if you suspect others changed something, Google Reader still comes along frequently and says an error occured and asks me to refresh.  It’s telling me they relied on Eventual Consistency and I have an inconsistent result.

As I mention, these approaches can still be wasteful of servers because of all the data copies that are flowing around.  This leads us to wonder, “What’s the next alternative?”  Instead of just using servers to copy data to other servers, which is a prime source of the waste, we could try to employ what’s called a sharded or Federated architecture.  In this approach, there is only one copy of each piece of data, but we’re dividing up that data so that each server is only responsible for a small subset of it.  Let’s say we have a database keeping up with our inventory for a big shopping site.  It’s really important to have it be consistent so that when people buy, they know the item was in stock.  Hey, it’s a contrived example and we know we can cheat on it, but go with it.  Let’s further suppose we have 100,000 SKU’s, or different kinds of items in our inventory.  We can divide this across 100 servers by letting each server be responsible for 1,000 items.  Then we write some code that acts as the go-between with the servers.  It simply checks the query to see what you are looking for, and sends your query to the correct sub-server.  Voila, you have a sharded architecture that scales very efficiently.  Our replicated model would blow out 99 copies from the 1 server, and it could be about 50 times faster (or handle 50x the users as I use a gross 1/2 time estimate for replication time) on reads, but it was no faster at all on writes.  That wouldn’t work for our inventory problem because writes are so common during the Christmas shopping season. 

Now what are the pitfalls of sharding.  First, there is some assembly required.  Actually, there is a lot of assembly required.  It’s complicated to build such architectures.  Second, it may be very hard to load balance the shards.  Just dividing up the product inventory across 100 servers is not necessarily helpful.  You would want to use a knowledge of access patterns to divide the products so the load on each server is about the same.  If all the popular products wound up on one server, you’d have a scaling disaster.  These balances can change over time and have to be updated, which brings more complexity.  Some say you never stop fiddling with the tuning of a sharded architecture, but at least we don’t have Eventual Consistency.  Hmmm, or do we?  If you can ever get into a situation where there is more than one copy of the data and the one you are accessing is not up to date, Eventual Consistency could rear up as a design choice made by the DB owners.  In that case, they just give you the wrong answer and move on. 

How can this happen in the sharded world?  It’s all about that load balancing.  Suppose our load balancer needs to move some data to a different shard.  Suppose the startup just bought 10 more servers and wants to create 10 additional shards.  While that data is in motion, there are still users on the site.  What do we tell them?  Sometimes companies can shut down the service to keep everything consistent while changes are made.  Certainly that is  one answer, but it may annoy your users greatly.  Another answer is to tolerate Eventual Consistency while things are in motion with a promise of a return to full consistency when the shards are done rebalancing.  Here is a case where the Eventual Consistency didn’t last all that long, so maybe that’s better than the case where it happens a lot. 

Note that consistency is often in the eye of the beholder.  If we’re talking Internet users, ask yourself how much harm there would be if a page refresh delivered a different result.  In may applications, the user may even expect or welcome a different result.  An email program that suddenly shows mail after a refresh is not at all unexpected.  That the user didn’t know the mail was already on the server at the time of the first refresh doesn’t really hurt them.  There are cases where absolute consistency is very important.  Go back to the sharded database example.  It is normal to expect every single product in the inventory to have a unique id that lets us find that part.  Those ids have to be unique and consistent across all of the shards.  It is crucially important that any id changes are up to date before anything else is done or the system can get really corrupted.  So, we may create a mechanism to generate consistent ids across shards.  This adds still more architectural complexity.

There are nightmare scenarios where it becomes impossible to shard efficiently.  I will over simplify to make it easy and not necessarily correct, but I hope you will get the idea.  Suppose you’re dealing with operations that affect many different objects.  The objects are divided into shards naturally when examined individually, but the operations between the objects span many shards.  Perhaps the relationships between shards are incompatible to the extent that there is no way to shard them across machines such that every single operation doesn’t hit many shards instead of a single shard.  Hitting many shards will invalidate the sharding approach.  In times like this, we will again be tempted to opt for Eventual Consistency.  We’ll get to hitting all the shards in our sweet time, and any accesses before that update is finished will just live with inconsistent results.  Such scenarios can arise where there is no obvious good sharding algorithm, or where the relationships between the objects (perhaps its some sort of real time collaborative application where people are bouncing around touching objects unpredictably) are changing much too quickly to rebalance the shards.  One really common case of an operation hitting many shards is queries.  You can’t anticipate all queries such that any of them can be processed within a single shard unless you sharply limit the expressiveness of the query tools and languages.

I hope you come away from this discussion with some new insights:

-  Inconsistency derives from having multiple copies of the data that are not all in sync.

-  We need multiple copies to scale.  This is easiest for reads.  Scaling writes is much harder.

-  We can keep copies consistent at the expense of slowing everything down to wait for consistency.  The savings in relaxing this can be quite large.

-  We can somewhat balance that expense with increasingly complex architecture.  Sharding is more efficient than replication, but gets very complex and can still break down, for example. 

-  It’s still cheaper to allow for Eventual Consistency, and in many applications, the user experience is just as good.

Big web sites realized all this long ago.  That’s why sites like Amazon have systems like SimpleDB and Dynamo that are built from the ground up with Eventual Consistency in mind.  You need to look very carefully at your application to know what’s good or bad, and also understand what the performance envelope is for the Eventual Consistency.  Here are some thoughts from the blogosphere:

Dare Obasanjo

The documentation for the PutAttributes method has the following note

Because Amazon SimpleDB makes multiple copies of your data and uses an eventual consistency update model, an immediate GetAttributes or Query request (read) immediately after a DeleteAttributes or PutAttributes request (write) might not return the updated data.

This may or may not be a problem depending on your application. It may be OK for a del.icio.us style application if it took a few minutes before your tag updates were applied to a bookmark but the same can’t be said for an application like Twitter. What would be useful for developers would be if Amazon gave some more information around the delayed propagation such as average latency during peak and off-peak hours.

Here I think Dare’s example of Twitter suffering from Eventual Consistency is interesting.  In Twitter, we follow mico-blog postings.  What would be the impact of Eventual Consistency?  Of course it depends on the exact nature of the consistency, but lets look at our replicated reader approach.  Recall that in the Eventual Consistency version, we simply tolerate that we allow reads to come in so fast that some of the replicated read servers are not up to date.  However, they are up to date with respect to a certain point in time, just not necessarily the present.  In other words, I could read at 10:00 am and get results on one server that are up to date through 10:00 am and on another results only up to date through 9:59 am.  For Twitter, depending on which server my session is connected to, my feeds may update a little behind the times.  Is that the end of the world?  For Twitter users, if they are engaged in a real time conversation, it means the person with the delayed feed may write something that looks out of sequence to the person with the up to date feed whenever the two are in a back and forth chat.  OTOH, if Twitter degraded to that mode rather than taking longer and longer to accept input or do updates, wouldn’t that be better? 

Erik Onnen

Onnen wrote a post called “Socializing Eventual Consistency” that has two important points.  First, many developers are not used to talking about Eventual Consistency.  The knee jerk reaction is that it’s bad, not the right thing, or an unnecessary compromise for anyone but a huge player like Amazon.  It’s almost like a macho thing.  Onnen lacked the right examples and vocabulary to engage his peers when it was time to decide about it.  Hopefully all the chatter about Amazon’s SimpleDB and other massively scalable sites will get more familiarity flowing around these concepts.  I hope this article also makes it easier.

His other point is that when push comes to shove, most business users will prefer availability over consistency.  I think that is a key point.  It’s also a big takeaway from the next blog:

Werner Vogels

Amazon’s CTO posted to try to make Eventual Consistency and it’s trade offs more clear for all.  He lays a lot of good theoretical groundwork that boils down to explaining that there are tradeoffs and you can’t have it all.  This is similar to the message I’ve tried to portray above.  Eventually, you have to keep multiple copies of the data to scale.  Once that happens, it becomes harder and harder to maintain consistency and still scale.  Vogels provides a full taxonomy of concepts (i.e. Monotonic Write Consistency et al) with which to think about all this and evaluate the trade offs.  He also does a good job pointing out how often even conventional RDMS’s wind up dealing with inconsistency.  Some of the best (and least obvious to many) examples include the idea that your mechanism for backups is often not fully consistent.  The right answer for many systems is to require that writes always work, but that reads are only eventually consistent.

Conclusion

I’ve covered a lot of consistency related tradeoffs involved in database systems for large web architectures.  Rest assured, that unless you are pretty unsuccessful, you will have to deal with this stuff.  Get ahead of the curve and understand for your application what the consistency requirements will be.  Do not start out being unnecessarily consistent.  That’s a premature optimization that can bite you in many ways.  Relaxing consistency as much as possible while still delivering a good user experience can lead to radically better scaling as well as making your life simpler.  Eventual Consistency is nothing to be afraid of.  Rather, it’s a key concept and tactic to be aware of.

Personally, I would seriously look into solutions like Amazon’s Simple DB while I was at it. 

Not Everyone Can Program and Not Every Programmer is Great

Reg Braithwaite wrote a column recently called, “The Optimistic View”.  He’s tired of organizations pushing lowest common denominator tools because they’re afraid lowest common denominator programmers will hurt themselves (well, hurt their code really) with stronger tools.  He suggests:

But maybe, just maybe, most programmers rise to the occasion when challenged. Maybe instead of catering to the bottom 10th percentile we can be open-minded about what the 50th to 75th percentile are capable of achieving, much less the top quarter.

It’s an interesting thought, but I confess I haven’t seen an awful lot of evidence it’s true.  I have certainly been surprised by programmers before.  But the surprise has been in just how fast a really talented “junior” programmer can take their place in the ranks of the really good architects.  I have not seen a lot of evidence of the much less talented stepping up to deal with some of the real rocket science of our trade.

He gets a little more detailed in his thinking here:

Spreadsheets taught us that anyone can program. Maybe we should open our minds to the possibility that although not everyone does program in this manner today, they are capable of it if we stop telling them that they aren’t.

And when we say “Anybody can program,” we say that without reservation or disclaimer. We don’t mean anybody can program provided they are banned from using certain langauges, or anybody can program provided we hide the details of relational databases behind ORMs, or anybody can program provided they are not required to learn any idioms beyond those authorized by the local High Priest. We mean anybody can program, period, without being lashed to a heavy yoke and driven through the fields under the whip.

This is the point where we part company.  Spreadsheets did teach us that anyone can program, BUT, they did so by doing some of things he suggests we quit doing with less talented programmers.  They hide lots of details that one would have to know to accomplish the same thing using normal programming.  It is quite instructive to think about what spreadsheets do for us.  Largely, they allow us to quit thinking abstractly and organizationally, two big skills that programmers must possess.  In other words, they eliminate the need to think like a programmer. 

Consider the business of organization, or specifically of ordering.  Spreadsheets let their users examine the trees and the forest just emerges magically, and without much planning.  Tell the spreadsheet the relationship between a few cells using a simple formula.  You never have to worry about what order you do that in.  The spreadsheet does a topological sort and figures out the “natural order of recalculation.”  Because of this, there are no loops to think about.  Control structures are pretty straightforward and are seldom nested.  A simple “IF” is about all there is, and easily 90% of spreadsheets don’t use them. 

There’s a lot more clever hiding that happens as well.  Consider the whole user interface issue.  Spreadsheets give you that for free using their grid metaphor.  It’s amazing what can be accomplished there, but it isn’t really programming.  Perhaps most importantly, it doesn’t try to be.  Most of that look is fixed at design time and is not dynamically programmable. 

Data is another biggie.  For those who are stuck on static typing, spreadsheets have none, nor would it make them any “safer” to offer it.  But there are also no data structures.  Each cell stands more or less alone, with the exception of ranges (or blocks as they came to be called) referenced in formulas.  There is almost no ability to go further save Excel’s confusing array formulas which I’ll wager are used in much less than 1% of cases, and most of those were forced as a way use the LINEST builtin.

The interesting questions are how much of this can be done for conventional programming and how would one go about it?  The first issue is seriously missing.  It’s not clear we understand and agree just where the problems are.  I’m certainly not going to claim to lay it out here, but I will say this.  Layering more and more complexity on is not the answer.  That’s not how spreadsheets got where they are in terms of enabling more programming.  Adding a bunch of syntax around typing may have value for some things (chiefly making the tools better in my mind), but it isn’t how you help people.  If you want to help lowest common denominator people, figure out how to eliminate or radically reduce complexity.  Eliminate abstraction.  Eliminate the need for the peculiar kinds of thinking great programmers must do.

Spreadsheets, BTW, are a graphical dataflow paradigm.  They really are visual programming, just not using boxes and arcs.  They are also constraint oriented programming.  I could go on, but spreadsheets don’t prove everyone is a programmer.  They do something quite different that is even more useful: they make it possible to write programs even if you can’t think like a programmer.

Making All Software Into Tools Reduces Risk

Here’s a radical idea:

All software should be a tool or language because it reduces your risk of failure.

I’ve written before that if an area is important enough, it eventually becomes a language.  We’ve watched it happen over and over again, sometimes in the most unlikely places.  For example, Adobe made it’s start by turning printing into a language called Postscript.  It gave them enormous advantages over the competition.  Here’s another prediction: we won’t see the real universal Social Graph until it is expressed as a language not an application.  Why?  Because as I have said before, it is a living breathing dynamically changing entity that means a lot of different things to a lot of different people.

Why do I say tools and languages are lower risk?  Certainly this flies in the face of conventional wisdom.  Many VC’s, for example, want no part of a tools play.  They see it as extremely risky for a variety of reasons.  Dealing with IT and other technical types seems hard.  As customers they are seen as too demanding.  They want it all for free as Open Source, etc., etc..  That’s one particular market, and there are answers to those questions, but making software into a tool does not necessarily require that you sell to that market.  Sometimes, the tool nature is almost invisible.  Spreadsheets, for example, are really languages.  Creating a spreadsheet is an odd form of programming.  In fact, it’s great because it’s probably the most widely used and understood language that’s ever been created.

Here’s another way to look at it.  I recently wrote an article called “Why Small Software Teams Grow Large“.  It was a response to a number of questions that had come up over my proposition that you need a small team to write really great software.  Here is a question that relates to this post:

Aren’t all the great small team examples tools?

Linux, Delphi, and Quattro Pro are described by Chris as “generic tools built without regard for any specific business logic.”  There are two ways to think about this.  Chris takes the path that says the example is irrelevant because most software isn’t that way.  I take the path of saying that the example is spot on because all software should become a language if done right.  The benefit?  Your small team is enormously more productive as are your customers if you can actually make the language something they can grasp and use.  There is a reason these small teams built languages (though I’m not sure I view Linux as a tool/language).  In fact, Quattro Pro had no less than 18 interpreters buried in the guts.  Very few of them were surfaced for end users, most were there to make the code simpler and the product more powerful.

You see what’s at work there?  Languages/Tools make it possible for smaller teams to be even more productive.  If you buy into the idea that small teams are the best, you must want them to have the best possible tools as well?  What could be better than to make the software they’re working on a special purpose tool?

I’m certainly not the first one to think of things this way.  There is an entire area based on the idea of Domain Specific Languages.  The idea is to create a language around the problem you’re trying to solve in order to make it easier to solve the problem.  Tools like Ruby on Rails or Lisp are extremely good at that task.

Rather than dive into a highly technical tangent, let’s get back to the theme here.  Why would making your software into a tool reduce risk?  We’ve already hit on one aspect–it can make the developers much more productive.  Here is another: a tool makes it much easier for the software to adapt to changing needs of customers.  I do a fair amount of what I call “Termite Inspection” for various VC’s around the Valley.  Some of these engagements boil down to, “This company has a great idea, but they seem to be stuck in a rut.  What should they do?”  Many times what has happened is a company started out with a great idea and some knowledge of the domain.  They built a piece of software that is a very literal embodiment of their view of the domain.  So long as the whole domain and all the customers fit their view, life is good.  However, for most non-trivial (i.e. interesting market size) domains, nobody knows the whole domain.  You learn as you go.  Customers throw curve balls.  If the only way to adapt to that is to either keep adding tons of features to the software, or do expensive custom work on each deal, that’s way too much friction to scale.  A domain specific language makes it possible to manuever a bit at low cost and give customers what they want.

I finally put two and two together on just how important this can really be when reading Fred Wilson’s excellent article on Why Startups Fail.  Interestingly, he classifies all of his failures (and presumably others) into two categories:

1) It was a dumb idea and we realized it early on and killed the investment. I’ve only been involved in one investment in this category personally although I’ve lived through a bunch like this over the years in the partnerships I’ve been in.
2) It was a decent idea but directionally incorrect, it was hugely overfunded, the burn rate was taken to levels way beyond reason, and it became impossible to adapt the business in a financially viable manner.

Can you see where this is going?  Category 1 won’t be helped, and Fred correctly says you can identify this and cut your losses early on.  Building the software as a tool is a perfect antidote to #2.  The “decent idea but directionally incorrect” is strikingly similar to Marc Andreesen’s concept of achieving a product/market fit, which he says is the only thing that matters.  Fred Wilson gives us a great anecdote:

Dick Costolo, co-founder of FeedBurner, describes a startup as the process of going down lots of dark alleys only to find that they are dead ends. Dick describes the art of a successful deal as figuring out they are dead ends quickly and trying another and another until you find the one paved with gold.

Achieving the product/market fit is a matter of trial and error.  It is searching through an unknown territory filled with dead ends.  To survive, succeed, and prosper, your software needs maximum flexibility and adaptability.  That’s the definition of a Tool.  It can be adapted even as the requirements keep changing radically, and it can be adapted very cheaply and efficiently.

The principle should be clear by now, but there are troubling questions of practice. 

First, creating a tool sounds harder than writing an application.  My answer is that it’s harder because of who can do it and who can’t.  Once the tool is created, any developer should be able to use it to create new features.  Creating it is the province of developers a notch or two above the lowest common denominator.  The good news is that you don’t need to many of them.  Given my preferred maximum team size of 10 developers, you should easily get buy if 2 or 3 of the 10 are language creators.  Look for people who’ve done it before and are comfortable with the idea.  Look for people who are adept with dynamic languages and tools like Ruby on Rails. 

Second, delivering a tool sounds like a nightmare for users.  What do you do if your users are not technical?  Just because you’ve delivered a tool doesn’t mean you have to leave sharp blades and flaming torches hanging out of the toolbox.  Nothing is easier to attach a user interface to than a language.  That’s one of the great beauties.  But here’s a real quantum leap: combine your application-as-tool with a ui-as-tool for true power and flexibility.  I’ve been working with Adobe’s Flex along those lines and the results are amazing.  Other combinations that are similar to this would be combining something like PHP, which could be viewed as UI-as-tool with your application.  This is how the various social web 2.0 sites have gotten going so quickly and cheaply.  PHP straddles both sides of the spectrum and allowed these organisms to evolve very quickly.  In the end, you will wind up with a much better user experience because you’ll be able to quickly evolve your UI based on feedback.  Take it into usability testing early, and the combination of application and UI languages will make it straightforward to act on recommendations.

I hope you can get a sense from this post how powerful a competitive weapon having a language versus an application can be.  It actually reduces your costs, increases your flexibility, and let’s you respond with ultimate nimbleness to market demands until you’ve found the best possible product/market fit.

Why Small Software Teams Grow Large And Other Software Development Conundrums

Ever since I can remember there is a background noise in the software development community around a relatively small set of topics:

- Are small teams better than big teams?

- Does language matter?

- What’s the best language?

- How do we achieve quality?

- Is software art and talent driven, or is it engineering and process driven?

The list is probably longer, but you get the idea.  Lately I’ve been involved in some back and forth over team sizes.  For the record, I think small talented teams crush big teams with tons of process in terms of productivity and final results.  They build better software in less time.  It’s been a good discussion, but commenters on various threads bring up issues that are important to address.  Let’s see if we can’t tackle a few here.

Chris Winters’ post is a good place to start, as Chris delivers a lot of meat in a very few lines.

Don’t these small team pundits completely ingore the timeframe?

As Carmack said, “a team of 3 focused and creative people can accomplish almost anything” — given sufficient time. But time is the knob that many (most?) projects have little control over. Everybody has competitors, and everything needs to get out yesterday.

This is truly Mythical Man Month territory.  You can deliver a project sooner with more than 3 developers, but I have serious doubts about delivering a core module with more than 10.  The question is whether what you are building is amenable to being broken down into what are essentially very separate projects.  Sometimes this is possible, more often it isn’t.  To make it possible will require some architectural investment to keep the modules separate, and that investment is serialized into the schedule.

A much more effective knob to twiddle when it comes to time to market is scope.  Cut scope.  Cut it early and cut it again if the project is drifting off schedule.  It is amazing how much scope can really be done away with and still have a release that completely satisfies commercial needs and makes customers happy.  It is a mistake to go for the many-year feature abundanzas that result in products like our friends in Redmond like to ship once every 5 years.

Does “small team” include QA?  People to gather requirements?  People to write documentation?

Great question!  First, on the issue of QA, I have never found a point of diminishing returns, and I have had the luxury of spending a LOT of money on QA during my Borland days.  Simply put, it is extremely difficult to find all the bugs.  No matter how many we could find in a week with a huge team of testers, a small core team of developers was able to keep up.  We started with a ratio of 1 QA per developer and added more QA on top of that as we moved towards shipment.  Those days are probably gone, but boy did it ever help. 

We got a lot of mileage over mobilizing the company and customers to assist.  We paid bug bounties to these folks based on severity of bug, and I know of at least one case where a very talented SE paid for a kitchen remodel with the funds.  OTOH, I had friends working on Microsoft Windows XP where much larger developer teams were involved, and even more humongous QA teams.  According to one friend, they completely deleted the entire bug database and started over again multiple times during the project because the developers simply couldn’t keep up.

As a final thought on this, the number of resources you can bring to bear outside the core developers is a function of the communication load they put on the developers.  Communicating via a bug tracking system is pretty efficient, hence lots of QA didn’t bring down the team.  Having a doc or QA person want to camp out so the developer can educate them is not going to scale.

Isn’t there a huge distinction between creating something and maintaining/improving it?

No, not necessarily.  If your view of improving something is to canvas the user base for every possible feature, build a giant laundry list, and build all of that, you’re going to find it’s very daunting.  But you’re also going to find you quickly defocus the conceptual integrity of your product and wind up with bloatware that does not make your customers happy.  The art of great software is in understanding what’s important to the user experience versus what is merely urgent.  A small team can keep a product fresh and current for years if they’re good at this because they don’t waste time on a load of features that clog things up.

Exceptions?  Beware the things that require tons of code yet qualify as only 1 more feature on the checklist.  Platforms are the arch example of that.  Supporting many printers and display adapters was the old school.  Supporting many databases or app servers is more recent.  It’s not worth it and it is a huge sink on resources.

Aren’t all the great small team examples tools?

Linux, Delphi, and Quattro Pro are described by Chris as “generic tools built without regard for any specific business logic.”  There are two ways to think about this.  Chris takes the path that says the example is irrelevant because most software isn’t that way.  I take the path of saying that the example is spot on because all software should become a language if done right.  The benefit?  Your small team is enormously more productive as are your customers if you can actually make the language something they can grasp and use.  There is a reason these small teams built languages (though I’m not sure I view Linux as a tool/language).  In fact, Quattro Pro had no less than 18 interpreters buried in the guts.  Very few of them were surfaced for end users, most were there to make the code simpler and the product more powerful.

Do we know what we’re building?

Chris hints on another big problem when he says:

And doing all that for non-trivial businesses, with a small team, no matter what language, is a tough job. (Not even mentioning figuring out what it is you’re supposed to build.) In a short amount of time? Approaching impossible.

The one thing that will sink a project faster than too many cooks with not enough talent is poor requirements at the outset.  If you truly don’t know what you’re building, you’re doomed.  You need to see pretty clearly what it is before the first line of code is written.  If you must build a prototype to get there, consider it time and money well spent.

By the way, creating a DSL for your domain is a way of creating a formal specification.  It takes rigor to create a language.  Hand waving won’t get you very far.  Also note that a giant laundry list of use cases and feature requests is not a specification.  It’s a wish list.  You need to understand the commercial dynamics of what really matters to customers, why it matters, and have a pretty good sketch of how you’ll solve their problem.

Don’t kid yourself that a giant team running around gathering requirements and hacking out use cases in monolithic modules is building software, business or otherwise.  It’s creating an unmaintainable mess that makes nobody but the IT guys that assembled that laundry list happy.  Certainly the business uses who it lands on are going to find it isn’t what they thought they were getting and it is almost impossible to change it into what they wanted.

So then why do small teams grow large?

This is my own editorial, but I’m surprised it isn’t talked about more.  We’ve all seen teams start small and grow large as a product succeeds.  Why does this happen?  The prevailing wisdom seems to be that as you gain customers, their demands for new functionality outstrip what the small team could provide.  I don’t believe it.  I’ve looked at a lot of software releases, and by release 3 or 4, it’s all too easy to get caught up delivering much sound and fury signifying nothing.  Did that release 3 or 4 really turn out that much more functionality than 1 or 2?  Was it really revolutionary?  Were the features all of the same caliber as the original?

No, I don’t think so.

Another source is failed negotiations between VP’s of Engineering and CEO’s.  Most CEO’s do not understand creating software.  They come from fields like Sales and Marketing where throwing dollars and bodies at problems can make a difference.  So they want their VP of Engineering to do the same.  They don’t want to hear the team is running flat out and can’t produce more.  They want what they want when they want it, and they’ll write the check to make it happen.  I have not personally run afoul of this (I am stubborn about agreeing to do something I know will fail), but I have heard of it.  This is not the big issue, though.

In my experience managing multiple release cycles for something like 50 products, team growth almost always boils down to aspirations and career growth.  People get tired of working on the same code base.  They get tired of working on the boring parts of the code base.  They want new challenges.  They want career growth.  They want to be architects and managers.  Who can blame them?  It’s human to want these things.

Pretty soon, a person who was a senior developer is an architect or a manager.  We carve some subsystem off and hand it over to them.  Or, we hire someone incredibly junior to be in charge of some unimportant code that the senior guys want nothing more to do with.  A fiefdom is born and the big team is on it’s way.  This is a mistake!

We are better off to create entirely new products as a path for career development.  If we can’t justify a product, or worse, can’t justify giving a person a product, then we need to be honest about that.  You may lose the person as they seek opportunity elsewhere, but you may also need to get some new blood into your small team.

This points up something to look for in your hiring practices.  Beware too much naked ambition among developers.  The guys that push for promotion every year can be awesome, but they will be high maintenance, and they will try to push you into doing something that’s bad for the team and perhaps even bad for themselves.  Beware especially the talented developer who wants to move into management because they want to make the decisions on architecture.  The developers who are in love with the act of creation and who have great chemistry with their coworkers are the gems.  They will not push you for much more than clear and exciting direction on where to take the product next.  Don’t exploit these guys either.  Reward the heck out of them.  After all, it doesn’t take too many to work miracles.  Make sure they’re happy, whatever it takes.

Give me a few guys like that who are really good and there isn’t any piece of software I can’t get built faster and better than the big team.  We’ll also keep that software vigorous and cutting edge for years too, and we’ll run circles around the competition.

What is Talent if Not Good Design for Programmers? (Talent is No Myth for Programmers)

I love the Discipline and Punishment blog, but he’s taken a left turn at Albuquerque when he talks about the Myth of Talent for Software Engineering and then goes on to say:

What is the equivalent of “capital” in programming? It’s not ‘talent’ in any meaningful sense. Rather I’d suggest it’s what most people call “good design.” What’s really paying dividends in any large project is good design simply because design adds value faster than it adds cost. Good design, purely because of its “compound interest”-like nature, is largely what drives success in software and what separates successful projects from unsuccessful projects. The many software projects that fail (really they are abandoned) inevitably do so because of bad design.

I absolutely agree that it is the design and not the languages that matter most, but how else do we measure talent among programmers if not by good design?  Who do we look up to most among programmers we know?  It’s the Uber Architects.  These are the ones that have the good design ideas.  Yeah sure there are those guys who crank out devilishly clever but unmaintable code.  In the old days, they were called hackers, back before that term was a good thing.  It was a term of derision.  Great software engineers were the ones with great designs. 

Let me provide yet one more reason why talent matters.  If you really do believe it takes fewer people to write good code, and there is an overwhelming body of evidence to support that, how could you not also believe that if you have only a few slots on the team it means that talent matters even more?

In struggling to understand whether I’ve misunderstood the original post, I almost get a sense that the wrong meaning is being attributed to the word “Talent” when I see terms like “Rock Star Programmer” being thrown in as bad examples of “Talent”.  There are certainly situations where the talent label is misapplied to someone who in fact, does not have the talent.  Don’t blame that on the idea that talent is valuable! 

I sense a little bit of tendency to feel that the design is handed to programmers, and hence that’s why design and talent are separate.  That’s a bad idea!  Design is a participative process under the watchful eye of the right benevolent but fascist dictators.  A good dictator creates an overall framework that leaves a ton of freedom for individual programmers to express their own design sense in their corner of the world.  A programmer who just wants to be handed all the design detail on a stone tablet so he can “write code” is not someone I want to hire or work with on my small teams.  The guys I want will leave if they don’t get a hand in design.  You will understand a design so much more fully if you participated in creating it.  By the same token, I hate the idea of architects that never write a line of code because they’re too busy thinking of great architectures for someone else to build.

There is also a certain uncomfortableness with the idea that talent implies really great programmers are born and not made.  Let me lay that one to rest too.  I’ve hired hundreds of programmers to work on really difficult software in small teams and I have seen overwhelming evidence that great programmers, programmers with talent, are very definitely born and not made.  We’ve built languages and tools of all shapes and sizes, database servers, desktop application software, web software, data mining, genetic algorithms, and a host of other stuff ranging from the mundane to the completely weird.  I’ve worked with extremely famous programmers at companies like Borland, and I’ve worked with brilliant folks who are obscure.  The one area I don’t have experience with is games, but I think that world believes you’re born and not made too. 

Throughout those hundreds of programmers hired, only one pattern sufficed to hire more great programmers.  It had nothing to do with age or education.  It had nothing to do with giving them clever tests in the interview process.  It had to do with whether they’d ever done at least one great thing or not, and whether they could communicate that thing well enough to expose some native design sense.  I’ve seen guys with fantastic MIT or Stanford CS degrees (including graduate degrees) that could not program their way out of a paper bag.  I’ve seen high school dropouts that started programming relatively late code circles around the average CS grad. 

I’m sorry folks, but talent matters, talent is the ability to design software systems and algorithms, and you either got it or you ain’t.

Serendipity is the Key to Code Reuse

In watching yet another ping pong game of compiled versus dynamic languages, I found myself mostly siding with the dynamic language crowd in the form of Steve Vinoski and Patrick Logan.  This time around the discussion was all about what to do with “average programmers”.  It all got started when some others posited that dynamic languages and RESTful architectures are for those who view progamming as art while statically checking the contract, whether speaking of languages or SOA’s, is what those who want to view programming as engineering should do.  I don’t want to get sidetracked by all of that background, because I will write about it some other time (is programming art or engineering, what to do with average programmes, yada, yada).  For this post, I was struck by one of the linked in tributaries that make the blogosphere such a powerful idea generator because it triggers that sudden tunneling of ideas between relatively unrelated spaces that sparks creativity.

Specifically, Vinoski mentioned almost completely in passing that REST is leads to serendipitous code reuse.  He provides just the one word, serendipity, with the link I’ve added to the word.  Well I can’t resist diving down a good looking rabbit hole when I see one, just to see what else there may be, and sure enough, it was worth the trip.  Stuart Charlton has a neat little post on serendipity and code reuse that crystallized some fuzzy thinking for me.  Stuart is an Enterprise Architect for BEA, which seems to me ought to predispose him to be much less enlightened, but being an INTP, I guess he couldn’t help himself blurting out some good stuff about REST.  Here is the goodness of Stu:

One problem with SOA is that it is very “heavy”, with a partial focus, like CBD before it, on planned reuse.

In some industries, planned “product line” reuse has been shown to work, such as with car platforms. It’s also appropriate for very general purpose programming libraries, etc., and could also be appropriate in software (there’s a fair amount of “software product lines” literature out there).

From this viewpoint, “build it and maybe people will use it later” is a bad thing. SOA proponents really dislike this approach, where one exposes thousands of services in hopes of serendipity — because it never actually happens.

Yet, on the Web, we do this all the time. The Web architecture is all about serendipity, and letting a thousand information flowers bloom, regardless of whether it serves some greater, over arching, aligned need. We expose resources based on use, but the constraints on the architecture enables reuse without planning. Serendipity seems to result from good linking habits, stable URIs, a clear indication of the meaning of a particular resource, and good search algorithms to harvest & rank this meaning.

This led my own little ESTJ overly top-down and logical mind to instantly make a list or continuum of choices surrounding code reuse, SOA, REST, and dynamic languages:

1.  Preplan everything and rigidly enforce the contracts:  Heavy SOA.  No hope for serendipity.

2.  Expose thousands of services in hopes of serendipity.  Lots of work if you use SOA-style mechanisms.  That’s why SOA proponents say it never happens.

3.  Use a lightweight protocol that exposes thousands of services almost for free: REST, we love you!

4.  Expose a language that can create any service as needed.  Whoa!  Where did that come from?

I’ll admit, #4 popped up unbidden.  In my defense, I have a sort of “Warfield’s Law” when it comes to computers that leaps to mind more frequently than it should:

All things become languages if they’re important enough.

If Adobe can make a language out of printing, I don’t see why this doesn’t merit a language too.  The Law is true too.  Computers do one thing especially uniquely: they are Turing machines.  Languages are their manifestation of that.  If it isn’t a language, it could be done by a toaster at some level and doesn’t need a real computer.  Conversely, if you don’t make it a language, you’re robbing yourself of the full power a computer can offer.

Getting back to the problem at hand, what could be better than a dynamic language for enabling serendipitous code reuse?  We can reorganize available code assets under the latest DSL of the day to solve an entirely new problem.  In the grand scheme of these distributed computing conglomerations, REST becomes the sort of assembly language for these DSL’s.  It gives us the universal impedance match we need to hook up our components.  The dynamic language massages that somewhat generic socket into something more recognizable and powerful for the particular domain we want to conquer today.  Tomorrow, we’ll build a new one, and it’ll be easy, because we have lots of RESTful components lying about and a nifty dyamic language that’s potent at composing them.

Perhaps I’ll run this thing on a Pile of Lamps while I’m at it.  I say that only partially in jest, because the Pile O’ Lamps crowd are speaking my language, at least in terms of making things into languages.  Aloof Schipperke writes that he sees SPOTs when thinking about the Pile of Lamps.  He goes through an abstraction exercise that got me thinking again.  Here is the original Lamp:

• Linux
• Apache
• MySQL
• P{HP,erl,ython}

Aloof says, “Wait, it’s more like this if we go up 10,000 feet”:

• Scripting Language
• HTTP
• Operating System
• Database

Do you see Warfield’s Law (everything becomes a language) at work yet?  What did the Scripting Language replace for many architectures?  The middle tier.  The application server.  Devilish complex, painful to operate, often extremely inefficient.  Beans of every variety abound.  J2EE et al ad nauseum.  Why put up with it?  Replace it with (drumroll please!):

A Scripting Language!

Now can we successfully combine all of these ingredients?   

• Component Bus for Reuse and Communication:  (RESTful or similar) 
• Scripting Language
• HTTP
• Operating System
• Database
• Utility Computing Infrastructure

That seems to me a potent next generation architecture for web software. 

Replacing pieces of code, framework, or whatever with a language is an extremely powerful chess move.  It’s turning a pawn into a queen.  It’s not easy to do, as most are not facile at creating languages.  But why bother, there are lots of nifty dynamic languages to choose from.  Now you’ve got another excuse to go look one up.