SmoothSpan Blog

For Executives, Entrepreneurs, and other Digerati who need to know about SaaS and Web 2.0.

Not Everyone Can Program and Not Every Programmer is Great

Posted by Bob Warfield on December 2, 2007

Reg Braithwaite wrote a column recently called, “The Optimistic View”.  He’s tired of organizations pushing lowest common denominator tools because they’re afraid lowest common denominator programmers will hurt themselves (well, hurt their code really) with stronger tools.  He suggests:

But maybe, just maybe, most programmers rise to the occasion when challenged. Maybe instead of catering to the bottom 10th percentile we can be open-minded about what the 50th to 75th percentile are capable of achieving, much less the top quarter.

It’s an interesting thought, but I confess I haven’t seen an awful lot of evidence it’s true.  I have certainly been surprised by programmers before.  But the surprise has been in just how fast a really talented “junior” programmer can take their place in the ranks of the really good architects.  I have not seen a lot of evidence of the much less talented stepping up to deal with some of the real rocket science of our trade.

He gets a little more detailed in his thinking here:

Spreadsheets taught us that anyone can program. Maybe we should open our minds to the possibility that although not everyone does program in this manner today, they are capable of it if we stop telling them that they aren’t.

And when we say “Anybody can program,” we say that without reservation or disclaimer. We don’t mean anybody can program provided they are banned from using certain langauges, or anybody can program provided we hide the details of relational databases behind ORMs, or anybody can program provided they are not required to learn any idioms beyond those authorized by the local High Priest. We mean anybody can program, period, without being lashed to a heavy yoke and driven through the fields under the whip.

This is the point where we part company.  Spreadsheets did teach us that anyone can program, BUT, they did so by doing some of things he suggests we quit doing with less talented programmers.  They hide lots of details that one would have to know to accomplish the same thing using normal programming.  It is quite instructive to think about what spreadsheets do for us.  Largely, they allow us to quit thinking abstractly and organizationally, two big skills that programmers must possess.  In other words, they eliminate the need to think like a programmer. 

Consider the business of organization, or specifically of ordering.  Spreadsheets let their users examine the trees and the forest just emerges magically, and without much planning.  Tell the spreadsheet the relationship between a few cells using a simple formula.  You never have to worry about what order you do that in.  The spreadsheet does a topological sort and figures out the “natural order of recalculation.”  Because of this, there are no loops to think about.  Control structures are pretty straightforward and are seldom nested.  A simple “IF” is about all there is, and easily 90% of spreadsheets don’t use them. 

There’s a lot more clever hiding that happens as well.  Consider the whole user interface issue.  Spreadsheets give you that for free using their grid metaphor.  It’s amazing what can be accomplished there, but it isn’t really programming.  Perhaps most importantly, it doesn’t try to be.  Most of that look is fixed at design time and is not dynamically programmable. 

Data is another biggie.  For those who are stuck on static typing, spreadsheets have none, nor would it make them any “safer” to offer it.  But there are also no data structures.  Each cell stands more or less alone, with the exception of ranges (or blocks as they came to be called) referenced in formulas.  There is almost no ability to go further save Excel’s confusing array formulas which I’ll wager are used in much less than 1% of cases, and most of those were forced as a way use the LINEST builtin.

The interesting questions are how much of this can be done for conventional programming and how would one go about it?  The first issue is seriously missing.  It’s not clear we understand and agree just where the problems are.  I’m certainly not going to claim to lay it out here, but I will say this.  Layering more and more complexity on is not the answer.  That’s not how spreadsheets got where they are in terms of enabling more programming.  Adding a bunch of syntax around typing may have value for some things (chiefly making the tools better in my mind), but it isn’t how you help people.  If you want to help lowest common denominator people, figure out how to eliminate or radically reduce complexity.  Eliminate abstraction.  Eliminate the need for the peculiar kinds of thinking great programmers must do.

Spreadsheets, BTW, are a graphical dataflow paradigm.  They really are visual programming, just not using boxes and arcs.  They are also constraint oriented programming.  I could go on, but spreadsheets don’t prove everyone is a programmer.  They do something quite different that is even more useful: they make it possible to write programs even if you can’t think like a programmer.

About these ads

9 Responses to “Not Everyone Can Program and Not Every Programmer is Great”

  1. dsickles said

    The spreadsheet cell is MVC all in one.

  2. […] Not Everyone Can Program and Not Every Programmer is GreatBy smoothspanWe don’t mean anybody can program provided they are banned from using certain langauges, or anybody can program provided we hide the details of relational databases behind ORMs, or anybody can program provided they are not required to …SmoothSpan Blog – http://smoothspan.wordpress.com […]

  3. Ray said

    Seems similar to “everybody can paint”. Sure, everybody can put colour on a blank sheet of paper. But to produce art, it needs an artist.
    Someone who’s able to understand the background. All of it.
    When you do, the paper will disappear and the colours will be replaced by thoughts.

    In my opinion, there are very few people who can really program.

  4. sofal said

    After reading this post, I have to say that I don’t think you and Reg actually part company. I think you’re both agreed that good, clean abstractions can be extremely helpful in terms of reducing the cognitive distance between a human and a computer. The concepts that Reg refers to (folds, metaprogramming, closures, expressive typing) only add complexity in the same way that regular expressions add complexity to string parsing. Once these concepts are understood, they can be used to hide more complexity than they introduce.
    Unnecessary complexity, the kind that doesn’t give any net benefits, is added when, as Reg illustrates, we are “lashed to a heavy yoke and driven through the fields under the whip”, or in other words, when we are forced to use only the language constructs and abstractions that a novice would understand, for the sake of having a code base that is supposedly save for novices (a false conclusion in my experience). The process of creating a spreadsheet isn’t immediately intuitive to the user who has not been exposed to it before, but once the user learns how to create cell dependencies and such, a world of power opens up for them.
    Unnecessary complexity can also exist when moving down the abstraction continuum (as you point out here). But I think that Reg would agree with you there.
    It may be true that given the right amount of abstraction, anybody can program, but we need to be careful about what we call “complexity”. Things such as closures and metaprogramming are not rocket science and they do eliminate complexity.

  5. smoothspan said

    Dsickles: cool!

    Sofal:

    “The concepts that Reg refers to (folds, metaprogramming, closures, expressive typing) only add complexity in the same way that regular expressions add complexity to string parsing. Once these concepts are understood, they can be used to hide more complexity than they introduce.”

    This is the question, isn’t it? Do these things ultimately hide complexity for LCD programmers? I have doubts, but perhaps I am wrong. Wouldn’t be the first time!

    Your example, BTW, string parsing, is an area where the LCD programmers I’ve seen don’t do any better with or without regex. Programmers seem to get parsing and formal languages or not. It’s a ticklish area, and was a more effective weed out course in my comp sci school days than programming 101. The second big weed out course was algorithms.

    The real point I’m making is that if you literally want everyone to program, we’re nowhere close to talking about the right stuff for it. Most programming languages are all scrunched up against one end of the continuum and we want to be at the other end entirely. My guess is that if we were to solve the problem well, it would also make real programmers fantastically more efficient too.

    Cheers,

    BW

  6. Indeed, that is the big question — do abstractions make complexity better or worse? Those of us who understand and regularly work with a particular abstraction mechanism always see it making things easier, often without realizing that those who don’t understand the abstractions think (perhaps mistakenly) that we’re doing something really wizardly and hard.

    I’ve worked with programmers who complain that I’ve made code too complicated by factoring the problem out into several different classes. From their point of view, it was complex because there were more files to look at and try to figure out how they went together. I felt that my design was less complex because each file/class was small and handled a specific responsibility that could held in one’s head easily, while deferring certain gory details to other classes that have a specific responsibility to deal with them. These programmers, however, would have preferred to see it all in one place — but my opinion was that such a piece of code — for example, a single function that would initiate the database connection, construct the SQL, send the query to the database, massage the returned data into another data structure — was my idea of needlessly complex. You either get OO (or for that matter, something as old and crusty as structured programming — I’ve worked with programmers who could barely factor a new function out of something, let along a new class), or you don’t, in which case, my suggestion is often, either get back to me when you figure it out, or switch professions.

    String parsing, regular expressions, same deal. Many programmers believe regular expressions to be complex, but my idea of complex is a page-long tangle of nested loops and substring comparisons.

    Perhaps some kinds of abstractions are easier to grasp than others, and some abstractions even enable other easier ones. Consider the graphical “regular expression builder” tools some programmers use. Regular expressions are easier to generate programmatically than the aforementioned tangle of loops. Also, tons of web developers use Flex Builder’s data binding without giving much of a thought to the fact that they are a syntactic abstraction for event dispatch, which is itself an abstraction of the Observer pattern.

  7. extrapreneur said

    I totally agree, you have to work hard to be the best and not everyone can cut it.

    Check out my blog at http://www.extrapreneur.wordpress.com

  8. We are about to see the reality of your title as more and more programmers try to write applications for multicore machines. It is just too hard for normal coders to do.

    Your old associate Anders Hejlsberg stated recently “only the top 10 percent of programmers can adequately program for concurrent environments and it is not something for the masses.”

    Bill G. his-own-self says “One approach to handling concurrency is to write multi-threaded programs that allow data to travel along many paths. But as any developer who has written multithreaded code can tell you, this is one of the hardest tasks in programming.”

    Dr. Tim Mattson of Intel states “If we ‘experts’ can’t agree on how to write a parallel program, what makes us believe parallel programming is ready for the masses? In our quest to find that perfect language to make parallel programming easy, we actually harm our agenda and scare away the software developers we need. We need to spend less time creating new languages and more time making the languages we have work.”

    You are talking about string parsing as a challenge!

    In the spirit of full disclosure, our Java framework is our attempt to make this far easier for Java developers by using the dataflow approach, as seen here: http://en.wikipedia.org/wiki/Dataflow.

    steveh
    http://www.pervasivedatarush.com

  9. smoothspan said

    LOL, Steve, believe me, I understand that true distributed computing is another level of complexity yet that even fewer developers are up to. There are so many issues and tradeoffs involved that I wouldn’t even begin to get into it on a comment thread like this. I know you’ve seen my various posts on the multicore crisis as well.

    Cheers,

    BW

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

Join 325 other followers

%d bloggers like this: