Part2 of my follow up from Cusec2010, regards Douglas Crockford’s talk on “The Software Crisis”.

Short version for those headline skimmers:  Crockford says that the Software industry is in a state of crisis, we need to fix it and we must fix it.  I agree with all his recommendations and examples of flaws in our industry, but I argue that it isn’t  necessarily a flaw but an inherent relationship between the software industry and the business world.  Buggy software is often good enough, so lets just try to make it a bit better.

As a precursor to this talk, for the last few months I have been thinking about a similar (if not the same) problem: The software crisis. I had been talking with friends about something that bothered me about the software industry, imperfect code, imperfect process, the imperfect software industry.  When Crockford stepped on stage and showed his first slide “The Software Crisis”, without him even continuing, I knew exactly what he meant.  First off “Thank you Douglas for putting a name to an issue for me”, it was like remembering the lyrics to a song for a week without being able to remember who sings it.   His talk was on aspects of the crisis, why they exist and how we can minimize and pave a brighter future.  He had my attention….

I won’t go into the dirty details of the crisis itself (a quick google search should help you with that), but in short: we are getting lazy, we are producing bad production code, we are delivering late, we are testing too late, we are in major trouble.  Why does this happen, and what can we do?  Crockford opened by stating that creating software is the most complex task a human being can do. (Paraphrased, someone please msg me if you can remember the exact quote)

Woah, he didn’t even take a breathe before continuing.  Back up please.  While I’ll agree saying that in a room full of young, eager software engineers will get people excited, it is quite a bold statement!  I didn’t particularily like this statement, mostly because taken literally, it is pretty upsurd.  “Creating perfect software is the most complex task a human being can do” might be more acceptable, but no less obsurd.  “Ian Chan makes the best veal schnitzel in the world”, is true, but in the same category of absurd statements,  but I digress.  Our problem domain is hard, moving along…..

Douglas’ lecture continued by showing examples of how we can improve our industry, and things that we should do to make our software better.  This lecture, coupled with Greg Wilson’s talk on “bits of evidence” are all lessons truths about our industry that I am glad people got to hear.  It is unfortunate that we need smart people like these to lecture others about what we are doing wrong, just to get people to realize we are doing it wrong in the first place.  (For most of the CUSEC crowd, it was more of a pre-emptive strike).   See here for full notes on Crockfords advice, a great summary by Joey deVilla.

Crockford provides an abundance of information about existing problems, and causes of the crisis as well as great practices and solutions that we need to live by.  For me the summary of the presentation was that given these problems, and given our options, we can collectively work to eliminate the crisis and create a better software industry (forgive me Douglas if I over simplified your thesis).  This is where I did not buy it. There is a big difference between “best practices” and “in practice”.  Surely Crockford would agree to that.  My rebuttal to his informative yet over optimistic set of ideals:

The software industry isn’t in crisis because we refuse to follow best practices and learn from our mistakes; The crisis exists and we continue to produce sub-par deliverables because the industry allows us to.

Back to Crockford’s original claim about the difficulty of creating software.  Creating a flower-shop website is not ground breaking science.  Yet it is still in the domain of software development, weather you want to accept it or not.  The flowershop owners do not care about software best practices, though they would like a nice and affordable product.  Scale it up to a slightly larger problem, the Point of Sale software in your favorite bar, likely cost thousands to produce.  It isn’t perfect, there are probably bugs, it probably went over budget and missed deadlines.  The bar tender doesn’t care.  The bar owner doesn’t care.  No one who uses that software cares if it was code reviewed, or if went through regression testing each time someone touched the build.  The business was to produce a POS interface, and it was delivered.  If the product does more or less what it was supposed to, the customer will likely be happy.  Software projects often go to the lowest bidder.  The team or shop that can turn around the project in the shortest amount of time, and with the lowest price tag.  I am not going to hire Fog Creek Software (sorry Joel!) to make me a flower website when I have a $1,000.00 budget.  Even though I know they will do a better job.  Sometimes it is more financially viable to build “ok” software than “great” software.  And yes I am aware that bad code comes with maintenance costs and inherent long-term “software dept”, but many projects don’t even make it to the maintenance stage, or factor that into the cost of building the first iteration.  At the end of the day the guy writing the check has no clue what Software Engineering is (unless you are lucky).

I spoke briefly about this issue with Douglas after his talk, and he reminded me: “Well, now you are talking about negotiation”.  Well, yes, that is exactly what I am talking about.  Arguably a good development team can work just as fast as a ‘bad’ team producing inferior software.  I hope that a ‘good’ team charges more than a ‘bad’ team.  One must then build a reputation so that a client will trust their price over the others, with that trust relying on experience and assurance that a good product will be delivered on time.  If a good team doesn’t charge more than a bad team, then what is the point?

Morally, as a software developer, I feel obligated to fulfill all of Douglas’ suggestions, and I will.  To knowingly produce anything but perfection is nothing but a compromise.  Each time we say “I don’t have time to test this component”, or “We need to make this deadline so just code it quickly”, or even “fix it when it breaks”, we are doing a great disservice to our industry.  Not just because we lower the standards of the industry, but because we lower the expectations of customers and we reinforce their understanding that software development is NOT a complex task.  So back to Douglas’ talk,

I believe there is a need for people in the industry like Douglas who advocate a better industry and a solution to the crisis.  I don’t think the world needs a solution to the problem because for most non-life-critical systems, it doesn’t have to be perfect.  There will always be development shops on the cusp of best practices and shops whose software always breaks a year later.  This is a reality, and not something that will go away.  Look at the auto industry, where the quality varies dramatically from company to company, yet year after year people continue to buy inferior yet affordable products.  I don’t really see software being much different.  The best we can hope for is that we work for Mercedes or Porsche and not a <insert American car manufacturer here>

Thomas Ptacek did a talk on software security also at CUSEC, reminding us that no matter what, there will always be security flaws and the best we can do is minimize them.  He gave an anecdote of  security expert Dan Bernstein’s qmail.  qmail stood an almost impossible 10 years without a known security vulnerability.   Crockford would have to agree that Bernstein was setting a good example in this case.  But as an audience member asked Ptacek: “So are we all doomed or do we have a chance?”  Ptacek answered answered so eligantly:  “There are not enough Dan Bernsteins in the world, we are all doomed”

I can’t end on such a sour note.  I really enjoyed Crockford’s talk.  But the main thing I think he missed (but likely knows) is the he is a Dan Bernstein himself, and he is preaching [updated: bad choice of verb] encouraging people to be more like him.  It is necessary, especially to students, but (unfortunately) not everyone is in that school of thought, and that passionate about good software.  I’m sure Douglas will always have a job, and usually land the juicier contracts.  But he is only considered a “good” developer because there exists so many bad ones.  So let’s try harder, let’s make better code, let’s make fewer mistakes, let’s learn from our mistakes, but more importantly, let’s just make good software and move on with our lives [removed that last part, sounded really bad since software is part of my life atm]

Advertisement