Software Development: Why is it Such a Mystery?

When tackling a software project, most people have difficulty grasping exactly what needs to be done and how they are to go about doing it. Software is intangible.  It is hard to see the inner workings, the bits and bytes floating through computer chips and making things work.  We see the user interface but that only gives us a minute glimpse into what it really takes to make computer programs go.

The same could probably be said about a building.  There are all sorts of things going on that we do not see.  There is plumbing and electrical, heating and cooling, plaster board and wall studs. However, people seem to have a greater grasp of what goes into a physical structure and all the work it takes to build them.


What is it that that we do not see in software that we do see in physical construction?


A physical structure is tangible

We can see and touch a physical structure.  Numerous books and TV shows guide us through the construction process for buildings, roads,bridges, dams and so on.  It is easy to describe with pictures.  It is easy to visualize in our minds.  We can quickly see the size of a structure and size often translates to cost and complexity.

In the world of computer software things are different.  There is very little to show the construction process for software.  Frankly, I cannot imagine a TV show or book that would walk a person through the actual build of software.  To me it seems it would be dry and rather boring, even to a software engineer like myself.  There is lots of cryptic text.  And the best applications do all sorts of complex things while appearing to the user to be incredibly simple (we call this good usability or good user interface design).  In fact, the simpler a program is, the more time and money it probably took to create it, the opposite of what many people would expect.

I like to think of it as it was described in the movie I Spy. Size matters, but in the spy world (in our case the computer world), it isn't how big it is, it is how small it is that is important. Impressive buildings are large.  Impressive computers and software are small and/or simple.


We SEE and can feel the inner workings as they are assembled

Human beings have 5 senses.  We see.  We feel.  We hear.  We smell.  We taste.  The first two of these is easy to experience during the construction process for physical structures.  I guess you could argue the first four but the first two are the most important in grasping the scope of a physical construction project.  On a regular basis we pass by construction projects as we drive to and from work or the grocery store.  We see how long it takes to finish a project and see what is added as the project unfolds.  We are exposed to the construction process on a regular basis without having to even be a part of it.

Software is completely different.  In general, people only see the finished or beta product.  Software engineers and developers build away, hiding the process from the world until one day, whala, the product is released.  This is great in several regards.  It allows construction to commence without revealing the work to competitors until it is done.  It keeps ongoing construction from adversely affecting peoples' ability to use the current product.  And it increases our ability to try things out in testing environments before unleashing the product on the rest of the world allowing us to experiment more without having to define every last detail before we begin building (sometimes called prototyping).  The down side is that people never see what it takes to build the software and have little to no idea what goes on.  The general public has no way to gauge how much time or money it really takes to build software or what the individual components even are that make it work.


We SEE and can feel the inner workings when maintenance needs to be performed

Individuals have constant exposure to the inner workings of physical structures. We have to deal with a blown circuit breaker.  We have to fix that leaky pipe.  We have to work on or call a maintenance person in to fix the squeak on our noisy furnace and see what is wrong so it doesn't break down.  We rewire our homes to put in a new light switch that is more conveniently placed.  And by doing the work or seeing someone do the work in front of us we get a grasp and appreciation for what it takes not only to build but maintain and improve those physical things.

In software we prize hiding the complexity from view.  The best software empowers us to do a lot with very little effort.  With the Internet and web sites and services, we now consume those services which are built and maintained out of sight and out of mind.  The less we have to deal with it and the more automatic it is the better.  This puts us even further out of touch with what it takes to create this automation.  Quite often people say, "All I have to do it click this button and it is done for me.  How hard is that to build?"  But the simpler it is for the end user, the more complex, time consuming and costly it is for the software engineer.


Physical constructions is well established in our minds

From childhood we experience the construction process.  Daddy is building that new play set and painting the favorite cartoon characters on the wall.  Not a week, maybe not even a day goes by without us seeing some form of construction or maintenance go on around us as we are at home,visit places of business or drive down the road.

This is changing some as more and more kids are exposed to computers from a very early age.  However, there are many people who have not had that luxury.  Furthermore, using a computer program is all most people generally see and do.  They still do not see the construction process for software.  We software engineers hide it away until it is ready then like magic it just appears and is so easy to use that it must be that easy to build right?

Many business books describe this misperception in business.  People talk about the overnight success.  Sam Walton's autobiography talks about spending decades to build his business to one day being labeled an over night success story.  Twenty plus years is a long night!  It is because it is out of sight and mind until pow, it jumps up in front of your face.  It is easy to think something was suddenly, quickly and easily created when you didn't experience the hard work that it took to get there.


Physical construction techniques and components are more mature, refined, standardized and regulated

Physical construction has been around for thousands of years.  We have learned what works and what doesn't.  Whole engineering and architecture schools exist to teach what has been learned and refined over those thousands of years.  The industry has placed numerous standards for construction, safety, environmental considerations, etc.  Doors are generally a certain size.  Bathrooms have standard fixtures.  Many components are just plug them in and they work.  Yes, sometimes it takes a trained professional to install them right but the person creating the wall knows how far apart to place the joists and what size to make the door frame and the door is installed pretty much the same way every time.

Software engineering has been around for less than 100 years and Internet development for not even 2 decades.  The software industry is in its infancy.  We strive for standards and plug and play components and have made some great strides in that direction but there is so much more to learn and improve upon than has been completed.  Different companies and groups still debate over what standards are the best.  In short, we still have a lot of maturing still to do.  As we do mature,we are able to do more, better and faster.

The hard part in all of this is educating the general public about what it takes when embarking on their software project.  We commonly hear people say they thought their project was simple.  "All that needs to be done is..." is commonly said.

The best approach, or at least what has worked well for us, is to keep the dialog open, break down the tasks for our clients and show exactly where the work is going to happen.  When a client sees every little thing that has to happen on the project, they gain a greater respect for what they are asking for.  They also get an opportunity to descope the project to help control costs and get the application to the market quicker.  These detailed project plans also help us get more accurate estimates, do good capacity planning and keep track of what needs to be done and when to ensure an on time and on budget project.