One of the most enduring myths in software systems development equates it to the building of a house or similar structure. While a seemingly useful abstraction to express what parts of a solution we're creating (eg. foundation, framework, architecture), it really obscures the actual nature of the work involved by implying that engineering and software projects share a common basis in how they are planned, designed and delivered.
This false equivalence has in turn given rise to the fallacy of division of labour in software projects which presupposes that design activities (eg. requirements gathering, analysis, architecture) can be divorced from construction activities (eg. programming, coding, packaging) thereby enabling the creation of predictable delivery schedules according to team size. In other words, according to the John Heywood proverb, "many hands make light work".
Unfortunately, tasks in software projects do not divide as cleanly as they do in civil engineering projects like house building or widget manufacturing while assuring similar precision. Consequently, there is a rather significant mismatch between the reality of software development activities and the current plan-driven, single-pass/phased methodologies currently used to organize them. In this article, I examine the root causes of this mismatch.
The Default Model: Division of Labour in Engineering Projects
Much of our so-called "modern" ideas about the efficiency and effectiveness of IT project delivery borrow quite heavily from engineering and manufacturing disciplines - which is problematic because the nature of the work, as we'll see, is entirely mismatched to this type of organization. I call this the "Default Model" because it represents the status quo that is found in many IT and software shops.
Under this model, design and construction activities are separated to create a cost-effective, predictable schedule that can employ people with lower skills to build the end-product. Highly-skilled (and therefore expensive) designers and architects create "instructions" that can then be handed off to lower-skilled (and therefore inexpensive) workers to "construct". In terms of effort and cost, this tends to break down as shown below:
In a traditional civil engineering project, design activities account for the minority of overall effort and cost while construction activities constitute the majority.
This model presupposes that the delegated tasks undertaken can be perfectly partitionable with no required intercommunication between workers or partitionable with intercommunication between workers. In his 1975 landmark book, The Mythical Man-Month, Fred P. Brooks Jr. described this relationship between workers and months thus:
Men and months are interchangeable commodities only when a task can be partitioned among many workers with no communication among them… In tasks that can be partitioned but which require communication among the sub-tasks, the effort of communication must be added to the amount of work done.
In the first scenario, think reaping wheat, splitting wood or putting nuts and bolts together - these tasks can be divided evenly across a labour force without cross-communication while providing the requisite precision. Henry Ford famously exploited this concept through the use of automated tooling machines that created parts which could be easily assembled by workers into finished products.
In this case, effort effectively becomes a multiple of the men added to a project because there is an approximately one-to-one exchange:
Credit: Fred P. Brooks Jr.
In the second scenario, sub-division of tasks can only be accomplished through intercommunication between workers, creating a "drag" effect on productivity due to the effort required to ramp-up additional workers on tools, technologies, techniques, etc. This blunts the impact of adding workers to shorten schedules:
Credit: Fred P. Brooks Jr.
The time (and thus cost) required to "ramp" and coordinate additional team members comes in pairwise multiples. To illustrate, consider the following cross-communication between two team members:
Now, the same cross-communication that may need to occur between three team members (3x Single Pairwise Communication):
And now with four team members (6x Single Pairwise Communication):
The Fallacy of Division of Labour in Software Projects
In his 1992 essay for C++ Journal, What is Software Design?, Jack Reeves makes two critical observations that effectively blows apart the view of software projects as analogous to engineering projects:
• First, in software, construction is so cheap as to be free;
• Second, in software, all effort is design
Software is "constructed" through the use of automated developer tools that convert source code into machine language and package it for installation on target systems; "design" is performed by a team of highly-skilled (and often highly-paid) developers, business analysts, testers, UI/UEX designers and architects throughout the lifetime of the project. Given that I'm mostly directing this article toward business owners and managers, let me underscore this point:
In a software project, construction is performed by machines; design is performed by the entire development team over the lifetime of the project.
This has a significant impact on the divisibility of tasks in a software project and by extension how the project can be planned and executed. In the default approach described earlier, the dual nature of the effort involved makes the separation of design from construction possible. With 90% of the effort involved in physical construction, the activities readily lend themselves to efficient division across a workforce. As the project takes on added technical complexity, however, intercommunication is required to enable other workers to assist.
In a software project, men and months are not interchangeable commodities because the work involved is by definition technically complex, requiring corresponding interrelationships between team members. The effort required to communicate knowledge to ensure the necessary precision for sub-dividing tasks quickly overtakes any advantages of having multiple workers and further makes it impossible to control schedule or cost through the simple addition of manpower:
Credit: Fred P. Brooks Jr.
This realization is known as Brooks Law: Adding manpower to a late software project makes it later.
This point confounds traditional phased software delivery projects that divide their teams according to specialization and phases of activity with the expectation that men and months are interchangeable and can be done so without much consequence. The necessary interrelationships are weak with infrequent cross-communication between team members in different phases where it would be most beneficial. Activities even within the same phase become siloed according to specialization, leading to the most injurious and risky practice in software projects where all the disparate pieces developed over months or even years come together in a cataclysmic rush in the waning days of the project called "big bang integration".
The imprecisions in technical fit (aka "bugs") arrive in torrents at a time when the team is least able to resolve them due to budget and schedule commitments that were made long ago. The outcomes are unsurprising: Late delivery of solutions, uneven quality, demoralized/disinterested teams, unsatisfied customers and a moribund industry plagued by projects with nearly-ruinous cost overruns.
Coming to Terms with Reality
Software projects are fundamentally different from traditional civil engineering projects like house building or widget manufacturing: The nature of the work is almost entirely design-oriented, making the use of processes and methodologies based on the separation of design and construction activities with little to no inter-communication pointless and futile. By attempting to coerce design activities into phases that disregard or downplay the realities of the work, tension is created that becomes expressed in schedule slippages and poor quality. An automatic reaction is to blame the developers and the managers rather than the process they are working under, which leads to counterproductive behaviours to plan more, be more precise with estimates, add more developers and the like - and all without any commensurate gain in productivity or ROI. This mismatch is the root cause of software project failure.
The remedy is to come to terms with the reality of the nature of design-oriented work and apply a process that permits it to flourish within controlled boundaries. In agile software development this is expressed in frameworks that promote the trinity of visibility, inspection and adaptation.
In the next article, we will delve further into the constituent elements of software development and how agile practices help to focus them for productive gain and resultant ROI.
Brooks Jr., Fred P., The Mythical Man-Month. University of North Carolina at Chapel Hill, Addison-Wesley, 1995.