Derailleur Blog

The Rise of the Agile Project Manager Chimera

Fri, 17 May 2013 12:24:32 GMT

Lately, I've been noticing an upward trend in job postings for Agile Project Managers, or Project Managers who possess agile certifications or experience working with agile teams. This is likely the next stage from the precursor of trying to hybridize single-pass/phased software delivery methodologies with agile frameworks which I've written about before here and here.

Usually the job descriptions will contain all kinds of language that is heavy on the PM side - words like "proactive", "direct", "drive", "manage", "ensuring", "leading" are featured in the description while somewhere under the responsibilities section will be wording like "Scrum Master using agile practices and methodologies, working with teams through sprints to write stories as required." For example, see this advert for a Project Manager with The Weather Network and this one for a Project Manager at Canadian Tire.

I call these roles Chimeras- after the Greek mythological beast with two or more heads - and they are not a good idea in any organization. As I noted in my blog posts about why not to mix waterfall and agile practices, it first communicates very loudly that you don't possess adequate knowledge of either model, and second that your organization's culture is now or will soon be fraught with distractions and dysfunctions.

Why Not to Hire Agile PM Chimeras

In brief, when you understand the context of each role, you understand how they are antagonistic to each other. You might think that your situation is different or special or that no one understands your organization or how business really works, etc. but this is all misdirection from the crux: You're likely not anywhere near where you need to be to make the transition to agile software development.

Here's some reasons why not to promote and hire for the Agile PM Chimera in your organization:

First and foremost: Knowledge work is in no way equivalent to physical work - it is entirely design based. Forget about the construction and engineering metaphors you have been dulled into accepting: All effort in software development, top to bottom, IS DESIGN. By extension, as Fred Brooks Jr. observed nearly 40 years ago in The Mythical Man-Month, there is NO EQUIVALENCE BETWEEN MEN AND MONTHS because of the high levels of cross-communication that is required to develop software.
As a result, there is no real delineation between "designers" and "doers" on a software project - I call this The Fallacy of Division of Labour and it effectively negates wide swathes of traditional software Project Management "knowledge" and practices.
Project Managers are roles that are based on maintaining a hierarchical leadership role above the team and between them and the business. They take on the responsibility to devise a plan and then struggle mightily to keep everyone working toward it - irrespective of the realities. It is a role that is, in my experience and opinion, almost eternally misaligned with the emergent qualities of software development or knowledge work.
Project Managers are often directed or take it upon themselves to become Subject Matter Experts (SMEs) for their projects - even though this doesn't help the team. Agile Coaches, in contrast, don't seek to be the go-to guy for all knowledge on the project because that resides with the team. Their role is to help the team become their own SMEs.
In contrast, Agile Coaches and Scrum Masters are in sync with the realities of software development because their role is dedicated to being a servant leader that sits on the sidelines, rising only to maintain the team's momentum as they self-organize around their work. They are geared to helping the team (which includes the customer) to delivering as much quality, valuable software as they are capable of doing - they plan, but aren't beholden to it. They are flexible, not rigid; adaptable not intransigent; empirical not defined. You could not find a role more different in skills and attitudes from a Project Manager than an Agile Coach or Scrum Master.
Lastly, and most importantly: Mixing two divergent roles with divergent responsibilities is a context-switching nightmare. People aren't machines, and research has shown that those of us who think we're great multi-taskers are in fact quite awful.

If you hire a Chimera, as the two organizations above will likely do, you're guaranteed one of two outcomes:

A fantastic Project Manager who is a mediocre Scrum Master, or;
A fantastic Scrum Master who is a mediocre Project Manager and will likely quit after a few months of experiencing your dysfunctional organizational jungle gym.

If Not Chimeras, then What?

In my experience, Chimera roles are devised to hedge a bet - and as I've hinted at above, it's a losing proposition. My advice is to skip the roulette table and be clear and intentional: If you need a Project Manager, hire one. If you need an Agile Coach to help you implement agile practices in your business, hire an Agile Coach or Scrum Master.

Moreover: If you don't know what you want, consider reaching out to consultants and practitioners in the agile space who can help you figure this out for yourself. This constitutes a good chunk of what I do on a daily basis - and in most cases I don't even charge for the advice.

In any case, being clear and intentional with your hiring needs will help attract the right candidate for the role: Someone who either works with or against the "grain" of software development.

Five Questions You're Probably Not Asking Your Software Project Contractor (or Team)

Sat, 04 May 2013 01:22:11 GMT

What critical questions should you ask to determine the feasibility of your next software project? How do you determine whether your investment is going to pay off or take you and your company under?

For most customers, it's a matter of time and cost: "How long will it take?" and "How much will it cost?" While seeming reasonable, these questions don't give you nearly enough information to make a decision about whether to proceed - irrespective of how much up-front work goes into the answers - because they're misaligned with the nature of software development work. At best, they will give you a sometimes-educated best guess about what the final tally might be, but not how you will protect and ensure your interests as the project progresses. In other words, your ROI.

Software development is a 100% design-driven process. As a result, unlike most physical construction or manufacturing projects, there's no real division of labour between "architects" and "builders", primarily because there's nothing physical about software. It's ethereal thought-stuff that's captured into digital format by developers, artists, user interface experts, analysts, and the like which is constructed by machines into software that runs on a device.

In this process, "construction" is so cheap as to be free, but design is so pervasive as to be volatile, risky and expensive. In a design-driven process, the longer the lead time between request and result, the greater the probability it won't meet a customer's (or user's) expectations. (For a deeper explanation of why this is, see my post Understanding Misalignments in Software Development Projects). I often use the following simple graph to illustrate this correlation with new customers:

The X axis shows lead time; the Y axis shows investment and therefore risk. The dotted line shows customer investment into the project (negative) over a period of time and any subsequent "returns" they receive in working, tested, software (positive). The more this line "breaks through" to the positive side, the better.

In this graph, we see the typical "investment" curve for a project guided by asking "How long?/How much?" and related questions: A large, up-front investment in activities that do not result in seeing a representation of the final product in working, tested, software until well after the optimisically-estimated launch date. In some cases this "long tail" can be so long that the project is either scrapped or the project sponsor loses their job or the company takes on serious debt and goes under. Two researchers wrote about this very phenomena in an article in the September 2011 issue of the Harvard Business Review, Why Your IT Project May Be Riskier than you Think, referring to them as "black swans":

When we broke down the projects’ cost overruns, what we found surprised us. The average overrun was 27%—but that figure masks a far more alarming one. Graphing the projects’ budget overruns reveals a “fat tail”—a large number of gigantic overages. Fully one in six of the projects we studied was a black swan, with a cost overrun of 200%, on average, and a schedule overrun of almost 70%. This highlights the true pitfall of IT change initiatives: It’s not that they’re particularly prone to high cost overruns on average, as management consultants and academic studies have previously suggested. It’s that an unusually large proportion of them incur massive overages—that is, there are a disproportionate number of black swans. By focusing on averages instead of the more damaging outliers, most managers and consultants have been missing the real problem.

So what questions should you ask a contractor to determine their ability to protect your investment under such volatile conditions and avoid these IT project "black swans" ? I suggest turning the traditional "How long/How much" questions and their variants a bit upside down and inside-out with an aim to quickly determining a software contractor's capabilities and tolerance for delivering valuable software under constraints. All of these should be answerable within five to ten minutes:

How soon can I see a working, tested representation of my final product and what will be the cost?
On what ongoing basis will I see working, tested representations of my final product and at what cost?
How long will it take to make the product "production ready" ?
Can I make changes to the product mid-stream?
What will it cost to cancel the project ahead of schedule?

Question 1) is perhaps the most important to ask because it subverts gross speculation in favour of a delivering against a concrete, short-term goal. This is considerably more challenging than deferring to some optimistic, faraway milestones because it requires an ability to aggressively pare down technical complexity without compromising quality. Note that the objective here isn't to get a cheap prototype or proof-of-concept that will be discarded: This is to obtain the first "slice" of end-to-end functionality which will form part of the end-product's nucleus, sometimes called a "walking skeleton".

Question 2) extends the intent of the first question by probing the contractor's ability to deliver and demonstrate valuable software on a regular cadence. This tells you what the anticipated cycle time for your project will be, ie. the periods of time between planning and delivering. In the graph above, there is only one "cycle" with extremely long lead times which exposes your project to high levels of risk - shorter lead and cycle times help to mitigate risk.

Question 3) probes the contractor's ability to ship at will. This requires a team that understands quality software engineering practices such as automated builds and testing. A warning sign here is if the contractor needs 2-4 weeks or more to "stabilize" the product - an indication that they're forgoing in-situ tests while accumulating un-addressed defects.

Question 4) probes the contractor's development process' resilience to change - you can and should expect your product to evolve as it undergoes development. Accommodating change mid-stream should be welcome and seamless, not discouraged and painful

Question 5) probes the contractor's process for a smooth wind-down in the even that your ROI objectives aren't being met - for example, if the purpose of the solution becomes irrelevant or if there's been budget cutbacks. As with Question 4) this shouldn't be painful.

Some ideal responses would include:

Outside of some preliminary work to set the project up, 2-4 weeks depending on an interval we agree upon. Cost would be commensurate with time and materials for an agreed upon team size.
Every 2-4 weeks, in accordance with our agreed-upon interval.
Within 2-4 weeks of your decision to ship.
Qualified "yes": Like-for-like feature swaps can be made at zero cost for the next interval period.
Approximately 20% of the remaining value of the contracted period.

Teams that work in 2-4 week planning/delivery periods or "cadences" help to mitigate your risk exposure by dramatically decreasing the lead time between request and result, allowing for "course corrections" or changes to be made when it is most cost-effective. Using the same sample graph from above, we can visualize this difference with the traditional (and misaligned) "How long?/How much?" process:

Viewed this way, it should be readily apparent which process is most effective at reducing risk exposure while delivering working, tested, software. By subverting questions about "how long/how much" in favour of the above queries, you can gain some valuable insights about candidate contractors who are capable of working this way without having to resort to lengthy RFPs or interviews. If you get responses in-line with those I've listed above, you've likely found a contractor who well-understands how to align the work they do with how they do it - and will put that expertise to work for you.

Feel free to comment below or on Twitter via @DerailleurAgile.

Understanding Why Estimates Don't Really Matter

Thu, 18 Apr 2013 03:33:03 GMT

"No scene from prehistory is quite so vivid as that of the mortal struggles of great beasts in the tar pits. In the mind's eye one sees dinosaurs, mammoths, and sabretoothed tigers struggling against the grip of the tar. The fiercer the struggle, the more entangling the tar, and no beast is so strong or so skillful but that he ultimately sinks."

- Fred P. Brooks Jr., The Mythical Man-Month (p. 4)

Think about your current software project: How important were the initial estimates that you used to justify it? If you're like most, probably quite a lot. Now think further: How accurate were the estimates? How long did it take before you began to see a significant departure from "the plan" and reality? What was necessary to do to resolve this problem? What epic struggles were required to mightily bring the project back into line with the estimates - if at all?

Last week agile practitioner Morgan Ahlström wrote a provocative blog post about why estimates in software projects don't matter. He expresses a nuanced view of the long-held position among the agile community that a primary dysfunction of the software project management industry has been the misinterpretation of "how long / how much?" guesses as contractually-bound predictions against which progress is measured.

Ahlström asserts that because of naturally-occurring errors estimators make in understanding their team's capacity to deliver to a presumed level of quality and functionality, estimates provide ever-diminishing returns on their investment as accurate predictive indicators over time:

So if predictability is what you’re looking for, don’t invest much in your estimates, instead you should make sure that your capacity is known and that quality (internal as well as external) is well understood. And that’s why your estimates don’t really matter.

Thus, estimates, as we understand them, can be construed as a form of organizational 'muda' or waste. This echoes the sentiments Woody Zuill expressed in his January 20 blog post by asking the hypothetical question "If you found estimates bring no value, what would you do?". He asked this of fifty people and found that most refused to treat the question seriously - such is the entrenched mindset that support estimates. Of those that did respond, they offered rationales instead of answers:

I still like estimates, even if they bring no value, because they help me think about my work
We NEED estimates to make decisions
I just use them to determine velocity
My boss still requires them, even if I think they are waste
We need estimates to be able to bid jobs – our customers require them
It depends on the context

Woody rarely, if ever, heard a respondent say that if estimates were wasteful that they'd eliminate them - such is the entrenched cognitive bias in favour of always estimating before doing. Interestingly, as if to reinforce the point, when he changed the question to ask "If waste is anything that has a cost but has no value, would you eliminate waste from your process were you to find it?" he found almost unanimous agreement.

In my experience, and in simpatico with some of Woody's respondents, I have founds teams and their organizations expend a significant amount of time, money and energy in generating estimates in software projects under the conviction that they need them to inform cost and schedule demands of customers - it's a strictly binary condition: Projects need estimates because the customer needs to know precisely how much this is going to cost. But contrary to the old axiom, in this case the customer isn't always right.

In quite a few instances, I've observed customers becoming ever more cynical about the estimates they're provided, often having a "secret reserve fund" of 25% or more of the total budget to cover the inevitable cap-in-hand request for additional funding as the latest milestones are exceeded for the second or third time in as many weeks. As a consultant at Microsoft, I learned early on that they institutionalized this dysfunctional behaviour with an internal funding pool to resolve estimate/reality discrepancies on customer engagements called "MIRF" or "Make It Right Fund". While originally intended as last-resort safety-net, it soon became a necessary line item in the budget as a positive feedback loop evolved between the escalating average cost of engagements and an overly-optimistic/naïve estimation process.

Indeed, as researchers Bjorn Flybjerg and Alexander Budzier noted in their 2011 analysis of over 1,400 large scale software and IT projects in both the private and public sectors in Europe and the US, it's almost a fait accomplis that your project estimates will not only be significantly wrong on average, but also stand a 1 in 6 chance of being massively wrong, resulting in cost overruns of 200%+ and schedule overruns of 70%+. In their estimation (pun intended), if your organization is unprepared to absorb cost overruns of 400% or more with 25%-50% of delivered functionality for a major IT project, it's best to sit it out because the evidence suggests it won't just cost your job, but potentially your entire department.

Understanding The Big Picture of Software Development

Ahlström and Zuill make the case for seeing estimates as a form of process waste due to their inherent limitations (which I support); however, there are other more fundamental reasons that compromise their effectiveness which have been long-established, but not really (as far as I can tell) stitched together into a bigger picture that can be easily understood by developers and customers alike. I've previously described these observations in earlier, related posts, which I've super-condensed below into three interlocking parts:

1. Humans Design Source Code; Machines Construct Software

In his 1992 C++ Journal essay, What is Software Design?, Jack Reeves makes a rather startling and important observation that has some serious implications for how we think about, plan and develop software solutions:

After reviewing the software development life cycle as I understood it, I concluded that the only software documentation that actually seems to satisfy the criteria of an engineering design is the source code listings.

There is one consequence of considering code as software design that completely overwhelms all others. It is so important and so obvious that it is a total blind spot for most software organizations. This is the fact that software is cheap to build. It does not qualify as inexpensive; it is so cheap it is almost free. If source code is a software design, then actually building software is done by compilers and linkers.

2. All Human Participants in a Software Project are Designers, Not Builders

As a consequence of source code defining the real design documents that are used by machines to "construct" a useable software product, it follows that applying a methodology that divides human labour into "designers" (ie. architects, business analysts, etc.) who create blueprints for "builders" (ie. software developers) to assemble into software is fallacious. As Martin Fowler observes in his 2005 blog post, The New Methodology:

In software all the effort is design, and thus requires creative and talented people
Creative processes are not easily planned, and so predictability may well be an impossible target.

It follows from this that most software development methodologies have, from first principles, been designed at cross-purposes to the nature of the work they were intended to govern. As a consequence, to quote Scrum co-creators Ken Schwaber and Jeff Sutherland, "You have been ill-served by the software industry for forty years - not purposefully, but inextricably."

3. Men and Months Aren't Interchangeable Commodities

When we acknowledge that the human part software development is entirely involved with creative design, and as a consequence there isn't a supportable division of labour between "designers" and "builders" as we have in a civil engineering or manufacturing project, we need to question the basis of estimation for software projects. Ahlström makes an oblique reference to the problem in his blog post where he questions the notion of capacity not being considered in estimates (emphasis mine):

Knowing that something will take 6 man weeks to implement has no value unless we know that we have 6 man weeks at our disposal. We need to combine our estimate with some kind of capacity measure in order to get predictability. There’s a big difference if our team can give the task 6 man weeks worth of attention within the next two week iteration or if they’re overloaded with other work and need 4 calendar months to finish the requested 6 man weeks.

The problem Ahlström exposes, of course, is one that Fred P. Brooks Jr. brought attention toward over 38 years ago in his book, The Mythical Man-Month:

"Cost does indeed vary as the product of the number of men and the number of months. Progress does not. Hence the man-month as a unit for measuring the size of a job is a dangerous and deceptive myth. It implies that men and months are interchangeable.

Men and months are interchangeable commodities only when a task can be partitioned among many workers with no communication among them. This is true of reaping wheat or picking cotton; it is not even approximately true of systems programming." (p. 16)

This is a subtle yet elusive point to most project managers and customers that bears reinforcement: Systems programming requires A LOT of communication between workers in order to do to any sufficient degree of accuracy and competency. Lengthy use case specifications and functional specification documents are an attempt to obviate the need for communication - to make systems programming like the reaping of wheat or picking of cotton. Of course, this always fails. Like water bursting dams developers need to communicate with each other to ensure they're on track. This becomes glaringly apparent when we onboard new people to our teams and painfully discover the famous law Brooks deduced from his debunking of the man-month: "Adding manpower to a late software project makes it later." (p. 25)

The lesson we should extrapolate is, as Alistair Cockburn noted back in 1999, to avoid characterizing people as first-order components in software development. They're people, not cogs or sparkplugs or hands at keyboards.

Overstating the Importance of Estimates

The importance and value of estimates have been grossly-overstated in software development for some time: At best they are poor stand-ins for reality until reality intervenes. As Flybjerg and Budzier's research demonstrates, relying on methodologies that try to plan out software project complexity with big-up-front-designs and estimates can give you an odds-beating chance of having an unmitigated career or company-ending disaster on your hands.

Morgan Ahlström and Woody Zuill challenge us to re-think the role that estimates play in our processes - agile or not - and by extension question how we develop and release software products. Pioneers in our industry like Jack Reeves, Martin Fowler, Alistair Cockburn and Fred P. Brooks Jr. laid the logical and intellectual foundation to rationally do so:

Software is designed by human beings in a highly communicative, collaborative process;
Software is constructed by machines into useable products from source code;
Because people are not machines, software projects will always be unpredictable to various extents;
As a consequence of people being people and not machines, it makes little sense to use "man-months" as a measurement or estimate of progress.

Ergo, estimates don't really matter: Individuals and interactions, working software, customer collaboration and empiricism do.

Agree or disagree? Have your say below or online via @DerailleurAgile on Twitter.

How I Surprised Myself at Agile Open Toronto 2013

Mon, 15 Apr 2013 15:19:02 GMT

One of the tenets of an Open Space meeting, especially those hosted by the agile community, is "prepare to be surprised", and indeed this is exactly what happened when I attended Agile Open Toronto 2013 this past Saturday. I was surprised on two fronts:

First: I wasn't expecting to lead a conversation about the controversial topic of software delivery without estimates;
Second: When I did, I wasn't expecting to have so many people attend (almost 25% of the total attendees for my timeslot).

In fact, I was just expecting to take in some of the excellent presentations like that of professional coach Sue Johnston on "coaching agile coaches" or Alexi Zheglov on A3 thinking. To be clear, this wasn't my first rodeo - I've been to many Open Space meetings, but I still surprised myself by getting up and pitching the topic. Open Spaces are a marketplace for ideas, and your session is a Minimum Viable Product that gets kicked around by real "customers" for an hour - and that never gets old.

The Topic: Can We Create Quality Software without Estimates?

In crafting my topic, I had to relate it back to the theme of this year's Open Space, "building quality". I decided to use the opportunity to gather attendee impressions on whether we can create quality software without using estimates, based on the original work of Woody Zuill who is doing just that with a customer in the US.

At its core, software delivery without estimates is the recognition that in spite of the time and effort that can go into them, estimates themselves aren't deliverables. Subsequently, they can be viewed as a form of waste or 'muda' that may not be providing and contributing value to the software being developed and in fact might be doing the exact opposite. (See my post from February on Understanding Why Estimates Don't Really Matter for more background.)

While seemingly contrarian, I think this topic challenges some very ingrained behaviours and cognitive biases we have about how we approach projects, making it an excellent candidate for exploring at an Open Space conference.

(Photo credit: Alexei Zheglov)

The Reactions

There were many. So many I had a hard time capturing them with a good degree of fidelity. Among the participants there was near-universal agreement that estimates weren't particularly useful beyond a very limited time horizon, and degraded to the point of futility for long-ranges (one astute young lady referred to them as SWAG - Silly Wild-Assed Guesses). There was a commensurate agreement that the context of the project is often the main driver for requiring estimates, ie. business structures or government regulations.

Perhaps the most profound revelation that came from the discussion was captured in one word: TRUST. When we have no trust, we find ourselves needing estimates as a proxy with our customers and managers; however, when we do have trust, the proxy becomes redundant.

One participant, in exploring this trust gap, nailed the underlying reason: Lack of shared understanding. I expanded this point noting that because many managers and business owners have limited appreciation for the distinctions in how intellectual projects "work" (ie. there is no physical labour) they fail to understand why traditional divisions of labour between "designers" and "builders" fail to apply. Estimates are just one manifestation of this misunderstanding.

Another participant made a rather obvious, yet very insightful contribution: We need to spend more time coaching business to understand these differences and to build commensurate trust in software teams so we can move beyond the mentality of estimates as deliverables and contractual obligations.

For more insights and highlights from the session (along with some a posteriori references and topic expansions) see my AOT2013 Dialogue Map. I *wish* I had the foresight to bring a laptop and pocket projector with me - but since this was a spontaneous talk, it wasn't in my original plan.

Lessons Learned

In my session I quickly encountered two logistic limitations that impacted how well I and the participants could explore our wicked problem on doing software delivery without estimates.

While our breakout room was large, two other groups were in the same space trying to explore their own problem domains; the volume level in the room got to a point that necessitated me having to rove around the circle of participants to hear them clearly and then try to keep it in my head while I ran back to the easel to capture the points on the pad. SUB-OPTIMAL.
Speaking of writing things down on wall chart paper, these archaic tools of a bygone era are totally inadequate for exploring a wicked problem. I found myself desperately wanting to get everyone crowded around a Dialogue Map, helping to build out their thoughts and to ensure I was capturing them with better fidelity than my rushed, stream-of-consciousness, chicken-scrawl on flip-chart paper. TOTALLY SUB-OPTIMAL.

Indeed, I think that under similar circumstances I will be packing a "kit" for doing dialogue mapping on-the-fly in future. I find it such a useful tool for exploring wicked problems and capturing the movements in conversation that I feel deficient when not using it.

Of course, you can do Dialogue Mapping without a laptop, but it requires some lateral space. If I had a little more presence of mind, I might have taped up some sheets on the wall behind me to build the map - but as you can see from the photo above, I had limited space. And I was also jealously eyeing that whiteboard between my corner of the room and that of another convenor.

I have some ideas on how to do dialogue mapping under these circumstances that I'll be exploring in a future post.

Conclusions

In spite of my challenges exploring the wicked problem of no estimates with participants, I really enjoyed this year's installment of Agile Open Toronto. As I mention at the outset, I've been to several in my time and some have been a little on the flaky side. This was not the case with AOT 2013. The sessions I attended were really engaging, and I met some really fantastic like-minded folks who were eager to share their insights with me (and I with them) and, like March Madness, I wish it was longer. As one participant put it at the conclusion of the event: "This is too much learning for just $25."

Thoughts, comments? Feel free to let me know your thoughts below or on Twitter via @DerailleurAgile.

Misaligning How You Work with the Work You Do: Why Waterfall and Agile Don't Mix Redux

Wed, 10 Apr 2013 16:05:15 GMT

Note: Be sure to also read my subsequent post, Understanding Misalignments in Software Development Projects, for a deeper analysis of the concepts I discuss here.

A recent blog posting about Top 10 Project Management Trends for 2012 suggests that for the coming year, teams will be blending agile software development practices with waterfall to create a wondrous new "hybrid" model:

4. Agile blends with waterfall for a new “hybrid” approach

Having moved from “manifesto to mainstream,” Agile has confronted project teams with the difficulty of implementing the experimental and hyper-collaborative approach. To transition an organisation into fully adopting certain aspects of Agile, project teams are combining traditional and Agile elements to create their own hybrid approach. In areas such as planning, requirements, and team communication, organisations are designing custom-made methodologies to do what works for them.

As I stated in my post of last October, Five Reasons Not to Mix Agile with Waterfall, this is a flawed approach that won't help teams adopt agile practices - even as a transitory step - because it is in conflict with how both approaches to project delivery work. In other words, when you try to mix agile and waterfall, you demonstrate poor understanding of both systems. Here's why:

Agile == Empirical, Waterfall == Defined

A principal conflict between agile and waterfall processes are their foundational principles. The former is based on a transparent process that is perpetually open to inspection and adaptation; the latter is a closed process based on static phases of execution that are intolerant to inspection and adaptation once the project has started as it causes churn that exceeds the boundaries, and, ironically, necessitates iterating to resolve.

Software projects deal with significant complexity on multiple fronts and scales that are highly resistant to linear planning. This is because they exhibit traits of being wicked problems. This necessitates using an empirical process to continually refine the solutions. I explore this topic more fully in my post, No Silver Bullets - Just Hard Work.

In Software Projects, All Effort is Design

Jack Reeves observed in his 1992 essay, What is Software Design?, that the source code developers create is the de facto design of the system, with construction performed by machines (the compiler and packagers). This has a serious implication in that it effectively negates the whole notion of drawing parallels between the division of labour on real-world engineering projects and software development where "skilled" architects do all the designing that is dutifully followed by "lesser-skilled" workers. I explored this topic more fully in my post The Fallacy of Division of Labour in Software Projects.

Waterfall is predicated on dividing design from construction - this means that when you attempt to blend it with agile frameworks by say, planning everything up-front and fixing delivery dates, budget and scope, you effectively negate the relevance of having a self-organizing team that iterates and learns. Contrary to some popular assertions, you can develop complex architectures using emergent techniques.

Productivity Will Worsen, Risk Will Escalate

Choosing to create an agile/waterfall mashup needs to be examined for the root-cause motivations. If it is to be used as a "stepping stone" to adopting a process framework like Scrum, the rationale needs to undergo a more rigorous examination, eg. Taiichi Ohno's "5 Whys". I'd suggest a facilitated dialogue mapping session using IBIS notation on a projected screen.

My experience and intuition tells me that the driver to not fully transitioning has more to do with mindset, courage and a willingness to enter into a potentially painful cycle of continuous improvement that will involve a chaotic period of successes and failures. This is the "change adoption tax" - it doesn't come for free, it can cost customers but it is only through this effort that you can reach a place where agile practices will benefit your organization.

This diagram shows Virginia Satir's Change Process Model - it illustrates the shifts that occur when organizations begin to adopt and internalize new practices. It begins with the Late Status Quo, a place where people are comfortable in familiar surroundings, which then becomes disturbed by a challenge that reaches a tipping point where the status quo is interrupted and enters into a period of Chaos. During this period, there is a lot of volatility as the organization begins to experiment and learn with how to realign how they work with the work they do. Results are unpredictable due to group and individual dynamics, organizational intransigence or openness, enforcement or relaxation of bureaucratic rules, presence or absence of shared understanding of the motivations and solutions to change and ultimately vision. Moving into the next phase of Integration and Practice, where organizations become proficient, productive and efficient is dependent upon the effort put into continuous improvement during the period of chaotic challenge.

Mixing defined processes with empirical processes will elongate this chaotic period: Productivity gains will be unpredictable as traditional command and control roles and practices conflcit with agile framework practices. The ensuing chaos within the teams that are told to self-organize and iterate against the fixed forces that enslaved them before will drive up risk more than previously. Results will be mixed and almost assuredly will sour managers and customers from commiting to agile adoption.

In other words, counterintuitively, it will make things more painful and more expensive to delay the effort to move entirely toward agile adoption.

Conclusion: Agile Adoption is about Realignment of How You Work with the Work You Do

The take-away from this post and my previous one is that agile frameworks are based on and enact significant changes and realignments in how teams and organizations work with the nature of the work they are undertaking. Defined processes, like single-pass/phased or 'waterfall' are designed to substitute expertise with process by dividing design and construction activities. This model excels when the tasks to be undertaken require very little interpersonal communication or shared understanding of problems and responds well to adding addtional manpower.

Software development, however, does not respond well to this type of organization because it is entirely intellectual, design work. It requires an empirical approach with high interpersonal communication, substituting process with the expertise of a team's aggregate skills. Because problems can shift in unpredictable ways. it benefits from short iterations of planning, inspection and adaptation.

By attempting to mash-up the two models, an unpredictable mis-alignment of how people work will occur, resulting in a protracted period of head-scratching on what went/is going wrong. Change is hard, it isn't free, but it needs to be approached intentionally and with courage. As Ralph Stayer of Johnsonville Foods writes in his 1986 essay How I Learned to Let My Workers Lead (which I HIGHLY recommend anyone thinking about how to introduce change to an organization should read),

"Getting better performance from any group or individual, yourself included, means a permanent change in the way you think and run your business. Change of this kind is not a single transaction but a journey, and the journey has a specific starting point and a clear destination."

If there is serious desire to change, then get on with the change and begin to discard old practices. Otherwise, defer until the last responsible moment when such a move is possible.

No Silver Bullets - Just Hard Work

Mon, 08 Apr 2013 13:03:04 GMT

For every complex problem, there is a solution that is simple, neat and wrong. - H.L. Mencken

Early in my experiences evangelizing agile practices with teams and managers, I was often met with an allegation that was meant to marginalize my arguments: "Agile isn't a silver bullet." In other words, while Scrum might work on that skunkworks project over there, it certainly isn't going to work on this enterprise project. Big boys use big toys and Big Design Up Front.

Arguments like this led me to write a feisty 2006 blog post that put the feckless and reckless traditionalist perspective on notice with a pithy retort: "Scrum isn't a 'silver bullet', but at least it puts a round in the chamber."

I was recently reminded of this while speaking with an attendee to a talk I gave at a local user group about agile project delivery. They expressed the rather well-worn perspective that agile and waterfall practices have their place in IT project delivery - some are just more suited and "successful" when delivered using single-pass/waterfall delivery. In other words, agile isn't a "silver-bullet".

This is a perspective I've encountered over the years, and one that stems from misunderstanding of how software development projects really "work", what constitutes "success" and why the notion of any silver bullet solutions is misplaced - software development is hard, and not just because of the programming involved. Despite this, it's quite ironic that waterfall is given prominence as the default, "silver bullet" for project delivery, especially given how it has been misinterpreted from its author's original intent as the description of a flawed process that should be iterated at least twice. This said, I do believe that agile processes and frameworks are the only responsible and ethical choice for structuring a software project.

Of course, the notion of silver bullets emerged from old folklore tales of supernatural creatures like werewolves and vampires who could only be slain by them: A quick, easy, effective solution to a terrifying and apparently unstoppable problem. Fred P. Brooks Jr. (of Mythical Man-Month fame) adapted this meme to the problems with software project delivery in his 1986 essay, No Silver Bullet: Essence and Accidents of Software Engineering, where he wrote:

"So we hear desperate cries for a silver bullet -- something to make software costs drop as rapidly as computer hardware costs do.
But as we look to the horizon of a decade hence, we see no silver bullet. There is no single development, either in technology or in management technique, that by itself promises even an order of magnitude improvement in productivity, in reliability, in simplicity."

For Brooks, the very reason people were searching so desperately for a silver bullet solution to late projects with blown budgets stemmed from their inability to perceive the real underlying causes for their problem in much the same way early medicine was practiced in the belief of exorcising demons and examining the four humours for imbalances. Thus, the first step toward having a mature conversation about the realities of software delivery is understanding it at its cellular level: How the body technical functions, so to speak.

In my talk, I ascribed the problems in software development to six elements that together create a wicked problem, one that stubbornly refuses to respond to the usual "treatments" of a defined, stage-by-stage linear process with division of labour:

All effort is intellectual.
The intermediate products created are at best marginal representations of the thoughts involved.
There is more un-predictability than predictability.
The raw materials are volatile.
There are emergent rather than apparent answers.
People are involved throughout the process.

All Effort is Intellectual

In his 2005 essay, The New Methodology, Thoughtworks Chief Scientist Martin Fowler made three key observations about software development: First, construction is so cheap as to be free (because it is the result of running automated tools that convert code and artifacts into executable packages); second, all effort is creative and thus requires creative and talented people; third, creative processes are not easily planned, and so predictability may well be an impossible target. The common denominator is the application of intellectual over physical labour, which ultimately negates the usefulness and applicability of defined, staged linear processes.

Intermediate Products are Marginal Representations of Thoughts Involved

Scrum co-creator Ken Schwaber makes this observation when describing software development complexity in his book , Agile Project Management with Scrum. It is a reality that as requirements are communicated and translated into code and other artifacts that they become abstractions of the thinking that created them. As they get passed along, this abstraction can be magnified by successive abstractions making seeing the whole a challenging exercise.

There is more Unpredictability than Predictability

As described above, because software development involves entirely creative, intellectual effort it is difficult if not impossible to forecast - a situation that compounds with each individual on the team. This is a primary reason why estimates are difficult to discern, even when looking at similar projects:.

The raw materials are volatile

User requirements are raw materials of software projects, and because they are communicated using an imprecise language (ie. English or other tongue of the customer) they are quite susceptible to misinterpretation until verified in a working solution . And even then, they are subject to change once they are seen in their full context.

And so we have one of the great challenges in software projects: Change. It is anathema to traditional, plan-driven projects because it plays havoc with the schedule, causing unpredictable ripples and cascades of activities that are difficult to predict. This leaves two options: Padding the budget/schedule (which transfers risk back to the customer) or using expensive change requests. While padding seems prudent, depending on the changes it can and often does disappear quickly.

There are emergent rather than apparent answers

Creating software is a process of gathering and applying knowledge to resolve various aspects of a problem domain. Even when we're using technologies to which we claim competency and proficiency, things do not always go according to plan. We discover problems, solutions and workarounds as we go along: Some of these developments are fortuitous; some are disastrous. None are predictable.

People

The "X-Factor" in every project is the team assembled to design and deliver it: They take the fragile knowables and explode them into a galaxy of permutations based on their interpretations, their personal experiences, interpersonal relationships and biases, moods and emotions. This is one of the reasons why defined processes have been pursued with such zeal: To eliminate the variability that people bring into a project by separating design from construction. As discussed above, the nature of the work makes this an impossibility.

Similarly, in his essay Brooks began to touch on the inherent "wickedness" of the software development conundrum when he observed:

"I believe the hard part of building software to be the specification, design and testing of this conceptual construct, not the labour of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared with the conceptual errors in most systems.

If this is true, building software will always be hard. There is inherently no silver bullet."

Brooks attributed this bleak, realist outlook to what he saw as four irreducible essences of software development: Complexity, Conformity, Changeability and Invisibility. Combined they define the intractable nature of IT projects that negate the effectiveness of a simple, "silver bullet" solution. Moreover, and more problematically, they tend to inspire solutions that treat the symptoms rather than the causes.

For Brooks, complexity is the base essence of software: It is multivariate, manifold and cannot be simplified. It grows non-linearly as a product increases in scale and scope and affects all aspects of the effort. Conformity plays a part in this as it defines the often ad-hoc and arbitrary rules that a solution is built against - whereas a scientist has to conform to the laws of nature, developers often have far less certainty and a lot more capriciousness. Changeability relates to the malleability of code and the expectation of ever-evolving change and upgrades. Finally, software is itself invisible and unvisualizable because it doesn't exist in a real space making traditional mapping techniques impossible. While we can see how certain parts work, it is difficult, even with the aid of high-level tools to see and understand everything.

Definition of Success

Intrinsic to the pursuit of a "silver bullet" process is that once found it will be successful. But what is success? I would argue that our traditional definition of success, ie. a project that is on-time, on-budget and has all requested features and quality is misinformed because it ignores how usable the delivered solution will be and thus whether it has delivered true ROI. Moreover, it suggests that the parameters within which it operates (the estimates and budget) were sufficient to meet the desired outcomes.

What we often find is that projects are significantly underestimated at the outset and forced to conform to these parameters over the entire timeline while meeting expectations of sponsors and stakeholders. There are few opportunities to change course, and so we ride the rails to a conclusion that leaves all parties dissatisfied.

However, while timelines and budgets are critical, more important is getting the best solution to fit these constraints. When we plan projects using a linear, one-shot approach our risk for missing schedule, budget and delivering a solution that delights our customers is significant.

Agile No Silver Bullet - But It Works

So we come full circle: As discussed above, software development is an intractable, wicked problem that resists traditional, linear Big Design Up Front command-and-control methodologies because it is an entirely intellectual effort with intermediate products that are marginal representations of the thoughts involved, using volatile raw materials with more unpredictability than predictability contributing to emergent rather than apparent answers to problems - and all complicated by the fact that it is accomplished with teams of complex people.

If, as Brooks observed, there is no silver bullet to give us a measure of success, what do agile practices offer? The opportunity for success, even if our definition of success shifts by organizing projects under consistent and predictable rules of engagement that make the process and product visible, inspectable and adaptable. Agile frameworks like Scrum accomplish this with the trifecta of teams that are self-organizing and cross-functional, defined time-boxes called Sprints and regular inspection events that occur daily and at the end of each iteration. In this context, agile rules like Scrum harness rather than surrender to the chaos of software projects that result from wicked, intractable problems. By breaking delivery down into regular iterative increments, sponsors are given more engagement and control over their investment because they are engaged in an empirically-driven process where real data informs decision-making. In plan-driven approaches, their hands are effectively off the steering wheel and faith and hope are in control.

Agile frameworks like Scrum aren't a silver bullets, but they do produce results in exchange for sustained, hard work. And ultimately, this is what software development is about and why it is naive and irresponsible to look for easy solutions or to put faith in plans without continually balancing them against reality.

What do you think? Feel free to post your reactions, comments and questions below.

Understanding Misalignments in Software Development Projects

Sat, 06 Apr 2013 12:45:26 GMT

Note: This post is a work-in-progress; I expect to iterate over the design and wording of the misalignments to improve clarity and incorporate new ideas and information which may change some aspects.

Misaligning the work we do with how we do it:

When introducing Scrum to your managers and colleagues, you may find yourself in the position of having to convince or "sell" them on the merits of a very different way of working. While you may all see and feel the outward manifestations of why things haven't been working well like missed deadlines, runaway budgets, sub-par quality, stress and low morale, changing how you work seems like a very risky proposition - especially when weighed against faulting the team, the customer's unrealistic expectations, poorly-written and/or changing requirements and an unsympathetic QA department. The typical management response is to add more process, more management oversight and exhortations for employees to do things "right" or "better".

However, while in the throes of a critical project with money, jobs and credibility on the line, many of us fail to recognize that these "pain-points", contrary to our ingrained cultural beliefs and schooling, are signals that we need to work smarter, not harder. Much like a door that doesn't shut right or a wobbling bicycle wheel that needs truing, there are fundamental misalignments occurring between ourselves, our work, and how we do it that are preventing us from doing better. They can be ignored, of course, as most organizations do - but this comes at a cost of compensation in other areas. The result is an organization that is imbalanced and will always struggle to meet expectations.

The aim of this post is to explain six of the most critical misalignments that occur between the nature of the work of software development teams and how they are directed to approach them. A subsequent post will explore how agile frameworks like Scrum help to address them with simple process changes that bring about often dramatic improvements.

The Six Fundamental Misalignments

A misalignment occurs when something works against its intended purpose or design; this causes side-effects which we interpret as either indications of underlying problems or an intended consequence. These interpretations take on different characteristics depending on who makes them: The further from the root causes an individual is, the more likely cognitive biases will interfere with correct diagnosis. In other words, those who do the work tend to be more in-tune with what's going wrong than those who are directing them. The caveat to this is when the underlying problem is well-understood.

In traditional software projects, there are six fundamental misalignments that contribute to teams working at cross-purposes, making the goal of delivering valuable software and ROI difficult to almost impossible:

To make the misalignments clearer below, I've created a notation that takes the form of {Misalignment} // {Is Misaligned With}:

Fallacy of Division of Labour // All Effort is Design

This first, and most fundamental, misalignment is the consequence of equating how software deveopment projects are planned, organized, staffed and managed with their real-world civil engineering and manufacturing counterparts - specifically the extent to which the effort involved can be effectively partitioned among project team members.

In the case of an engineering project, say building a bridge or skyscraper, labour is divided into two phases, Design and Construction, with highly-skilled engineers, architects and designers drafting plans and instructions in the former for assembly by lesser-skilled trades and labourers in the latter. This division of labour is reasonably cost and schedule efficient because the tasks which constitute the majority of effort, ie. construction, can be well-partitioned among multiple people with little-to-no communication overhead between them. All that's needed is to follow the plans. Behind schedule? Add more men.

This paradigm quickly degrades, however, when applied to the context of software development for the simple reason that what constitutes "design" is widely-misinterpreted as "construction". Software is "designed" by human beings using source code which is subsequently interpreted and "constructed" by machines into usable applications and solutions. The implication of this distinction is that software "construction" is "so cheap as to be free", yet its design so pervasive as to impose significant risk and expense.

We can gain efficiencies of scale by "farming out" the compiling of source code across multiple machines because the rules for doing so are fixed in each compiler and can be run in isolation; we can even do complex pre-processing work across multiple machines to aid in the running of programs, such as the famous Stanford University Folding@Home Distributed Computing Project demonstrates. No such analog, however, exists for designing software - it is intensely intellectual, personal and very difficult to distribute in the same way as a physical task. In fact, as Fred Brooks Jr. noted in The Mythical Man-Month, doing so contributes to lengthening the schedule and thus cost because of the additional communication overhead required between workers to understand how to perform and integrate their work with each other.

An interesting exception to this misalignment is the practice of Pair Programming where two developers work together on a single problem for an undetermined amount of time. This works because the developers are seated next to each other and share a workstation - one who "drives" the keyboard and the other who partners in the problem-solving - thus removing the overhead of communication. Go beyond a pair of developers, however, and problems will follow.

The misalignment of the Fallacy of Division of Labour is vital to understanding the root causes of software project failure because it equates intellectual labour with physical labour, starting a cascade of successive misalignments which I discuss below. I've written on this phenomena in a previous post: The Fallacy of Division of Labour in Software Projects.

Defined Process Control Models // Weakly-Defined or Continuously Variable Inputs

As a consequence of the first misalignment, a second is created that aligns a software project using a defined process control model based on product manufacturing schemes with the intent to produce repeatable outcomes from well-known, defined or fixed inputs. Most software development projects are not simple enough to have fixed inputs - they continuously vary based on the degree of agreement on requirements with customers/end-users, certainty on the technologies employed and the skill and abilities of the people who will be creating it. It is a wicked problem that requires an empirical approach based on the principles of visibility, inspection and adaptation at regular, short intervals to tame.

This misalignment also creates another by establishing a "push" model to move artifacts and work products across phases - often completely out-of-synch with the people who need them, eg. too soon, too late or too much all at once. As a consequence, the work system - such as it is - can become stressed and overloaded. This sets the stage for schedule slippages, quality cutting and cost overruns.

I've written about the underlying conditions that create this misalignment in a previous post: No Silver Bullets - Just Hard Work

People as Role-Based, Replaceable Components (Resources) // People as Unique, Skilled Contributors

When a software development project is considered as the outcome of 90% construction effort controlled using a manufacturing-based defined process, the logical extension is to view the people involved as role-based, replaceable components. This misalignment is made explicit by the repeated-use of the HR terminology of "resource" instead of "contributor", which recognizes that people constitute the most significant variable input to a project. As a result, organizations "shop" for team members based on the role-based skills they list on their CV instead of looking at their attitudes and abilities. The resulting frictions create team dysfunctions like heroics, demoralization (why be passionate about what you do when you're just a "Drone in Sector 7-G?"), hiding behind titles ("not my job") and scapegoating.

It's very difficult to promote true teamwork under these conditions because people are disconnected from their work and an innate desire to be recognized for their contributions. For more insights on this misalignment, see my earlier post on Why Financial Incentives Don't Work which references fantastic examples of people and team dysfunctions by Patrick Lencioni and Dan Pink.

Command & Control Team Management // People as Self-Organizing, Rational, Skilled Adults

As a result of viewing people as replaceable components and not unique contributors, the next misalignment of using command and control team management arises. This comes from a mindset/cognitive bias that only through direct and persistent management oversight, intervention and coercion can organizations be led. At the employee level, this has an infantilizing and degrading effect, building into people the expectation that they will be told what to do instead of being trusted to organize themselves around and be responsible for their work. Worse: It actively discourages the opportunities for creative thinking by making it explicitly risky to do so. Who wants to lose their job over trying something new that might not work? Overall, this misalignment is about trust and the issues managers have with giving control over to those who are better-suited to solving on-the-ground problems.

While an unfortunate reality for many, the good news is that this misalignment is undergoing corrections across multiple industries, including software. For example, see how the highly-successful video game company, Valve, operates without any direct management oversight. Also see Ralph Stayer's excellent 1985 essay, How I Learned to Let My Workers Lead, for an example of how a CEO transformed his business to let workers set the pace and direction of change. Also see this recent CNN article that outlines a shift in the thinking of business leaders to connect and collaborate over command and control.

When managers begin to view their people as people who are rational, intelligent adults capable of independent decision-making, a new style of management emerges, which Takeuchi and Nonaka observed in their 1986 Harvard Business Review article, The New New Product Development Game, as subtle management. I'll expand on this in a subsequent post.

Context Switching/Multi-Tasking // Individual and Team Effectiveness

When we think of people as replaceable components in a fixed system, it's not uncommon for an additional misalignment to occur where we view their individual capacities to respond to ad-hoc or multiple requests as nearly infinite. This is called "multi-tasking" or "context switching" and is rooted in a mythology that equates the practice with being highly-effective. In reality, multi-tasking impairs our ability to focus and see any task through to completion.

In a 2009 Stanford University study of high and low multi-taskers, the hypothesis of high-multitasking effectiveness was tested with not-so-surprising findings: Those who identified themselves as high-multitaskers underperformed those who were low-multitaskers or single-taskers in response to multiple and varied stimuli. Why?

Part of the explanation resides in how the human brain works: When we are called upon to switch between tasks in different problem domains (eg. projects) we need to mentually "unload" one to "load" or "reload" another. This takes time and is impacted by things like fatigue and distractions. This is why we feel slightly confused when we switch contexts between different problems and begin to grasp for mental guideposts to re-establish our original train of thought. This effect can be countered by taking reminder notes of your thought processes, but this incurs additional overhead.

When the effects of context switching and multi-tasking are multiplied across teams and their organization, the impact of this misalignment becomes evident in the stress it creates, causing interpersonal relationships to degrade as employees become overloaded. Context Switching has become an unfortunate, false badge of honour in the workplace: Its practice is an indication of an organizational culture that places a premium on capacity over throughput.

Hero Culture/Worship // Clarity/Constancy of Purpose

A direct consequence of the aforementioned misalignments is an organization that is in a perpetual struggle to keep itself together in spite of itself. As a result, it is seen that only "heroes" can help save contracts and steward departments toward profitability. Thus we see absurd job ads that make the call for "rockstars", "ninjas", "samurais" and a host of other mythological warriors to come to the rescue - for a great reward.

This culture of hero-worshipping is a misalignment with having individuals and organizations not having clarity and constancy of purpose.

Clarity of Purpose is a skill as much as a mindset - it speaks to understanding the context of a problem domain, ie. why it is being taken on and for what purpose. If an organization's ethos is to "push" work on to teams and provide little context, it is demonstrating incoherent purpose and will likely suffer a host of side-effects such as team demoralization and an emphasis on bureaucracy over entrepreneurialship and true innovation.

In their book, Personal Kanban, Jim Benson and Tonianne DeMaria Barry note from their experience that the "A-List" developers who demonstrated clarity of purpose on their software teams were ones who "took the time to learn why they were building the software in the first place. They sought clarity from the onset, gathering vital information and incorporating it into their design."

Constancy of Purpose is a closely-related concept that W. Edwards Deming described in his landmark book, Out of the Crisis, as the first of 14 Points of Effective Management:

Create constancy of purpose toward improvement of product and service, with the aim to become competitive, stay in business and to provide jobs.

When we allow a culture of hero-worshipping to govern our hiring and reward systems, we are attempting to superficially solve deeper problems in the work we do and how we do it which keeps us from gaining constancy of purpose. Instead of seeking ways to continuously improve our delivery, we instead look toward HR practices, organizational bureaucracy and hierarchies to boost our performance - with mediocre-at-best results. Any gains we might make are illusions as they disguise the effectiveness of the above-mentioned misalignments, excusing them as the failings of our people rather than the system they have to work within. It is no surprise, therefore, that hero-worshipping organizations have the most difficulty letting this misalignment go when transitioning toward superior team-based collaboration.

(See my earlier post on Deming's Red Bead Experiment for a practical demonstration on the ineffectiveness of focusing on people over their system of work.)

Almost Inextricably Out-of-Alignment

In their recent book, Software in 30 Days, Ken Schwaber and Jeff Sutherland remark that, "[businesses] have been ill-served by the software industry for forty years - not purposefully, but inextricably". This is the consequence of introducing layers of cascading and inter-related misalignments into organizations and projects between how software teams work and the work they do. As a result, the organization works against and in spite of itself, often in the dark about the fundamental missteps that are occurring, and often exacerbating the situation by inappropriately treating symptoms.

A critical tipping point is reached when the symptoms of the misalignments begin to overtake the ability of people within the organization and/or their customers to tolerate them. It is at this time that an exploration for new ways to manage the work becomes the imperative. Recognizing and understanding the six fundamental misalignments I've outlined above will enable business owners, managers and team members to begin to deconstruct their mindsets, work habits and philiosophies toward their work and start to introduce corrections to bring them back into alignment for greater effectiveness. This is the key to understanding the raison d'etre for almost all agile software development frameworks, including Scrum.

In a subsequent post I will explore how simple changes in behaviour works to realign teams with their purpose of creating valuable software (and thus ROI) for their customers.

Teaching Smart People to Learn with Dialogue Mapping

Sun, 17 Mar 2013 12:25:50 GMT

This morning I came across an article by Harvard Business School professor Chris Argyris, Teaching Smart People to Learn, over Twitter that describes the dilemma of defensive reasoning among professionals within an organization. This is the tendency for people to displace blame from themselves in the face of criticism - what we might more commonly refer to as blamestorming. Despite being highly educated and competent in all other respects, they do not possess the skills for critical self-examination , partly out of fear of appearing a failure. As a result, they don't know how to build a shared understanding of their problems and themselves.

For Argyris, the solution lies with open dialogue and reflection of one's behaviours back to oneself through others in order to promote more intelligent and effective problem-solving. Sometimes we need to see that we can be part of the problem and that changes in our own behaviour or reactions are required to remove impasses.

Upon reading and reflecting on the dialogues between managers and consultants that Argyris uses to illustrate the problems with defensive reasoning, it struck me that these were perfect circumstances to apply the technique of dialogue mapping, a process of illustrating and exploring a problem with a group in real-time through the use of a simple set of idioms on a shared display. (For more details, see my previous post, Making Sense of Wicked Problems with Issue Mapping and Scrum, for a review of the main features of Dialogue Mapping)

Map as a Mirror

Jeff Conklin, notes in his book, Dialogue Mapping: Building Shared Understanding of Wicked Problems, that dialogue maps can act as a sort of 'mirror' for a group, allowing them to easily see disruptive patterns in their conversations that can act to limit or shut-down exploring various avenues of thought:

"Disruptive behaviour patterns show up rather obviously. If a participant goes off-topic with a comment, the dialogue mapper can work with him or her to find out how to capture the comment either as a new question or as a node linked to an existing part of the map; either way, the topic shift shows up clearly in the geometry of the map. If a participant challenges the whole frame of the meeting (eg. 'Why are we talking about this? This isn't the real issue!') the dialogue mapper can recapture that comment in a clear way… that captures the participant's concern and any discussion about it without undermining the rest of the meeting's discussion. If someone asks a closed question (eg. 'Should we abandon the product line?') the yes/no options in the map reflect the limiting frame of the question. Thus the dialogue map allows the group to see the conversational patterns it uses." [p. 160]

Take this instance below where Argyris notes how consultants and their manager "talk past" one another on the issue of improving their delivery of value, resulting in a deflective and unproductive blamestorm that goes nowhere:

Professionals: ‘‘The clients have to be open. They must want to change.’’

Manager: ‘‘It’s our task to help them see that change is in their interest .’’

Professionals: ‘‘But the clients didn't agree with our analyses.’’

Manager: ‘‘If they didn't think our ideas were right, how might we have convinced them ?’’

Professionals: ‘‘Maybe we need to have more meetings with the client.’’

Manager: ‘‘If we aren't adequately prepared and if the clients don ’t think we're credible, how will more meetings help?’’

Professionals: "There should be better communication between case team members and management."

Manager: ‘‘I agree. But professionals should take the initiative to educate the manager about the problems they are experiencing."

Professionals: ‘‘Our leaders are unavailable and distant.’’

Manager: ‘‘How do you expect us to know that if you don't tell us?’’

With a facilitated dialogue Mapping session, the inconsistencies and disruptive behavioural patterns in this conversation would become glaringly apparent - recall that this is projected on a screen for everyone in the meeting to see and focus on as a tool for building shared understanding during mapping:

If you've never read a dialogue map before, it's quite easy: You begin with the question node at the left and read to the right. Ideas are presented in response to questions, which in turn may have subsequent questions or pro or con arguments. This map has a lot of hanging con arguments (the red "negative" symbols) which illustrate the dysfunctional nature of the conversation. There's no real resolution - it wanders off course and dead-ends: Do we need more meetings? Are managers really unavailable and distant? It's a lot of conjecture with little substance.

Before moving to the next dialogue, a few quick observations about the key features of dialogue maps that make them so well suited to facilitating these conversations:

First (and most importantly) none of the comments are attributed or directed to a particular speaker. This is by design - it allows the comments and responses to stand independent of those who made them, removing bias and the potential for ad hominem attacks.
Second, this map opens with an Instrumental Question focused on exploring the means and methods for achieving some objective - in this case how to improve the value of services provided to the customer. From here, three ideas are presented in response that are met with a lot of challenges (the red dashes are "cons" or statements that contradict or disagree with the ideas presented). It's a bit of a struggle to get toward the real nub of the problem which is "How can we convince our customers to be more receptive to our analyses and services?" - A potentially wicked problem if there ever was one!

This is where dialogue mapping shines: It provides a conversational and intellectual roadmap to surface and "pin" exploratory questions, ideas, and supportive and critical responses. It helps take the starch out of potentially defensive reasoning situations by making the validity or absurdity of what's being said visible. Once this is done, finding a path out of the mire becomes more attainable.

Mapping Toward Productive Reasoning

Argyris describes the polar opposite of defensive reasoning as productive reasoning, in which involved professionals apply rigorous soft-skill analysis techniques to learn surface how they react to the problem domain and project those reactions to others. To accomplish this, Argyris recommends an approach where an individual writes a case study about their problem and uses it as a simulation to assess their own reactions and those of others against. It's a means of "reflecting back" how the individual thinks they project concerns and how they are actually received by facilitating a context of shared understanding.

While a good idea, I think dialogue mapping could be used to greater effect in this respect. Consider the following dialogue Argyris relates in the article between a manager and his team about an unresolved issue concerning the perceived arrogance of customers toward consultants:

Manager: "You said the clients were arrogant and uncooperative. What did they say and do?"

Professional #1: "One asked me if I had ever met a payroll. Another asked how long I've been out of school."

Professional #2: "One even asked me how old I was!"

Professional #3: "That's nothing. The worst is when they say that all we do is interview people, write a report based on what they tell us, and then collect our fees."

Manager: "The fact that we tend to be so young is a real problem for many of our clients. They get very defensive about it. But I'd like to explore whether there is a way for them to freely express their views without getting defensive.

What troubled me about your original responses was that you assumed you were right in calling the clients stupid. One thing I've noticed about consultants - in this company and others - is that we tend to defend ourselves by bad-mouthing the client."

Professional #1: "Right. After all, if they're genuinely stupid, then it's obviously not our fault they aren't getting it!"

Professional #2: "Of course, that stance is anti-learning and overprotective. By assuming that they can't learn, we absolve ourselves from having to."

Professional #3: "And the more we all go along with the bad-mouthing, the more we reinforce each other's defensiveness."

Manager: "So what's the alternative? How can we encourage our clients to express their defensiveness and at the same time constructively build on it?"

Professional #1: "We all know that the real issue isn't our age; it's whether or not we are able to add value to the client's organization. They should judge us by what we produce. And if we aren't adding value, they should get rid of us - no matter how young or old we happen to be."

Manager: "Perhaps this is exactly what we should tell them."

Here's how this exchange could be mapped:

As with the previous example, this map only traces the conversation and not the actors, but differs in that it reaches a conclusion or decision point which is illustrated with the larger handshake icon. A few other notable characteristics:

First, the map is opened with a deontic question, or one that asks "what should we do" about a particular situation or scenario. This is inferred from the manager's opening comment.
Second, the examples that the consultants provide are tucked under a Background? question node to keep them visible, yet distinct from the main root of the map. In this way we don't diminish or disregard what the consultants are saying by omission, while keeping the "signal-to-noise" ratio low for exploring the real problem.
Third, we can see that this dialogue flows from identification of a root problem ("youthfulness" of team as a customer concern) to a supportive statement ("no perception of value-add") to subsequent questions on how the problem is being addressed now and what could be done to change customer perceptions going forward. Disruptive behaviours are identified ("bad-mouthing the customer") and made visible along with reinforcing arguments, taking the starch out of them.

While I have reconstructed this map from the written dialogue, a similar outcome could be achieved in real-time if I were facilitating the conversation. By making statements, questions and responses visible, the group is able to see their behavioural responses and test their validity. In this way, they begin to learn more about themselves, how they are perceived and the latent causes and/or solutions to their problems.

I'm not sure if Argyris was familiar with Dialogue Mapping or the work of Horst Rittel (inventor of the IBIS form notation used in dialogue maps) at the time he proposed his solutions within Teaching Smart People to Learn, but I think he might find the exercise at the very least interesting if not in simpatico with his objective of making the dysfunctional behaviours behind defensive reasoning visible and in plain sight while building skills in participants to work toward productive reasoning.

What do you think? Could Dialogue Mapping help your organization or customers resolve these types of issues? Feel free to comment below or over Twitter to @DerailleurAgile.

OREO Agility at the Super Bowl

Mon, 04 Feb 2013 16:22:38 GMT

One of the most significant benefits of moving software development teams and their customers toward an agile process is the dramatic shortening of feedback and approval cycles that allow for quick, timely decision-making in response to various external factors, eg. political or market forces. Typical single-pass/phased projects, characterized by their lengthy, laborious (and sometimes indeterminate) stages, often have many interceding steps and sign-offs that prevent this type of response until well after the event occurs.

A neat, real-time demonstration of how a team and customer, when partnered in an agile manner, can seize timely opportunities was demonstrated during last night's Super Bowl football game.

Just after half-time the lights in the stadium went dark for over half an hour due to malfunctioning power equipment. As maintenance crews worked feverishly to restore power so one of the most anticipated sporting events in the United States and Canada could continue, a team at advertising agency 360i was working on turning the time-sensitive event to their customer, Mr. Christie's OREO cookies', advantage.

"We had a mission control set up at our office with the brand and 360i, and when the blackout happened, the team looked at it as an opportuniy," agency president Sarah Hofstetter told BuzzFeed. "Because the brand team was there, it was easy to get approvals and get it up in minutes…

The key? Having OREO executives in the room, and ready to pull the trigger."

Here's the result that was fed to Twitter and in a matter of minutes re-tweeted over 12,000 times:

In the business, this is called "earned" media - it happens for little to no cost when the public runs with your idea and takes it viral. While appearing a case of "right-time, right place" for the uninitiated, this is the essence of what having agile teams is all about: Shortening lead times to allow businesses to seize opportunities in the market at the expense of their competition.

In this case, the OREO brand was able to capitalize on an opportunity because they were already prepared to take advantage of it - from the ad design and copy through to executive approvals. As I read about 360i's approach the parallels to the eXtreme Programming (XP) agile framework came to mind. One of the key tenets of XP, which has been lamented as impractical by many critics, is the "On-Site Customer":

"One of the few requirements of extreme programming (XP) is to have the customer available. Not only to help the development team, but to be a part of it as well. All phases of an XP project require communication with the customer, preferably face to face, on site. It's best to simply assign one or more customers to the development team."

The intent with the On-Site Customer is to shorten the communication lead time between the customer and team to make key business decisions at the point when they are most pertinent - in other words, just-in-time. Had 360i and OREO entered into a project relationship like most software and IT organizations with lengthy approvals lead times and gatekeepers, they'd never have been able to respond to this opportunity in time. At best, they'd have to do an ad that referred to the blackout well after the fact - possibly too long afterward to be that effective. And, they'd have to pay for it, instead of getting 12,000 re-tweets of earned media exposure and numerous subsequent spinoffs.

Frequently, the push-back to having this much customer involvement in a project is that it's "impossible" - there's not enough time, too much to do, etc. In these situations I try to impress upon the candidate Product Owner that the success of the project is commensurate to their participation. Software development isn't a "fire and forget" operation. It's an ongoing dialogue that has frequent, timely decision points. Because OREO's ad team included an executive who was empowered to approve decisions, they were able to seize an opportunity to get millions of dollars worth of brand exposure for almost nothing.

The lesson for business is simple: When you invest in re-aligning your teams to employ lean and agile organizational behaviours, you're better positioned to take advantage of opportunities in the market as they emerge.

Food for thought.

Productive Contention with Dialogue Mapping

Fri, 11 Jan 2013 06:41:05 GMT

Earlier today I came across an article on a local blog about a series of public consultations my local government (the City of Toronto) is hosting regarding the possible construction of a publicly-funded and operated casino that went seriously awry - well, at least from the perspective of the first event's organizers. As you might imagine, this topic is contentious with impassioned and sometimes ill-informed opinions on all sides of the debate. Anticipating this - and wanting to avoid the unproductive fiascos of past events that get turned into open-mic sideshows - city hall organizers decided to employ a bait-and-switch tactic:

Things started sedately enough. Residents entered the City Hall rotunda to find the usual assortment of open house accoutrements: printed information sheets, large posterboards with diagrams and key details, blank survey forms to fill out, and name-tagged City staff on hand to answer questions.

It wasn’t quite what many had expected, however. When they heard “public consultation meeting,” they thought that there would be some sort of opportunity to speak, well, publicly—to hear from other residents and share their own opinions in an organized setting. To get a sense, perhaps, of whether there was anything like a consensus view about casinos, and hear some arguments for and against.

And this is where things went predicably sideways. Within moments, a number of city councillors who were in attendance began to get mobbed by constituents (theirs and others) about the opportunity to be heard on the matter. Public spleens needed venting! It didn't take long before the councillors decided to take action and redirect people to an adjoining committee room where they could participate in a parallel meeting on the merits of establishing a city-run casino:

As the article notes, within 30 minutes of the start of the "sedated" open-house, twice as many people had abandoned the first meeting for the latter where some sides of the debate were, I suspect, rather unproductively aired, given the impromptu nature with which it was cobbled together. Likely there were aides who were furiously trying to scribble notes and make sense of the non-linear patterns of argumentation that emerged as points were raised, argued, abandoned, resumed, branched, and then argued further, all while enduring raucous cheering, boos, hisses and other verbal intimidation and derailment tactics.

The article's author doesn't arrive at any conclusion about how much more effective the second meeting was on outcomes, but it certainly was loud, raucous and at least some came away with the feeling their opinions mattered. For the City's part, they're vowing to ensure future meetings include public discussion, but likely aren't sure how they'll do that beyond more contentious and likely unproductive open-mic sideshows where the loudest voices hold sway.

So, how could City organizers solicit public feedback that was actually useful while avoiding descents into gong-show tactics? By making the dialogue that emerges from the consultations visible with Dialogue Mapping.

I've written about this technique, invented by Dr. Jeff Conklin in the early 1990s, and how it could be employed to help foster productive conversations in groups that need to grapple with wicked problems - those thorny, intractable balls of complexity, chaos and confusion that seem to have simple answers, but unfold into Byzantine labyrinths of argumentation and debate. Dialogue Mapping helps to avoid these outcomes by employing:

A neutral facilitator who is skilled in Dialogue Mapping techniques
A 'shared display' for rendering maps - usually a laptop connected to a projector
A simple notation for rendering arguments into the display - in this case IBIS or Issue Based Information System, which was created in the early 1970s by famed social sciences researcher Horst Rittel

By making the argumentation patterns in a meeting visible to everyone, Dialogue Mapped meetings take on a serious tone and tenor as the "nitty gritty" of the problems under discussion are "unpacked" and made plain on the map. As new avenues of thought and consideration emerge, the group present learns more about their problem domain, and more importantly owns it. While there is certainly going to be heat in these discussions, it doesn't last long because it is heard and put into the map for consideration by the group.

Dialogue Maps always begin by asking a question (of which there are seven key formulations), and in-turn are addressed with answers that are argued for or against. Here's one way the casino consultation could have been framed in a dialogue map:

One of the first things you may notice about a Dialogue Map is that you can intuitively read and understand it with almost no explanation. Starting with the root question on the left side of the map, the dialogue opens up as you move to the right. This is a key strength to Dialogue Mapping and the IBIS notation and why it excels where similar Problem Structuring Methods (PSMs) falter. By reflecting back the structure of the discussion, participants gain concrete mental "footholds" into their problem space - and begin to arrive at a shared understanding of the issues which can then be converted into a shared commitment on what to do next.

This speculative map begins with a question that City planners and organizers could have developed and selected beforehand to help focus the discussion. Having good questions is critical for having a productive Dialogue Mapping session: We need to avoid yes/no "closed" questions and instead look for more open-ended ones that provoke responses from the participants. I chose to get to what I think is the crux of the debate that will emerge at the consultations by asking a deontic question, ie. one that asks "what should we do about a problem": "Where should a casino be located within the City of Toronto?" Other questions could include "What are the benefits to Toronto for building a casino?" or "How should the City of Toronto manage casino development?"

Following the map from the root question you can see the lines of responses I imagine might emerge at a consultation: Nowhere; Near an existing facility (on the outskirts of town) and Near the SkyDome stadium downtown. Each is met with pro and con arguments on merit, and in some cases promotes further questions and answers. This is what the exploration of a wicked problem looks like when it is made visible.

What I've deliberately left out of this map is a resolution about what to do next; this would be the conversion of one of the answer nodes that directly answers the root question into a Decision (usually shown as a check mark or two shaking hands). This isn't likely to emerge out of a single meeting because it can take time for participants to arrive at a shared understanding of the question and emergent issues in the problem domain - it's likely to take several. However, the benefit to city councillors and key decision makers is significant: They and their constituents and subject matter experts gain shared insights into the problems and their potential solutions that might not have ever emerged any other way and they have a concrete, usable digital imprint of what was said, debated and resolved. This is called group memory and is a powerful aid for maintaining the momentum of a dialogue over time.

Perhaps, in time, the City may try to use a strategy like Dialogue Mapping to help provide productive shape to public consultations and in so doing avoid the gong-shows where councillors feel compelled to hijack carefully orchestrated and controlled public hearings to ensure their constituents' concerns are really heard.

For a case study on how Dialogue Mapping was employed in a similar circumstance of contentious public consultations, read the case study my friend and colleague Paul Culmsee relates in his co-authored book, The Heretic's Guide to Best Practices, where he facilitated discussions for the Directions 2031 Planning Framework in Perth, Australia to significant effect. I've also written and spoken on this topic as it applies to business and software development teams, most recently at Agile Tour Toronto 2012.

Speaking at Agile Tour Toronto 2012 on Dialogue Mapping

Fri, 23 Nov 2012 00:16:38 GMT

It's true: On November 26, 2012 I will be delivering a short, one hour workshop at Agile Tour Toronto 2012 on how to improve the outcomes of group meetings called "Making Sense of Wicked Problems with Dialogue Mapping" "Teaching Smart People to Learn with Dialogue Mapping". Based on two blog posts from earlier this year, Making Sense of Wicked Problems with Issue Mapping and Scrum and Teaching Smart People to Learn with Dialogue Mapping, the session will introduce attendees to this novel, yet powerful technique to help facilitate productive group meetings where participants may be quite divided or "fragmented" in opinion, alliances, beliefs and understanding. Lessons learned here can be applied almost immediately in-the-field, and will hopefully inspire attendees to learn more about Dialogue Mapping and how they can use it in their projects.

From the workshop abstract:

While the Agile Manifesto tells us to value "individuals and interactions" and "face-to-face conversation" as the most effective ways to convey information and solve problems, agile frameworks themselves haven't provided much guidance in the way that we can effectively conduct the meetings that can occur between and to software teams and their customers. Often, we find ourselves frustrated by situations that arise where we thought others understood our positions during a meeting and then discover that they hold an entirely different impression. We've failed to build a shared understanding of our problem domain.

Welcome to the world of complex, wicked problem-solving! By attending this workshop you will learn a deceptively simple technique for building a shared understanding of wicked problems with groups known as Dialogue Mapping. Dialogue Maps are themselves a form of shared "group memory" that you will learn to construct to provide a visual road-map to help your participants explore and resolve their problems while more thoroughly understanding one another.

The workshop will be delivered in an interactive/Socratic style, with the first half dedicated to learning about the fundamentals of Dialogue Mapping and how it can be applied to improve the common dysfunctions of the venerable and dreaded "meeting", and the second half a series of practical demonstrations where I will facilitate a "controversial" topic with participants.

I'm really looking forward to presenting and engaging with attendees on this topic - if you're going to Agile Tour Toronto 2012, I encourage you to join my session and have some fun with me untangling wicked problems. See you there!

There's A Lot of Room for Agile to Grow in 2012

Mon, 19 Nov 2012 16:23:58 GMT

Impatience asks for the impossible, wants to reach the goal without the means of getting there. - Hegel

InfoQ (a clearinghouse blog of current 'good ideas' in agile and software engineering) published a post today by David Bulkin lamenting the current and likely future state for agile implementations in the coming year: Looking Back at Looking Ahead, Gloom for Agile in 2012?

Reading over the choice excerpts, what struck me was not so much the pessimistic characterizations of what's going on "in the wild", but rather that it's not changed much over the past decade. While agile adoptions are definitely on an increasing trend (last year's Version One State of Agile Development Survey showed a number of continued positive indicators), change is proving an intractable problem for a lot of folks.

Industry commentators from Cutter Consortium identified all-too-familiar problems contributing to holding back businesses, teams and organizations from enjoying greater success with agile practices which can be distilled to significant misalignments in understanding and applying agile practices. For example:

Using traditional recruiters/head-hunters to hire agile coaches to "fix their agile implementations" without understanding deeper reasons behind their struggles;
Using Scrum as a local optima only while keeping the rest of the organization fixed in old command-and-control structures and practices;
Avoiding investments in improving developer skillsets and tooling to enable agility over adding bodies to projects riddled with technical debt (thus raising the probability of invoking Brooks' Law);
Investing significant time building massive, overly-detailed product backlogs prior to starting and maintaining throughout the lifetime of the project instead of allowing details to emerge through continuous dialogs - a significant source of muda or waste;
Avoiding changing parts of the organization or practices to enable agility out of fear of taking on the challenge - effectively allowing the impediment to metastasize;
Avoiding getting experienced help to guide adoptions in favour of "going it alone" with inexperienced in-house staff;
Adapting agile frameworks to fit dysfunctional company cultures instead of the other way 'round.

These points underscore just how difficult it is to change and realign bad practices into something more resilient, powerful and enabling. Some organizations can do this, some cannot. I'm an eternal optimist in that I believe any organization can change if they want it badly enough - and if they do, I really want to work with them to make this desire a reality.

However, I make this point to new customers who I am bridging into agile: What we're about to take on is not easy; it's going to take a lot of hard work, but if we persist it will be extraordinarily satisfying, productive and effective. When we allow ourselves to think it is easy or that we can take an easier shortcut, we've already begun down the road to failure. In other words, as Liz Keogh pointed out succinctly in a talk last year on Learning and Perverse Incentives, quoting the opening sentence of Waltzing with Bears: "If a project has no risks, don't do it."

This is a learning opportunity - it's something to embrace boldy, not shrink away from in fear!

This said, change doesn't come for free, nor does it have a lot of handy, jingoistic shortcuts. It goes in fits and starts, has growing pains, moments of uncertainty and then clarity, mirages of objectives that smash upon concrete realities, giving way to even more problems and opportunities for continuous improvement. There's no handbook to tell you what to do 100% of the time - merely the best advice based on what went right for some other people in a different context. This is the space that agile has to grow into in 2012: While there's a lot of continued dysfunction, having this naked and apparent is a good thing: It means that agile has delivered in surfacing (and continuing to surface) all the things we're doing wrong. All that remains is the hard-work of shifting the boulders.

If you're stuck and you're not getting good advice to get unstuck, try another advisor. Learn more about the transformation you're undertaking. Go and visit your teams to understand the nature of their problems. Give them responsibility to really get things done and learn to get out of their way and let them lead.

How Misalignments in Software Projects Affect Business

Mon, 05 Nov 2012 04:08:32 GMT

"Thanks for agreeing to meet on such short notice," said the well-dressed executive as we sat down in a pair of comfy leather sofa chairs at the local coffee shop with hot mugs of the local roast in hand.

"No problem," I replied, "Jeff mentioned that you wanted to chat about my work on understanding misalignments in software development projects."

"Right. I read your blog post, and I think I understand what you mean by working harder, not smarter and how the misalignments add a drag to our ability to be… 'nimble' in our turnaround times. I have a meeting with my partners this afternoon and they don't have the patience I do. We're in a tight spot with our business and Jeff has been agitating for changes in how we deliver solutions. He thinks that Scrum can help turn our fortunes around, but I'm not so sure. Can you give me a concise explanation on how the misalignments impact our business?"

"Sure. Scrum is only part of the solution - there's a bigger picture to consider. Have you read Eli Goldratt's business novel, The Goal?"

"Heard of it when I was in university - never got around to reading it," the executive quickly responded.

"You really should - excellent read. In a nutshell, it's about a plant manager, Alex Rogo, who is facing a dilemma: Head Office plans to close his operations in three months unless he can improve its sales and profitability. Despite investments in high-tech robots and training, the plant was lagging well behind its competitors.

With the livelihoods of over a hundred employees is hanging in the balance and in desperation, Alex reaches out to an old physics professor, Jonah, who now does business consulting. Long story short, he helps Alex discover how to turn the plant around by focusing on improvements that align to the goal of the business, ANY business, which is:

'To make money by increasing net profit, while simultaneously increasing return on investment, and simultaneously increasing cash flow.'

I emphasize each point by raising three imaginary columns in front of me. What I would give for a holographic display.

"Ok - I understand this well enough: The goal of my company is to make money at the end of the day. How does this relate to misalignments in software development projects?", asked the executive with some urgency. I had to step this up before our coffees began to cool. I moved my mug to a side table and leaned forward.

"I will get to that presently; Jonah explains to Alex that there are three measurements that express the goal and provide "levers" for re-aligning the entire organization while in its pursuit: Throughput, Inventory and Operational Expense. Throughput is the rate you generate money through sales; Inventory is money invested in purchasing things you intend to sell; Operational Expense is the money you spend to turn Inventory into Throughput.

For a knowledge-based company, Throughput is your pipeline of projects and solutions being delivered to customers from concept to cash; Inventory is everything your employees create to deliver the solutions - the ideas, documents, design artifacts, source code and so on; Operational Expense is everything you pay for to make the magic happen - lease, electricity, software licenses, whiteboard markers, post-its, consultant's fees, salaries, stocking the fridge with goodies and so on."

"Wouldn't salaries be Inventory?", the executive asks inquisitively.

"Read the book - for now, go with this definition; it reduces the confusion over whether a dollar spent is an investment or an expense. As Jonah explains, everything you manage in your organization is covered by these three measurements. They are the three primary gauges to which you judge any effort at improving your operations."

My coffee companion leans back in his chair and looks at me rather skeptically. As he begins to speak, I take the opportunity to take a gulp from my mug - ah, sweet nectar of the Gods.

"I'm not entirely convinced - just three measurements? We track dozens of KPIs on our Dynamics scorecards - it can't be that simple."

"Actually, it really is. Remember, what we're concerned with is your entire organization, not just local optimums in specific departments. This is why having a single team or department adopt Scrum without commensurate changes in the organization often leads to failure. But that's a story for another day.

Each of the six misalignments I discuss in my blog post directly impact these three measurements and by extension your goal. They cause your organization to tie capital up in wasteful activities in that hinder your throughput. It's why you might feel or sense that despite hiring and training more staff and having them work longer hours on big projects you're not really getting that much further ahead. You're effectively treading water."

The executive nodded in an approving, albeit wearily resigned manner while casting his eyes toward the ceiling. I continued my explanation.

"So, to put it in a way that you can discuss with your partners this afternoon, think of each of the misalignments as baffles in your throughput pipe that cause blockages as a project moves from concept to cash. Over time, these misalignments cause build-ups in excess inventory that will never be sold for profit, while your operating expenses continue to grow unabated. Eventually, you will reach a point where you need to take drastic measures to preserve what's left of the company.

Thus, the six misalignments can be viewed as the chief constraints a knowledge-based company needs to contend with in pursuit of its goal."

With the last sentence I finally seem to make an impact with my companion, causing him to lean toward me resting his elbows on his knees while clasping his hands. He seems to have a renewed sense of urgency about his situation. After some thought, he begins to speak in a firm and serious tone.

"So, if I understand what you are saying, much of how we run our entire organization - its culture, practices and structure - are what we need to address if we're to change our situation. Just adopting Scrum or Kanban or XP on a development team isn't enough? We need to look at how we handle our throughput as an organization?"

"Correct. Optimize the whole; avoid local optimums", I reply.

"And so, each of the six misalignments you describe can be traced to helping improve throughput, inventory and operational expense - in other words, our ability to make money", the executive presses on with increased enthusiasm. He's connecting the dots faster than I anticipated; not as out-of-touch as I had seen in others.

"Right, again. Each of the six misalignments point toward things you can do to change the relationship between the work your organization does and how they do it. This means re-evaluating the constraints that each imposes on throughput and determining how to resolve them in pursuit of the goal. Observe the measurements as you exploit the constraints in your favour and you'll improve your profitability."

The executive sits upright. He reaches for his BlackBerry which has just buzzed a notification.

"The partner meeting - it's been moved up an hour. I need to go - this conversation is to be continued. Can we resume later this week? Same time and place?", he asks.

"Sure. Let's get into how you can identify your constraints and begin to deconstruct them.", I offer.

The executive nods while standing and looking at his BlackBerry before rushing toward the door. My mind reels at the thought of what he might say to the partners and how well it will be received.

Five Reasons Not to Mix Agile with Waterfall

Mon, 05 Nov 2012 00:24:50 GMT

Note: I returned to this topic in a later post to expand on the reasons to not mix single-pass/phased methodologies with agile frameworks: Misaligning How You Work with the Work You Do: Why Waterfall and Agile Don't Mix Redux.

A perennial topic within the agile project management community (and that of their customers) is whether a middle ground exists that can "bridge the gap" between single-pass/phased (aka 'waterfall') methodologies and agile (Exhibit A: Agile Hybridization - Novel Experimentation or "They just don't get it.") as a pragmatic approach. In my experience this is a normal reaction when confronted with "radical" change from the status quo in software and IT project management that has ingrained an expectation of simplicity. I've likened this to learning how to swim while holding on to the edge of the pool. It takes a lot of cajoling from the instructor to "let go" and discover your body's natural buoyancy, but once done you wonder what the fuss was about.

Unfortunately, this strategy of mixing or hybridizing agile with waterfall rarely provides the advantages in productivity or predictability that the project manager or business wants because the two processes view projects, teams and people and how they work together in very different ways. They mix together about as well as oil and water.

In brief, this is because:

Single-Pass/Phased Delivery methodologies are used when the intermediate activities that are involved in a project (ie. the "phases") are well-understood and not subject to much variability or unpredictability, and therefore can be reliably laid out in a linear process that will produce acceptable, identical results irrespective of who does the work. Design activities are separated from construction. Think: An assembly line.

Agile Delivery frameworks are used when we the intermediate activities that are involved in a project aren't well-understood and contain numerous unpredictable and wicked problems that resist traditional linear planning techniques due to high uncertainty and risk. This necessitates an empirical process based on the principles of transparency, inspection and adaptation to manage effectively and outcomes are entirely dependent on the skills of the people involved because all the involved effort is design. Think: New product development.

All agile practices share a pedigree with the lean manufacturing techniques pioneered by Taiichi Ohno for the Toyota Production System. Lean practices view all the activities involved in delivering a project as a work system which can and should be regularly monitored and adjusted according to current conditions and problems; waterfall practices do not share this perspective as they are plan-driven.

Agile framework roles, events and artifacts are designed for maximizing productivity and efficiency within a succession of time-boxes that are regularly inspected and adapted; this is foreign to single-pass/phased methodologies which are infrequently inspected and adapted and generally intolerant to change as they are dependent on rigidly following an up-front plan.

Finally, you're delaying the inevitable decision: The sooner you transition fully to agile, the sooner you can begin learning how to apply the rules to your projects.

If you are introducing change within your organization toward adopting agile, you need to be fully aware of the underlying motivations. If there is a real need to improve productivity, profitability, ROI, customer satisfaction and team morale, agile frameworks like Scrum and XP can significantly help, but only when used in their entirety (ie. the roles, events and artifacts) because they comprise a complete work system. Like anything worth doing in life, it takes time to become proficient and the results are entirely dependent upon your team and organization and their attitude toward learning a new skill.

Visibility, Inspection and Adaptation: Competitive Power Tools
Understanding Misalignments in Software Development Projects

Queues, the Theory of Constraints and Subway Sandwiches

Sat, 03 Nov 2012 15:06:53 GMT

Update: See this video by Bill Hammack that shows why queues block up and how to untangle them - and why if you're in a single queue it always seems the other line is moving faster...

In my posts so far I have been describing how "agile" works to provide superior results over its traditional, waterfall based contemporaries. When viewed as a whole, each part forms a system of engagement and thought-processes for building software solutions under conditions that are hostile toward generating a return-on-investment.

Today's post is about another important facet of iterative/incremental frameworks like Scrum that enables practising teams to outperform the competition: Queues and the Theory of Constraints.

Queues, Queuing and Subway Sandwiches

Queues are all around us: They are the ordering or sequencing of people, items and processes into a system that is "loaded and unloaded" according to the principle of first-in-first-out (FIFO). When I line up at the local Subway sandwich shop to purchase a delicious ham and turkey sub or to send a parcel at the post office or pay for groceries, I enter a FIFO queue: The fellow at the head of the line (always a long way off from where I'm at) gets served first while I languish at the end of the line for what seems an eternity.

While this arrangement seems fair, it has a rather glaring constraint that should be familiar to anyone who has been in a line of more than a few people: The queue can only move as fast as the people serving it can process orders. If there is only one person, like a cashier or clerk, the line moves exactly as fast as they can handle it without fatiguing. As they tire, mistakes get made and the line slows. In the language of queuing theory (which we'll get to momentarily) we call this throughput.

Sometimes, as is the case with Subway, we try to alleviate the impact of queuing throughput by moving the customer's order through successive phases that are managed by multiple people. More people should equal more throughput, right? Start with ordering your sub sandwich, bread type and cheese; move along to the next employee to get your toppings and dressings; move to the next employee to pay for your order.

Seems to be an improvement, right? Not really: Again this is a sub-optimal queue because it is prone to uneven loading and stress as throughput is constrained by each step in the process as sandwich orders are "pushed" to successive phases in the queue. Sometimes, the fellow in the middle is working like mad trying to deal with a six sandwich order while the queue backs up at Step 1, resulting in the first and third employees idling. Other times, orders are taken and filled so fast that the employee handling cash is overloaded, backing up the queue and idling the first and second employees. In still other scenarios, customers foul the queue with confusing order changes or indecision or miscommunication.

Ultimately, this throughput problem can be a problem for Subway: If people in line don't feel that the pain of the queue's issues is worth it, they will implement the Law of Two Feet and go to the nearest competitor for a more speedy transaction. I know I do: If I see a line up of more than 8-10 people, I know it will just take too long for me to get my sandwich and I'll take my business elsewhere.

Queues in Software Development

Of course, those of us in software development are very familiar with the FIFO queuing problems of the Subway sandwich counter because it is the very model of the waterfall methodology where, instead of a sandwich, we push a software project through phases of discrete activities:

As I've described in earlier posts, this arrangement is very fragile and susceptible to disaster because software, unlike sandwiches, is a very complex product that sits at the convergence of agreement on requirements, certainty on the technology used and the skills and temperaments of the people involved in building. It is a process measured in weeks, months and years rather than minutes. If we have any miscommunication or mistakes along this queue, we get churn as a part of the product goes back to an earlier, supposedly completed phase for re-work, which fouls up the queue leaving some resources to idle and burn budget without a product to show for it.

Scrum and Goldratt's Theory of Constraints

Very early in the development of agile iterative/incremental frameworks like Scrum (during the late 1980s and early 1990s), the problem of waterfall's FIFO queue was identified as a critical weakness because of the multiple points of failure it contained, the primary one being that subsequent phases were bottlenecked or constrained until the entire work product from a subsequent phase was completed and handed off.

At around the same time, Dr. Eli Goldratt was advancing a similar observation that was occurring in how organizations and businesses managed their throughput that he called the Theory of Constraints (TOC). Goldratt's theory was based on the premise that the rate at which an organization achieves their goals (like building products) is limited by at least one constraining process which can only be overcome if "flow" through that constraint can be increased. Put another way, a system is like a chain that is only as strong as its weakest link - strengthen that link (and any other weak ones that emerge) and you have a more robust system.

For Goldratt, a constraint is anything that prevents a system from achieving its intended goal and could be internal or external, eg. people, processes, tools, equipment, policies, regulations, laws, markets, etc. Optimizing a system to adapt around a constraint was achieved through Five Focusing Steps:

Identify the constraint;
Decide how to exploit the constraint;
Subordinate all other processes to the decisions from #2;
Elevate the constraint;
Return to #1 once the constraint is removed; don't let inertia become a constraint.

Sounds a lot like the empirical process control aspects of agile frameworks which emphasize visibility, inspection and adaptation, doesn't it? That's no coincidence. Scrum and its kin are specifically designed around the notion of minimizing or eliminating the constraints found in traditional software delivery by radically altering the whole notion of a queue: Instead of having entire work products pushed from one phase to the next with attendant hand-offs, they are "pulled" into assembly and processed as soon as they are ready (the lean concept of just-in-time delivery). We see this when teams "pull" in User Stories from the Product Backlog that have been refined enough to be decomposed into viable software during Sprint Planning. Bottlenecking or constraints are minimized by:

Enabling the team to select or load work based on their estimated capacity for a defined time period (usually two weeks);
Empowering the entire team to handle the tasks that were previously separated into discrete phases - this is possible because we co-locate our team members together so they can communicate with each other frequently;
Inspecting the process every 24h with the Daily Scrum for impediments;
Inspecting the product every two weeks at Sprint Review with the customer for an ROI check (is it what they want?)

Scrumifall - Just Don't Do It, Ok?

Given the importance of queuing to the proper functioning of an iterative/incremental framework like Scrum, it shouldn't be surprising that when we try to marry it with waterfall in a hybrid arrangement (ie. 'Scrumifall') we get another sub-optimal disaster. Since all of Scrum's processes are designed to work with pull-based queues and empirical process controls, we introduce breaking constraints when we front-end it with waterfall FIFO queues, or push work products along a QA/Test queue that is disconnected from development.

In other words, it's a complete waste of time, energy and money that benefits no one, least of all the customer. In a subsequent post I'll explore the motivations for Scrumifall and how they point to larger dysfunctions that hold back teams and organizations and how to overcome them.

A Better Subway Sandwich Experience

Given all I have related so far let's apply some of the lessons learned for optimizing the customer experience for Subway Sandwiches by exploiting the constraint in their waterfall-style sandwich queue to improve "flow". As described, this constraint is that sandwich orders are moved along through stations, one at a time in a push-style train - any delay at any station and subsequent orders in the queue will pile up quickly causing delays and loss of potential revenue. Subway's goal should be to serve as many sandwiches as they can to provide wholesome nutrition in a timely manner.

How can this be done?

First: Remove the long-counter and single-pass queue.

Second: Replace the counter with several short ones, each with all the required ingredients for building a sub and a small cashier's station.

Third: Staff each station with two cross-functional employees who take your order and cash you out in one fluid transaction.

By implementing this arrangement, more orders can be taken with a higher degree of fidelity because the constraints of hand-offs of FIFO/push are removed: As you place your order with the Sandwich Requirements Specialist, they punch in the order to cost your "solution" immediately. As they do so, they repeat your order items to confirm them with you and to communicate them to the Sandwich Development Specialist who builds the sub to your specification. Any errors or changes can be accommodated immediately because there are no intra-queue handoffs:

What do you think? Feel free to leave your comments below. Me, I'm expecting a call from Subway any day now.

The Trend Toward Concurrent Testing Tools: NCrunch and MightyMoose

Thu, 01 Nov 2012 16:24:54 GMT

When I first began exploring agile software development techniques in 2003/4, much of what is now integrated into IDEs was yet to be invented - or at least refined. Test Driven Devlopment was beginning to gain traction and I used early versions of NUnit and csUnit to write tests against my .NET code. The notion of continuous integration (CI) - running batteries of unit tests on code check-in, was still in its infancy and it wasn't easy to configure - it took some finesse and expertise in knowing how to make beta plug-ins work against Visual Studio Team Server, or just abandon it in favour of a build engine like NAnt and roll-your-own solution.

In the end, there was some justification that writing good, tested code was orders of magnitude more expensive than just writing "it works on my machine" code (as bad as it was) because there wasn't a tight integration between the test suite, the IDE and the CI server. This causes developers and teams to spend time managing the relationship by shuffling between the various parts of the test system.

Over time, this improved as the test frameworks became integrated into the Visual Studio IDE, making it possible to write and run tests with a few flicks of the wrist. The lead time between writing and testing solid code was beginning to shrink in a manner similar to when Amazon and others enabled purchasing and shipping of books via a web portal. The constraints were shifting from teams not being able to do TDD because it was "too painful and time-consuming" to "how do we ensure we're writing good tests?" and "how do we run the tests in whole or in part?" - a good problem to have, overall.

While the situation improved with TDD, it becomes more complicated when we need to build and test the entire system. Continuous Integration servers help, but they are a manifestation of moving a key constraint downstream. What if developers could see there tests exercised in real time instead of waiting to run them through a framework or CI suite?

This hypothesis has been answered with the notion of concurrent testing. Concurrent test frameworks plug in to developer tools like Visual Studio and allow them to not only run test code on-the-fly as it is created, but also allow them to visualize the impacts their code changes are making across the system under test. This effectively shortens intellectual lead times that occur when writing code - on the order of when Amazon introduced the Kindle. The information is now accessible on-demand at the point the code is being written.

Two good examples of this are NCrunch (free to try, then buy) and MightyMoose (free for now). Both tools embrace the idea of making developers more productive by giving them a lot of "set it and forget it" automatic backstopping to their coding and testing tasks, along with some impressive visualizations. Click over to the respective sites and see the demonstration videos that illustrate just how forward-thinking these tools are.

As impressive as it is to see tests run in real-time, the visualizations are even more so. What has previously been locked into separate analysis tools, eg. for illustrating call-graphs, is now possible on-the-fly as these two screen captures show:

This kind of information is invaluable to developers when they are writing tests and debugging code because it helps them see the impacts that their changes are having in real time, not just after the nightly build and CI tests have run. By shortening this critical feedback loop, they make better decisions about the code they write. This should result in better builds and happier customers.

As always, feel free to comment below for 90 days or bring the conversation straight to me on Twitter via @DerailleurAgile.

Review: The July 2011 Scrum Guide

Thu, 26 Jul 2012 13:22:00 GMT

Just yesterday the gurus at Scrum.org released the awaited July 2011 successor to the February 2010 Scrum Guide, written by its co-creators, Ken Schwaber and Jeff Sutherland. In the past, this was the type of document you would purchase in a publication, like the seminal Agile Software Development with Scrum by Schwaber and Beedle, except without all the supporting theory, case studies and diagrams. Indeed, the most striking feature of the Scrum Guide is its austerity - it is intended to provide just the basic information on how the "game" is played.

The necessity to create a formal definition was something Schwaber had long-felt was necessary to undertake in the wake of Scrum's rather meteoric rise in popularity as the go-to agile process framework. It was becoming not only widely used, but also widely misinterpreted and bastardized into practices and strategies that, while inspired by Scrum, were definitely not part of the original specification, leading to teams that would "ape" agile behaviours but not understand why they were doing so and why they were failing:

"By early 2009, there were more than 60,000 CSMs. More organizations were using Agile processes than waterfall processes, and of those employing Agile 84% were using Scrum. However, less than 50% of those using Scrum were developing in incremental iterations, which are the heartbeat of Scrum. Martin Fowler wrote in his blog that he was encountering many instances of "Flaccid Scrum." Teams were using Scrum vocabulary but weren’t able to create a potentially shippable increment of functionality within a single Sprint."(Source: Genesis of Scrum, Scrum.org)

For Schwaber, the remedy to this conflagration was to first formalize the singular definition of Scrum in the form of The Scrum Guide. In contrast to his previous publications, a new emphasis would be placed on re-framing Scrum as a set of rules distinct the from the strategies that emerge from its application to various situations. The rules themselves, therefore, do not fail or succeed or become flaccid, but rather the manner in which they have been understood and applied. You are either "playing" Scrum or you are not - a very subtle yet critical distinction that Schwaber first alluded to in a blog post from this past April entitled Scrum Fails?:

"Scrum is like chess. You either play it as its rules state, or you don’t. Scrum and chess do not fail or succeed. They are either played, or not. Those who play both games and keep practicing may become very good at playing the games. In the case of chess, they may become Grand Masters. In the case of Scrum, they may become outstanding development organizations, cherished by their customers, loved by their users, and feared by their competitors."

This analogy with chess has become the foundation for the revisions to the 2011 Scrum Guide, as Schwaber explains in the 2011 Update Notes:

"For most all games and activities, rules are published and considered separately from strategy. For example, the rules of chess say nothing about the various strategies that have evolved for playing the game; so should it be for the rule book of Scrum... The Scrum Guide is the definitive rule book of Scrum and the documentation of Scrum itself. The Scrum Guide and the rules of chess offer simply the rules on how the pieces move, how turns are taken, what is a win, and so on."

Strategies for playing Scrum or chess vary widely and are explained in many books, articles, and blog posts on the respective subjects. For those of us working on a revision to the Scrum Guide, this meant that all tips, optional practices, and techniques should be removed from this document. This was done along with refining some language to correct some longstanding misunderstandings about Scrum."

What's New in the 2011 Guide?

And so it is with 2011 Scrum Guide: Gone are the sidebar "tips" boxes adding additional details about roles and practices, the Scrum "lore" like the old Chickens & Pigs joke and advice on how to draw a Sprint Burndown chart. In its place are very succinct definitions for each role, artifact and event. There have been, however, some notable differences that I think will generate some interest and controversy, although they do align with the new spartan approach for presenting the bare-bones rules.

Preamble & Roles

The preamble is short and sweet and gets right into the fray of understanding empirical process controls and the three "legs" of transparency, inspection and adaptation as they pertain to Scrum and its key events. From here, the Guide quickly turns to defining the characteristics of The Scrum Team and its canonical roles: Product Owner, Scrum Master and Development Team. In contrast to the 2010 edition, the definitions have been cleaned up and pared down to shorter paragraphs and bullets that sum up what each role does without radical alterations. I was anticipating some changes to be made to the Product Owner role, especially after attending a Professional Scrum Product Owner I session that Ken instructed this past April where there seemed to be some fluid thinking on equating the Product Owner as a Product Manager (as stated in the 2010 Guide), however this seems to have fallen under the heading of a "strategy" and thus left aside.

What was good to see is some clarity around Product Backlog Management and how the responsibilities for upkeep and grooming belong not just to the Product Owner, but the entire team who may have these tasks delegated to them (p. 5) with accountability remaining firmly with the PO.

In the 2010 Guide, the Scrum Master role was given a quick once-over inside of a paragraph, which while it did adequately state their purpose did leave a few things out about their responsibilities, some of which were scattered throughout the remainder of the document. This has been corrected in this year's guide, with bullets outlining the Scrum Master's responsibility as a "servant leader" to the Product Owner, Development Team and Organization.

Release Planning, Sprint Planning and Sprints

I was surprised to see that Release Planning had been deprecated to an optional practice in this year's Guide, pulling Scrum back into alignment with the practices defined in the original Agile Software Development with Scrum book. In its place is now a rather tidy run-through of just Sprint Planning. While the main ideas remain the same here with respect to time box sizes and the two portions outlining the "what" and "how", some subtle details have been added that help to give some definition to concepts that while commonly-held, haven't been written down with respect to how the team selects Product Backlog items and designs an increment with their forecasted expectations rather than setting some contractual expectation with the Product Owner (pp. 9-10). Overall, it's good to see the section re-written and cleaned up in this respect.

Daily Scrum, Sprint Review and Sprint Retrospective

While fundamentally unchanged, the language around what Daily Scrum is about has been clarfied from being just a 15 minute inspect and adapt meeting to a "timeboxed event for the Development Team to synchronize activities and create a plan for the next 24 hours". The rationale for maintaining cohesion while promoting faster decision-making and eliminating other tedious meetings remains the same.

Similarly, Sprint Reviews and Sprint Retrospectives are relatively unchanged from 2010 with revisions to improve clarity and purpose.

Product Backlog, Sprint Backlog, Increment and the Definition of Done

Unlike the 2010 edition, the Release Burndown isn't explicitly detailed as part of the Product Backlog in the 2011 Guide, replaced only with a passing reference to "other projective practices [being] used to forecast progress" and the wisened caution that "these do not replace the importance of empiricism". While valid, this was likely done to distinguish Scrum from Kanban and its near-fetish for tracking progress in Cumulative Flow Diagrams. It's a good point to remember that the only useful measure is the estimated work remaining, rather than what's been done.

Clarity on the ranking, values and ordering of Product Backlog features is provided that helps to expand a little on the main concepts from the 2010 guide, including some very welcome language around the concept of "done" as it relates to the granularity of the items being moved to the top of the list (eg. well-defined, complete definition of done vs. coarse, unrefined concepts with no definition of done). Additionally, it was good to see the concept of Product Backlog Grooming captured as a rule, along with the guidance for dedicating 10% of the Scrum Team's time towards it. This is something that everyone who has Scrum experience knows is necessary, but is not always well-communicated to new teams.

As with the Product Backlog, the Sprint Backlog has been clarified and the Sprint Burndown graph eliminated as a defined tracking "rule" and is replaced with guidance to sum up the estimates of remaining effort daily so as to forecast the likelihood of achieving the Sprint Goal. In the Notes for the Guide, Schwaber and Sutherland point out that the concept of Sprint Backlog Items has been deprecated in favour of a more prosaic formula:

Sprint Backlog = Selected Product Backlog + Plan for Conversion into Functional Increment

Gone is the notion of having granular tasks that are estimated in 4h-16h alotments - now there is just the recognition that you need to have a plan, which may include having these task items if you so choose. I'm uncertain how I feel about this as it is a rather fundamental part of having a team build a work breakdown structure, but I think I understand the point that how you build your plan is less important than having a plan.

In contrast to the 2010 guide, Schwaber and Sutherland take the time to pull out the concept of the Product Increment into its own heading, categorically stating that irrespective of how its made, it has to be usable regardless of whether the Product Owner intends to release it. This is a rather interesting and subtle distinction that I haven't seen in previous writing or rules: We all know we have a goal of creating a functional increment, but this rule now states that this is paramount to any secondary or tertiary concerns about whether it will go into a release.

The definition of "Done" has been edited into four short paragraphs that provides a more comprehensive picture of what this thorny topic means. In the 2010 guide, pains were taken to say that when a feature is considered "done", this includes things like coding, refactoring, unit tests, acceptance tests, integration tests, stability, performance, etc. - anything that the team agrees and commits toward for a Sprint. For 2011, "Done" is pared down to a concept that everyone on the team must agree to, and more importantly aspire to over successive Sprints as part of continous improvement. Nice touch to include this, as it helps to shore up the notion that while "done" is important to establish stability, progress and ROI across Sprints, it's also a journey for most teams who do not always possess the talents or abilities to meet high watermarks straightaway - especially when adopting a new way of working collaboratively.

Overall Impressions

After having read the guide a few times over, I'm quite pleased with the new tone and direction of the 2011 Scrum Guide, which is much cleaner and more concise in its presentation and explanation of the core rules than its predecessor. As a Scrum "Players Handbook", I think it goes some distance toward meeting the objectives that Schwaber and Sutherland set out for themselves to create a formal definition that divorces "strategy" from "rules". We now have a distillation of Scrum that makes the material unequivocal and its application self-evident: You are either "playing" Scrum or not. You're free to adapt the rules, but once you do you're on your own: It's no longer Scrum.

With this delineation re-established, it will be interesting to see how the "strategy" side now plays out to fill in the blanks that have been left by the deprecated practices which Schwaber has indicated will be addressed in future posts on his blog or the Scrum.org site.

Overall, I give the guide 4/5 stars - it's a good, concise, readable guide that can be handed to most anyone who will be playing a role on a Scrum Team or will be responsible for one.

The Fallacy of Division of Labour in Software Projects

Tue, 24 Jul 2012 13:02:09 GMT

One of the most enduring myths in software systems development equates it to the building of a house or similar structure. While a seemingly useful abstraction to express what parts of a solution we're creating (eg. foundation, framework, architecture), it really obscures the actual nature of the work involved by implying that engineering and software projects share a common basis in how they are planned, designed and delivered.

This false equivalence has in turn given rise to the fallacy of division of labour in software projects which presupposes that design activities (eg. requirements gathering, analysis, architecture) can be divorced from construction activities (eg. programming, coding, packaging) thereby enabling the creation of predictable delivery schedules according to team size. In other words, according to the John Heywood proverb, "many hands make light work".

Unfortunately, tasks in software projects do not divide as cleanly as they do in civil engineering projects like house building or widget manufacturing while assuring similar precision. Consequently, there is a rather significant mismatch between the reality of software development activities and the current plan-driven, single-pass/phased methodologies currently used to organize them. In this article, I examine the root causes of this mismatch.

The Default Model: Division of Labour in Engineering Projects

Much of our so-called "modern" ideas about the efficiency and effectiveness of IT project delivery borrow quite heavily from engineering and manufacturing disciplines - which is problematic because the nature of the work, as we'll see, is entirely mismatched to this type of organization. I call this the "Default Model" because it represents the status quo that is found in many IT and software shops.

Under this model, design and construction activities are separated to create a cost-effective, predictable schedule that can employ people with lower skills to build the end-product. Highly-skilled (and therefore expensive) designers and architects create "instructions" that can then be handed off to lower-skilled (and therefore inexpensive) workers to "construct". In terms of effort and cost, this tends to break down as shown below:

Thus:

In a traditional civil engineering project, design activities account for the minority of overall effort and cost while construction activities constitute the majority.

This model presupposes that the delegated tasks undertaken can be perfectly partitionable with no required intercommunication between workers or partitionable with intercommunication between workers. In his 1975 landmark book, The Mythical Man-Month, Fred P. Brooks Jr. described this relationship between workers and months thus:

Men and months are interchangeable commodities only when a task can be partitioned among many workers with no communication among them… In tasks that can be partitioned but which require communication among the sub-tasks, the effort of communication must be added to the amount of work done.

In the first scenario, think reaping wheat, splitting wood or putting nuts and bolts together - these tasks can be divided evenly across a labour force without cross-communication while providing the requisite precision. Henry Ford famously exploited this concept through the use of automated tooling machines that created parts which could be easily assembled by workers into finished products.

In this case, effort effectively becomes a multiple of the men added to a project because there is an approximately one-to-one exchange:

Credit: Fred P. Brooks Jr.

In the second scenario, sub-division of tasks can only be accomplished through intercommunication between workers, creating a "drag" effect on productivity due to the effort required to ramp-up additional workers on tools, technologies, techniques, etc. This blunts the impact of adding workers to shorten schedules:

Credit: Fred P. Brooks Jr.

The time (and thus cost) required to "ramp" and coordinate additional team members comes in pairwise multiples. To illustrate, consider the following cross-communication between two team members:

Now, the same cross-communication that may need to occur between three team members (3x Single Pairwise Communication):

And now with four team members (6x Single Pairwise Communication):

The Fallacy of Division of Labour in Software Projects

In his 1992 essay for C++ Journal, What is Software Design?, Jack Reeves makes two critical observations that effectively blows apart the view of software projects as analogous to engineering projects:

• First, in software, construction is so cheap as to be free;
• Second, in software, all effort is design

Software is "constructed" through the use of automated developer tools that convert source code into machine language and package it for installation on target systems; "design" is performed by a team of highly-skilled (and often highly-paid) developers, business analysts, testers, UI/UEX designers and architects throughout the lifetime of the project. Given that I'm mostly directing this article toward business owners and managers, let me underscore this point:

In a software project, construction is performed by machines; design is performed by the entire development team over the lifetime of the project.

This has a significant impact on the divisibility of tasks in a software project and by extension how the project can be planned and executed. In the default approach described earlier, the dual nature of the effort involved makes the separation of design from construction possible. With 90% of the effort involved in physical construction, the activities readily lend themselves to efficient division across a workforce. As the project takes on added technical complexity, however, intercommunication is required to enable other workers to assist.

In a software project, men and months are not interchangeable commodities because the work involved is by definition technically complex, requiring corresponding interrelationships between team members. The effort required to communicate knowledge to ensure the necessary precision for sub-dividing tasks quickly overtakes any advantages of having multiple workers and further makes it impossible to control schedule or cost through the simple addition of manpower:

Credit: Fred P. Brooks Jr.

This realization is known as Brooks Law: Adding manpower to a late software project makes it later.

This point confounds traditional phased software delivery projects that divide their teams according to specialization and phases of activity with the expectation that men and months are interchangeable and can be done so without much consequence. The necessary interrelationships are weak with infrequent cross-communication between team members in different phases where it would be most beneficial. Activities even within the same phase become siloed according to specialization, leading to the most injurious and risky practice in software projects where all the disparate pieces developed over months or even years come together in a cataclysmic rush in the waning days of the project called "big bang integration".

The imprecisions in technical fit (aka "bugs") arrive in torrents at a time when the team is least able to resolve them due to budget and schedule commitments that were made long ago. The outcomes are unsurprising: Late delivery of solutions, uneven quality, demoralized/disinterested teams, unsatisfied customers and a moribund industry plagued by projects with nearly-ruinous cost overruns.

Coming to Terms with Reality

Software projects are fundamentally different from traditional civil engineering projects like house building or widget manufacturing: The nature of the work is almost entirely design-oriented, making the use of processes and methodologies based on the separation of design and construction activities with little to no inter-communication pointless and futile. By attempting to coerce design activities into phases that disregard or downplay the realities of the work, tension is created that becomes expressed in schedule slippages and poor quality. An automatic reaction is to blame the developers and the managers rather than the process they are working under, which leads to counterproductive behaviours to plan more, be more precise with estimates, add more developers and the like - and all without any commensurate gain in productivity or ROI. This mismatch is the root cause of software project failure.

The remedy is to come to terms with the reality of the nature of design-oriented work and apply a process that permits it to flourish within controlled boundaries. In agile software development this is expressed in frameworks that promote the trinity of visibility, inspection and adaptation.

In the next article, we will delve further into the constituent elements of software development and how agile practices help to focus them for productive gain and resultant ROI.

References:

Brooks Jr., Fred P., The Mythical Man-Month. University of North Carolina at Chapel Hill, Addison-Wesley, 1995.

Three Contract Models for Scrum Projects

Thu, 24 May 2012 17:21:37 GMT

In this post I want to do a high-level fly-by of the options you have to establish a contractual relationship for your next agile project. Moving toward an iterative/incremental delivery process requires rethinking not only your software delivery practices but the legal framework that will enable them. As you may have discovered, traditional contracts are by definition inflexible agreements intended to apportion risk exposure in an adversarial relationship. Contractors are obliged to "make the date/cost" irrespective of the hurdles thrown in their way, while customers are dissuaded from making any changes mid-stream (irrespective of merit) through costly change order processes.

Obviously, this is the antithesis of agile software delivery which relies on an agreement that promotes a collaborative relationship between contractors and customers to promote sharing of risk with flexible options for accommodating changes while controlling scope. The following contracting models are an intermediate step toward this ideal; they re-balance the contractor/customer relationship by giving customers options for controlling scope and costs in exchange for agreeing to a covenant to abide by the rules of Scrum - which is mutually beneficial. In the case of Mitch Lacey's Ranges and Changes model, risks and latent knowledge are surfaced with a short, two-week exploratory project that can be used to prime an agile project with a preliminary Product Backlog and Release Plan.

Each model starts with using a standard fixed-price contract that includes time and materials backstopping as a starting point, thus addressing contractor risk exposure, and adds on options to accommodate changes and costs, thus addressing customer risk exposure. In turn, these provisions act as legal "interfaces" into Scrum, which governs project delivery on a day-to-day basis.

While these are great options for getting your customers to buy-in to a Scrum project, they should be considered waypoints in a journey toward an entirely different relationship management framework called alliancing, which my friend Paul Culmsee outlines in his recent book with Kailash Awati, The Heretic's Guide to Best Practices. However, this is a story for a subsequent post. As they say, "Start Here" and see where it leads.

As always, comments/suggestions/slings/arrows are welcome in the comments below or via Twitter at @DerailleurAgile.

Change For Free

Created By:

Jeff Sutherland, Scrum Co-Creator

In Brief:

Combines a standard fixed-price contract with an option for customers to make changes in scope (features) without incurring a penalty.

Key Feature:

Change for Free option clause that may be exercised by the customer to make changes in scope without penalty provided that they adhere to the rules of Scrum for each iteration by fulfilling the role of Product Owner and working with the team. In all other instances, changes are charged according to time and materials.

How it Works:

Per Scrum rules, the customer Product Owner is responsible for re-prioritizing their Product Backlog of features according to business value at the end of each iteration. Changes in the backlog's priority are free provided that the total contract work is not changed. New features may also be added for free during Sprint Planning or Sprint Review in exchange for an equal sum of low-priority features.

Customer and Contractor Obligations:

The customer must agree to abide by Scrum Rules and keep their Product Backlog in priority-order and involve their stakeholders and end-users to ensure that only the most valuable features are developed.
The contractor must provide assistance to the customer to help them build their Product Backlog and facilitate working with the development team under Scrum rules of engagement.

Money for Nothing

Created By:

Jeff Sutherland, Scrum Co-Creator

In Brief:

Adds an option to Change for Free contracts that allows customers to terminate a project at any time for a nominal charge of 20% of the remaining contract's value.

Key Feature:

Money for Nothing option clause that may be exercised by the customer for early termination of the contract provided they adhere to the rules of Scrum, otherwise remaining work to meet the cutoff is assessed using time and materials charges.

How it Works:

The customer may, at any time, decide an ROI-cutoff point, ie. the point where the cost to develop the next feature or set of features in the backlog exceeds the value to the business. Accordingly, the vendor agrees to terminate the project for 20% of the remaining contract value while assuming the risk of late delivery for any mutually-agreed upon features to meet the ROI-cutoff.

Example:

You have agreed to a $100,000 contract with a number of features that are being developed across eight sprints/iterations. During development on the sixth sprint, the customer decides that they should have enough features to release the software by the end of the seventh sprint. They request a meeting with the Scrum Master and their manager to announce the invocation of the Money for Nothing option.

Once it is confirmed that the customer has met their obligations and is greenlit for the option, the team is informed that this will be the final iteration and they are committed to complete the work that will be mutually estimated and agreed to for sprints six and seven as per usual Scrum rules and options.

With each sprint costing $12,500 ($100,000/8), the remaining value of the cancelled project for the last sprint is $12,500. After invocation of the Money for Nothing option, the customer retains 80% of this value ($10,000) with the 20% balance ($2,500) going to the vendor.

Thus, the total cost of the contract to the customer is $12,500 * 7 = $87,500.00 + $2,500 = $90,000.00

Customer and Contractor Obligations:

The customer must agree to abide by Scrum Rules and be a full-participant in their project;
The contractor must agree to the terms of the Money for Nothing clause, including the reasonable assumption of risk for features to meet an arbitrary cut-off date. This necessitates having good software engineering practices that keeps the software in a potentially-shippable state at the end of each iteration.

Ranges and Changes

Created By:

Mitch Lacey, author, The Scrum Field Guide

In Brief:

A two-part contract consisting of a discovery phase designed to provide customers with ballpark cost and date ranges for a preliminary set of features which can then be used to determine if it makes sense to proceed with a delivery phase using a Money for Nothing/Change for Free contract.

Key Features:

A Discovery Project Contract for a short, fixed-length investigation of a customer's software requirements and build a preliminary Product Backlog that is assessed a range of estimated costs and dates according to team size and expected velocity in story/feature points. This project should include enough funding for 2-3 developers for two weeks. Deliverables include an estimated Release Plan and Product Backlog along with an expected minimum number of feature/story points that the team could deliver based on the investigation.
A Delivery Project Contract that references the Discovery Project Contract deliverables and establishes a framework for delivery using the estimated ranges as a starting point. Also includes the team's Definition of Done to deliver each feature, how the work will be delivered and an explanation of how the customer can reject an iteration's worth of work. This contract will likely include the Money for Nothing/Change for Free options detailed above.

Customer and Contractor Obligations:

Same as with the Money for Nothing/Change for Free contracts;
Contractor agrees to provide all deliverables from the Discovery Project to the customer;
Customer agrees that all estimates derived from the Discovery Project are extremely preliminary and can be expected to change when implemented in a project.

Business Card "B" Side Agile Quotes

Tue, 08 May 2012 04:39:59 GMT

I've been working on some prototypes for the flipside of my business cards - here's a sampling using quotes pulled from my collection that I mocked-up in Visio and converted into PNGs. Feel free to grab them verbatim or use them for inspiration. Let me know your favourites - I'll mock them up and post them!

Derailleur Blog

The Rise of the Agile Project Manager Chimera

Why Not to Hire Agile PM Chimeras

If Not Chimeras, then What?

See Also:

Five Questions You're Probably Not Asking Your Software Project Contractor (or Team)

Understanding Why Estimates Don't Really Matter

Understanding The Big Picture of Software Development

1. Humans Design Source Code; Machines Construct Software

2. All Human Participants in a Software Project are Designers, Not Builders

3. Men and Months Aren't Interchangeable Commodities

Overstating the Importance of Estimates

How I Surprised Myself at Agile Open Toronto 2013

The Topic: Can We Create Quality Software without Estimates?

The Reactions

Lessons Learned

Conclusions

Misaligning How You Work with the Work You Do: Why Waterfall and Agile Don't Mix Redux

Agile == Empirical, Waterfall == Defined

In Software Projects, All Effort is Design

Productivity Will Worsen, Risk Will Escalate

Conclusion: Agile Adoption is about Realignment of How You Work with the Work You Do

No Silver Bullets - Just Hard Work

Definition of Success

Agile No Silver Bullet - But It Works

Understanding Misalignments in Software Development Projects

Misaligning the work we do with how we do it:

The Six Fundamental Misalignments

Fallacy of Division of Labour // All Effort is Design

Defined Process Control Models // Weakly-Defined or Continuously Variable Inputs

People as Role-Based, Replaceable Components (Resources) // People as Unique, Skilled Contributors

Command & Control Team Management // People as Self-Organizing, Rational, Skilled Adults

Context Switching/Multi-Tasking // Individual and Team Effectiveness

Hero Culture/Worship // Clarity/Constancy of Purpose

Almost Inextricably Out-of-Alignment

Teaching Smart People to Learn with Dialogue Mapping

Map as a Mirror

Mapping Toward Productive Reasoning

OREO Agility at the Super Bowl

Productive Contention with Dialogue Mapping

Speaking at Agile Tour Toronto 2012 on Dialogue Mapping

There's A Lot of Room for Agile to Grow in 2012

How Misalignments in Software Projects Affect Business

Five Reasons Not to Mix Agile with Waterfall

Queues, the Theory of Constraints and Subway Sandwiches

Queues, Queuing and Subway Sandwiches

Queues in Software Development

Scrum and Goldratt's Theory of Constraints

Scrumifall - Just Don't Do It, Ok?

A Better Subway Sandwich Experience

The Trend Toward Concurrent Testing Tools: NCrunch and MightyMoose

Review: The July 2011 Scrum Guide

What's New in the 2011 Guide?

Preamble & Roles

Release Planning, Sprint Planning and Sprints

Daily Scrum, Sprint Review and Sprint Retrospective

Product Backlog, Sprint Backlog, Increment and the Definition of Done

Overall Impressions

The Fallacy of Division of Labour in Software Projects

The Default Model: Division of Labour in Engineering Projects

The Fallacy of Division of Labour in Software Projects

Coming to Terms with Reality

Three Contract Models for Scrum Projects

Change For Free

Created By:

In Brief:

Key Feature:

How it Works:

Customer and Contractor Obligations:

Money for Nothing

Created By:

In Brief:

Key Feature:

How it Works:

Example:

Customer and Contractor Obligations:

Ranges and Changes

Created By:

In Brief:

Key Features: