I have been in software development for over 43 years. During that time, I have seen many patterns of success and challenge. I do not mean just in the area of what people do, but also in how people are. As an industry, we now know what to do but knowing what to do does not mean we do it. For example, in the mid-1990s, we saw the advent of design patterns and more recently, we saw coding practices such as test-first, continuous integration, and the delegation of responsibilities. As good as those practices are to achieve good code quality, they are still used rarely.

This extends to the process of managing the work of software development. XP, Scrum, Agile, Lean Software, Kanban, the Kanban Method, Scaled Agile Framework (SAFe™) and others have given us a robust knowledge base. And yet, we still face the challenge of getting people to do it.

Overcoming this challenge requires that we understand the nature of our problem and the patterns of success we can bring to it. And even so, this is no easy task. Change induces fear. Fear shows in questions such as “are we doing it right?” and “will this pay off?” In addition to understanding what to do, we have to understand they whys and how to help the transition.

I formed Net Objectives 15 years ago. From the very beginning, its vision has been “effective software development without suffering.” This had a two-fold meaning. First, people may have challenges, but they don’t need to suffer as they work through them. People don’t have to be victims or martyrs in their jobs or lives. A mental attitude of being responsible for our actions and situations can go a long ways towards this. In addition, knowing how to effectively shift our circumstances can also be of benefit. For example, we can also avoid suffering simply by being more effective – we don’t necessarily need to achieve enlightenment.

Recently, the vision of Net Objectives has evolved to be “software development organizations overcoming their challenges with purpose, passion and continuous learning.” The essence of effectiveness is learning to adjust to your situation, to continuously be learning and to do so in a manner that is consistent with your purpose for being.

As an industry, we have come a long way. The effectiveness of software development organizations, whether they be product or IT oriented, has generally improved. But not universally. And even those who have shown improvement have not achieved anywhere near what is possible. Fortunately, there are patterns in the challenges companies have experienced that can provide insights into how to do better.

Most of the companies I have talked to over the last few years that are undertaking Agile are in one of the following situations:

• They are still struggling with Scrum at the team level either because they cannot get the teams to fully implement Scrum or because there are issues that are bigger than the team that are causing everyone to struggle.
• They are successfully doing Scrum at the team level but cannot get teams to work together efficiently.
• There has been general improvement in software development but the business is still not properly prioritizing the work given to the development area.
• The Agile effort seemed to work for a while but has now stagnated and has either stopped progressing altogether or has even retrenched a bit.

The general pattern is one of making initial progress only to falter after a relatively short while. Much of this is because organizations often just jump into Agile with a “let’s do Scrum because it’s the most popular approach” or “let’s do the Kanban Method because it’s supposed to work anywhere.” All too often not enough investigation of what the proper approach is undertaken. This is understandable since one would expect to get some solid advice from the experts but most experts seem to favor one approach. It’s hard to get unbiased advice when only one path will lead to business for the advice giver.

To take Agile forward, we don’t need yet another hybrid of approaches. Rather, we need to examine what we have learned – both good and bad – from all of these approaches and build a new approach that incorporates these lessons. This will challenge us and will clarify many misunderstandings about systems and complexity.

One of my favorite quotes by Mark Twain is “It ain’t what you don’t know that gets you into trouble, it’s what you know for sure that just ain’t so.” And Keith Cunningham fills in the other half, “My biggest problem isn’t that I don’t know what I don’t know. It’s I don’t do what I do know.”

We can have amazing success by recognizing how people want to start and how they need to learn and by attending to what is known.

Net Objectives’ approach in a nutshell

Lean-Agile is about incremental delivery of business value, not team iterations. It recognizes that people want a well-defined place to start; that is, they want to be told what to do at the beginning. However, since no one size fits all, we must give them an easy way to determine from one of a few choices the best place for them to begin. After this start, a new approach for improvement must be presented. Instead of focusing on a prescriptive approach, we must recognize that people now need looser guidance as they move from novice to advanced beginner to competence. For this transition, the focus must shift from practices to follow to desired outcomes to achieve. Patterns are provided which are selected from to continue their improvement.

All of this is done within the context of human nature and the laws of software development. Overall, a pragmatic approach based on real-world experience is presented.

Why do we move forward only to stall

The patterns of success, challenge and stagnation that we face as an industry are due to a misunderstanding or to a lack of attention to the following:

• Systems thinking and complex systems
• How people tend to learn
• How organizations tend to change
• The nature of software development
• How one should start a transition to better methods of software development

Systems thinking and complex systems

Complex systems are characterized by dynamic relationships between their related components. One cannot understand the system by individually looking at any one or two or three components alone. The system is not merely the sum of the components but rather it is the integration of the components, their relationships and the interconnections between them. Systems thinkers start with this understanding. You must recognize that you cannot effectively investigate or learn about a system by decomposing it into its components and looking at them individually. Rather, to learn how the system behaves, and more importantly, how to improve it, you must take a holistic view of the system. A system thinker can investigate components of a system, but they recognize that the sum of the components isn’t the system. They attend to how the component under investigation interacts with other components of the system. Any approach they take to improve the system will consider all of the components and the relationships between them.

One of the salient characteristics of systems thinking is that it is the system within which people find themselves contributes greatly to their success or failure. While we respect people, attention to the system is paramount. In fact, it is precisely because we respect the people that we must strive for good systems. Telling people they need to improve when it is the system causing much of the failures is one of the highest forms of disrespect I am aware of. This is particularly true if the people don’t have any control over their system. Management must make system improvement a priority. They must work with the people doing the work, who typically know more about how it effects them than anyone else, to see how to improve it.

The validity of the impact systems have on the people working within them is easy to confirm. Consider two cases of coding and testing. In the first case, a programmer works side by side with a tester. The two of them work with their product owner figuring out what their tests should be. As the programmer writes code, he not only can run his own tests, but the tester can run additional tests. The synergy improves understanding of what to do and the teamwork enables coding and testing to happen virtually simultaneously. Discovering and fixing bugs could be done quickly.

In the second case, there is a group of programmers who have a group of testers to validate their work. In this case, the test team tests the programmers’ code a week after the code is written. In this current case, discovering and fixing bugs will be much harder and more time consuming. There is wait time and miscommunication is inevitable.

In both cases, the programmer and the tester are the same. In one case, discovering and fixing bugs is easy, in the other it is hard. What accounts for the difference in productivity and code quality? The answer is obvious: the system.

Once we recognize the importance of systems, we understand that to achieve improvement of our methods we must improve the system we operate in. But, if we remember that complex systems are interrelated, we must attend to all aspects of the system. We cannot just say “create teams” while ignoring flow (or vice versa). When we start we won’t be able to attend to every aspect in detail. And we don’t need to. Part of our attention should be to ask, “Which parts of the system do I need to attend to at the beginning?”

So it sounds like systems thinking would have us investigate all aspects of our system and then make decisions on how to improve it. Actually, this would be true if we could ignore one of the most important parts of the system – our people. The problem is as systems become complex, people get overwhelmed.

How people tend to learn

There are different ways to both model how people learn and how to measure where they are in their learning approach. Helpful ways to look at this include Chris Argyris’ double-looped learning approach, the Toyota Kata approach to learning as described by Mike Rother, and the Dreyfus model of skill acquisition of learning to see the stages of learning people go through. Double-loop learning will get us to learn we must often shift what are goals are, Rother’s work will present how people learn in small steps, and the Dreyfus model will present the stages people go through when learning.

Argyris’ and Rother’s work represent fairly unique approaches and I will present them because they are the best. I use Dreyfus instead of the martial arts based shu, ha, ri, both because it relates to knowledge work better and because it provides insights into how people want to be taught at different stages of their learning.

The Dreyfus model states people go through the phases of novice, advanced beginner, competent, proficient and expert. At the novice level, people want to be told what to do. But as they progress, they are looking more for guidance and insights. In recognition of this we will suggest starting points for people. But only after we’ve explored which is the best starting point for the company involved. Fortunately, for most organizations, it turns out that not that many questions have to be asked to determine which of about a half-dozen starting points to begin with.

How organizations tend to change

Changing organizations is difficult. Most companies could stand more trust and effective management. While it’d be helpful to have a preponderance of both, we can start where we are if we understand what keeps organizations both resistant to change and what has early gains often result in slippages back to the beginning after some initial trials. Organizations, like people, tend to exist in a comfort zone, and even when improvement results in us moving forward, there is a tendency to get back into our comfort zone.

Why this happens can be summed up in one word – fear. But of course, people can’t admit they are afraid (they’re afraid that looks bad), so they use other phrases to mask it. Things like “that’s risky”, “we’ve never done that before”, “that won’t work here”, amongst many others. We must be aware that, in most circumstances, people don’t really like change.

To make change in an organization people must understand why the change is both necessary and why it has a chance of working and improving their lives. People are naturally interested in “what’s in it for me.” They are also tribal in nature, that is, they care more about the people closer to them (their tribe) than others in the organization. Therefore, change will require a combination of understand and personal motivation. People will have to stop focusing on localized optimizations and be able to see why a broader measure is more important than their tribal one. We will find that this is one of the areas that Lean Thinking can provide us with the greatest guidance.

Many agilists believe that to change an organization you must create trust first. But creating trust is very difficult if it doesn’t exist. I believe that at the beginning we must focus on change that creates trust where trust may not exist. In many organizations people are looking at the wrong thing. There is no shared vision, everybody is looking out for themselves. While this is not good, the lack of a common vision presents the possibility to get people to work together without a lot of trust if they can see that a common goal would benefit all. We’ll see that this rallying cry will later be removing delays so that we can deliver value to market faster.

In other words, we want to shift our organizations’ culture to one of collaboration across the organization to deliver value quickly. But changing culture is difficult. Consider the following excerpt from David Mann’s excellent book, Creating a Lean Culture (Mann, 2010).

Should a company target its culture in its efforts to transform its production processes and all the roles associated with it? It is tempting to answer: Yes! But that would be a mistake. Culture is no more likely a target than the air we breathe. It is not something to target for change. Culture is an idea arising from experience. Our idea of the culture of a place or organization is a result of what we experience there. In this way, a company’s culture is a result of its management system. Culture is critical, and to change it, you have to change your management system. Focus on the management system, on targets you can see: leaders’ behavior, specific expectations, tools, routine practices. Lean production systems make this easier, because they emphasize explicitly defined processes and use visual controls.

Our approach in changing culture will therefore be an approach to our changing how management in our organization works. If we can get everyone on the same page though a common vision of business value delivery we can get more cohesive management decisions.

We must also be aware of how organizations change. There are three phases of learning in organizations. The first often has a negative effect as we try out a new unfamiliar process. Then we start to improve and get better than where we originally were. Then we tend to stagnate as the pain that initiated the initial transition has subsided to some extent. This initial drop in effectiveness and then later improvement is often referred to as “the J curve.” To maintain improvement we must continue to strive to move forward in successive waves. T

Some people think we can avoid the dips by avoiding revolutionary change. In other words, we want just slight improvements all the time. In fact, there is a middle ground – and it’ll be the one we propose. There are some changes to process that will result in immediate wins with very little effort. We will suggest that one starts taking advantage of knowing where an organization is and using patterns of success that have a high probability of working at the start.

To do this we must understand the nature of software development and look for changes where it is clear immediate improvements will be achievable.

The nature of software development

A lot of people relate software development to art or a craft. There are definitely aspects of that in it. But as in art, there are also laws underneath it. It is interesting to observe the history of the great impressionists and see how their early paintings were very classical in nature. It’s not just that they may have been looking for their own way of painting, it’s more that they were learning the basics of painting. Paint, brushstrokes, canvas all obey physical laws. These must be understood before applying one’s creativity to them.

There are many “laws” of development, so to speak. And all too often these have been ignored in software development. We must also remember that coming up with what to build is often more challenging than implementing it. We will decompose the nature of software development into:

• How to determine what to build, taking lessons from the Lean Startup community
• The principles of flow, learning about Lean flow
• The nature of writing code, learning the rules for creating code

In all of this, we want to remember that we are discussing complex systems so will have to relate these three components to each other. We will learn that they can create downward cycles of ineffectiveness or upward cycles of increasing the amount of value delivered.

Different approaches for beginning a transition

“For every complex problem, there is a solution that is simple, neat, and wrong.” H.L. Mencken

Stop and reflect on what we have discussed so far. It seems there is a dilemma. Systems thinking requires us to look at the whole and yet the limitation on beginning new things means we can’t look at everything. So what do we do? The two most common approaches has us take either of what we consider to be extreme approaches: one, jump to a structure that is known to work well at the team level or two, don’t change anything, create visibility on where you are, and manage the workflow through small incremental improvements. It seems that revolution or evolution are our two choices. But there is another. It is important to understand why both of these approaches have inherent challenges before explaining our approach.

There are two issues to deciding how to start that define possible approaches. These are:

1. Do you start with a set approach and then tailor it or do you start where you are and learn how to change incrementally?
2. Do you attend to few or many concepts?

There is a temptation to prescribe how to start when people are novices. Assuming they are committed to Agile a prescribed start satisfies the desire many have to be told what to do. However, once people reach competency, where do you go next? Sometimes the prescribed start is difficult and has people abandon a lot of the roles and rules they’ve known. So starting where you are and making incremental changes often makes sense too.

There is no question that one can’t start with knowing everything. The question is how do you decide what to start with? The common approach is to select a few key issues and start with them. On the face of it, this makes sense. But there are two challenges with this approach. First, how do you know which the issues should be? But more importantly, if you start teaching an organization what to look at, how do you get them to look at other things later? What we have found is that people tend to keep looking at what they start with.

The Lean-Agile approach breaks with both of these options. First, neither a purely prescribed choice nor a pure evolutionary approach is followed. Instead, a short investigation as to where to start is undertaken and then how to start there is suggested. To do this requires a different approach with how much and what to know as well. Instead of delving deep on some issues that are critical, Lean-Agile takes the systems thinking attitude that all pieces must be considered – but not all to the same depth.

It is suggested that after taking a systems view, we determine what the right issues to look at are, and then drill down deeper in to those. This enables us to focus on the pertinent issues for where the organization is and it also introduces other concepts to look at later when the organization is more competent.

This approach is illustrated in Figure 1.

The challenge with the focused and narrow approach is that many people never get beyond what they look at at the start. We have seen many organizations miss straightforward solutions simply because they didn’t look outside of the narrow focus the approach they started with discussed. While many claim the methods aren’t to blame, the frequent pattern of not investigating options outside the standard starting points make me think otherwise.

The levels of the organization

There are three main levels of a software development organization – portfolio, program and team. In large organization you could consider a sub-program level also exists. Scrum has typically taken the approach of starting at the team and moving up to the higher levels. Kanban often starts at the team or program level. This bottom up approach is not always the best approach and can actually cause problems later. When considering which level to start with you must consider:

1. At what level does the person driving the transition live?
2. At what level do the first challenges to work on exist

One of the challenges that many organizations have when undertaking an Agile transition is that the level that drives the change is not the level that needs the initial change. There is also a tendency for the level driving the change to talk in their own terms instead of the terms of the people they need to enroll. For example, if business stakeholders are overloading teams and the place to start is to have them be more selective on what programs are undertaken, talking to them about the benefits of Agile will likely fall on deaf ears. They are more interested in delivering business value quickly than learning about cross-functional teams and iterations.

In addition to these, one must remember that starting at one level, even successfully, without consideration for the impact it may have on other levels, can be counter-productive.

Starting a transition to better methods of software development

The decision of how to start requires us to acknowledge the following reality:

• People want a well-defined place to start
• We are finite: we can’t do everything
• Some things will change the environment within which people work which will make other things easier to change
• We must attend to the nature of software development
• We must respect our culture and where we are
• Our best starting point is likely somewhere between doing nothing at first to jumping to a predetermined solution

The first step is to look at which layer at which to start: business stakeholder, program or team? It is useful to see what starting at each of these levels would look like and then see which starting point would be most effective given the above issues while considering the potential benefit of each potential starting point. This requires being aware of the forces present to attend to. There are not that many and the most important ones center around delay in the workflow and in feedback cycles. These types of delays are caused primarily by:

1. Too many things being introduced into development
2. What is being introduced into development is too large
3. Colocated, cross-functional teams do not exist forcing people to wait on others
4. There are too many things in play at any one time
5. Teams are not coordinating well so that they work on related things at different times causing integration errors later
6. Developers and testers are not coordinating well

Each of these has a ready solution. Remember, however, the question isn’t what to do, the question is which actions are achievable given our culture and current situation and which set up the next steps. Each of the above challenges has the following as first-order solutions, respectively:

1. Institute a pull system by development where work does not get to the team except when it pulls it from a sequenced queue
2. Use minimal business increments to work on the most important items
3. Create colocated cross-functional teams to the extent possible
4. Set limits on Work-in-Progress (WIP) on each stage of the work flow (including queues)
5. Teams coordinating with each other should pull the work in a coordinated fashion so that work does not begin until all the necessary people and resources are available
6. Use acceptance test-driven development

We don’t want to address all of these issues at the beginning, of course. But we should be aware of if they are there. We can ask a series of questions to see where we should start (these relate to the prior list in their respective order):

1. Will management cooperate to restrict work hitting the team?
2. Can we convince stakeholders to use minimal business increments (MBIs)?
3. To what extent possible is it to make committed, cross-functional teams without adversely impacting the rest of the organization
4. Do we have a stable enough system where setting WIP limits will do us some good?
5. Do we have teams that are working together? Can we use common backlogs so that they work on things in a coordinated fashion?
6. Are our product owners/Business Analysts, programmers and testers willing to use Acceptance Test-Driven Development?

By recognizing we need to consider all aspects of the system yet knowing we must phase them in over time, we start at a high level and drill down only when there appears to be benefit.

What to do after starting

After teams get started, they face new challenges. One of them is that if they are successful, much of the motivation will be lost since the pain which got them started will be reduced. But this gives a clue as to how we need to guide the transition. It is not about getting better and better at our practices. It is about getting better and better in our outcomes. The outcomes are twofold:

1. Deliver more value faster
2. Improve as a learning organization

By focusing on values, we remove the need for pain to drive us. We always strive to get continuously better from where we are, yet we can be proud of what we’ve achieved. Our shift at this point goes from practices to the outcomes we need to achieve. This enables us to explore different practices to achieve better results.