New to Agile or Scrum?
For decades the software industry has been dogged by failed deliveries and late projects. Budgets and schedules ran out of control and, sadly, this is not a rare occurrence. Some studies estimate the project failure rates in the 60-70% range !
This inability to deliver what the business needed was frequently the cause of tension between the business and IT across many organisations. The perception of the abilities or capabilities of IT to deliver appropriate solutions to time and budget was low.
For years, the standard modus-operandi for delivery was simple : make a plan, and work through that plan to fruition. This approach is commonly referred to as ‘The Waterfall Methodology’. Where there was any deviation from the plan, this was considered a ‘fault’. It was either the fault of the Project Manager for not anticipating every eventuality or the customer for not having a full and clear understanding of all the requirements that needed to be delivered.
As you can imagine this approach led to many disputes between business and IT as well as a number of unexpected side-effects:
- Hiding the bad stuff : covering up when things were not going well.
- Playing the blame game : blaming each other for failures when the project slipped.
- Documentation overload : IT and business functions each went to great lengths to ensure that all correspondence was documented and provable so that when things go horribly wrong, there would be traceability. what is particularly interesting to note was the expectation that something would go wrong often became a self-fulfilling prophecy.
- A gulf between teams : the relationship between IT and business became somewhat adversarial. I remember as a young developer, how much time my PM used to take to prepare for his weekly ‘battle’ (meetings) with the customer. Most of his time was either spent preparing for, attending or worrying about the next meeting.
- Control the chaos : all the above results in a mistrust of IT, which can lead to a reactionary approach to micro-manage, and ‘crack the whip’ on under-performing teams, which most teams would be – using this approach. Many of my early projects felt as though we were battling through the chaos. Reviewing the above, it’s easy to see why.
These symptoms of the Waterfall Methodology where obviously problematic for the industry, but, the question remains at a fundamental level, what went wrong ?
What Went Wrong ?
The above really states the symptoms, rather than the actual causes of the problems. In fact the problem was a fairly basic one. For this I must briefly introduce you to some process control science concepts.
Without going into too much detail, within process control science there are two main types of processes :
- Pre-defined processes.
- Empirical processes.
A Pre-defined process refers to a process where the variables, parameters, and unknowns can be controlled to a large degree. In other words it is a ‘simple’ environment. For example, if you put a piece of wood of a certain type, quality and density into one end of a machine, you should get an identical widget at the other end. This process is repeatable and whilstthere may be some variation in quality, this should be minimal. What variance there is can be reduced through various techniques, such as Six Sigma.
An Empirical process on the other hand refers to a process which is applied in a ‘complex’, unpredictable environment. The variables are so large or numerous that human beings are not able to pre-determine the outcome of a single approach with any reasonable degree of accuracy. This process is also frequently referred to as an ‘Inspect and Adapt’ approach.
So, the approach of delivering traditional software projects using a waterfall approach is an example of a pre-defined process designed for use in a simple environment. Software development, however, is a complex activity with multiple uncertainties, unknowns and variables spread across the people, process and technology domains. Simply put, we were using the wrong process type for the job.
Obtaining a Predictable Outcome in an Unpredictable and Complex Environment.
I suppose the next question that may arise is, given this, how can we obtain a predictable outcome in an uncertain, chaotic or complex environment with unknowns spanning people, process and technology ?
Inspect and Adapt Cycle – a Stepwise Approach
In answer to the above question I’ll say this: move away from a pre-defined process approach to an empirical approach. All Agile approaches have this in common. They are based upon an ‘inspect and adapt’ process, which proceeds stepwise, reviews the results obtained and then plans and executes the next step.
Baking Quality Into the Process
The concept of designing quality into a process is not a new one. In fact, Edwards W. Deming, the father of process control science and quality talked about the need for building quality into the process as far back as the 1950s.
Agile methodologies seek to build quality early in the development process, as opposed to the Waterfall approach of a single testing phase towards the end of the project life cycle. The XP (Extreme Programming) practice of test-driven development is an example of introducing quality into the development process. This is by no means the full range of quality processes required in an Agile project, but it is perhaps the most vivid application of Dr Deming’s principle of designing quality into the process.
Once again Deming introduced the concept of ‘Continual Improvement’ which refers to general processes of improvement which encompasses many different approaches, covering different areas.
Continuous Improvement is a subset of continual improvement with a more specific focus on linear incremental improvement within an existing process. Retrospectives within Scrum are a modern software management ceremony, the objective of which is to learn from earlier steps within the process and apply this learning to aid improvements going forward.
Some Delivery and Management Methodologies
Below you will find a list of common agile delivery and management methodologies. Although this is not a comprehensive list, it’s a good start and covers the basics.
Scrum is a lightweight Agile delivery methodology. It’s based on principles of small releases, continuous improvement and customer collaboration.
Extreme Programming (XP) is a group of 12 software engineering practices that are beneficial to software delivery teams. This includes practices frequently used in other Agile
methodologies such as test-driven development, small releases, customer collaboration and pair programming.
Dynamic Systems Development Method (DSDM) Atern is an Agile project management methodology which may be used either in addition to Scrum or on its own, as it has its own
team-level development methodology.
MSP (Managing Successful Programs)
A program management methodology that consists of a set of principles and processes for use in large and often complex programs.
The Information Technology Infrastructure Library (ITIL) is a set of concepts and practices for service management.
PRojects IN Controlled Environments version 2 (PRINCE 2) is a robust project management approach that is primarily associated with traditional projects, but aspects of this have been known to be applied in an ‘Agile’ context.
Structured Systems Analysis and Design Method (SSADM) is a systems analysis and design method with a prescriptive approach for traditional waterfall projects.
Where Should I Use an Agile Approach ?
Today’s world is characterised by rapid change. This means that you cannot pin down requirements a number of months in advance and hope that they will remain unchanged. Additionally technology and market forces are moving at such a pace that they introduce complexity in ways that cannot always be anticipated or planned for.
This means that the vast majority of software projects will benefit from an agile approach.
Where Shouldn’t I Use an Agile Approach ?
There may be some very small simple projects where the business domain is well understood, requirements unlikely to change, and where it may be simpler, and may not introduce significant risk to take a waterfall approach, however, these kinds of projects are pretty rare.
Other examples are where you are unable to meet the fundamental requirements of agile or empirical processes such as :
- Cannot collaborate closely or frequently with the customer for whatever reason.
- Secrecy may not allow open visibility and communication amongst different teams or members of the same team working on different components, e.g. top secret defence projects.
- Requirements are such that there are a large number of stakeholders that need to approve each requirement, and there is a strong need for traceability, monitoring and general consensus on each point. e.g. Nuclear Power Plant Control System.
- Inability to or unwillingness to devolve decision making authority to the ‘do-ers’. This is common in many organisations that have a deep seated command and control approach to management of teams and individuals within teams
Whilst applying Agile methodologies or approaches to the delivery of software projects may be relatively new, the basis of Agile dates back to the 1950s.
- Understanding the underlying principles of Agile will help in understanding why certain practices exist, certain approaches are used and where they are appropriate in specific contexts and where they should be dropped.
- The very basis of empirical approaches is adaptability in the face of unknown challenges, risks, or obstacles. This should always be remembered, and Agile processes (such as XP, Scrum, DSDM, or any other), once fully understood, should never be given preference to common sense.
- An Agile approach is not suitable for every environment, though the complex nature of project delivery with the unpredictable ‘people’ aspect and fast changing ‘technology’ aspect mean that for a majority of projects, this approach will probably lead to increases in quality, lower costs, and a faster time to market.