How to apply Agile Principles to large scale transformations
I have been asked if Agile methodologies can be scaled successfully. The obvious answer is yes, for if they can not be scaled the methodologies have no agility. In reality the scaling process does involve methods that are additional to those identified as Agile Software Development.
These methods work well with and are closely aligned to agile principles. They are closely associated with the origins of Agile and Lean methods. They incorporate Kaizen which “means that all team members throughout the organisation are continuously looking for ways to improve operations”
The roots of Agile and Lean methods are founded the Toyota Production System of the 50’s. Around 1990 a detailed outline of the entire Lean system was explained in The Machine That Changed the World. At this time a problem existed in software development. Known as “the application development crisis” it referred to the gap between specifying requirements and delivery. At the time the gap averaged 3 years, but in some instances the gap was as big as 20 years. Against this background:
- In 1999 Kent Beck published Extreme Programming Explained: Embrace Change
- In 2001 the ‘Snowbird Meeting’ was held in Utah, where the authors of the Agile Manifesto met.
- In 2003 Lean principles were presented, as an Agile tool, in Lean Software Development.
Although it is common to view refer to Agile Practice and Lean Principles as just Agile and Lean (which I will do in subsequent parts of this post) both word are adjectives and should be followed by a noun (or the noun phrase) that they are qualifying.
Lean is built on the fundamentals of Purpose, Process and People. It recognises the need to do something as stemming from a problem. It asks if the methods of overcoming the problem are: valuable, capable, available, adequate and flexible. It requires that everyone is engaged in a process of continuous improvement.
The principles of Lean are:
- Eliminate waste.
- Amplify learning.
- Decide as late as possible.
- Deliver as fast as possible.
- Empower the team.
- Build integrity in.
- See the whole.
Agile is built on:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
The principles of agile are:
- Deliver early and often to satisfy the customer.
- Welcome changing requirements.
- Deliver working software frequently.
- Business people and developers must work together.
- Trust motivated people to do their jobs.
- Face-to-face conversation is the most efficient and effective method of conveying information.
- Working software is the primary measure of progress.
- Maintain a sustainable pace.
- Continuous attention to technical excellence and good design enhances agility.
- Simplicity—the art of maximizing the amount of work not done—is essential.
- The best architectures, requirements, and designs emerge from self-organizing teams.
- Reflect and adjust at regular intervals.
Lean proposes a holistic system as does Agile. Lean is not confined to manufacturing but can apply to many business activities, it is well established in service and software industries. Agile is equally not confined to the production of software. Despite being initially defined by programmers and intended to solve problems in creating software, Agile has been successfully adopted by other practices.
Employing both Lean and Agile principles provides benefits to wider and more general technology projects. They can also be applied to other business processes. The difference between Lean and Agile are not too far apart but as Lean has a more general set of principles, it are easier to scale.
As an example of how Agile/Lean methodologies can scale consider a largish project. A dynamic and successful business finds that it has grown out of the technology it uses. It employs 500 people, operates globally and is located across continents. It has a core platform for delivering its product but this can no longer cope with the capacity. It uses a number of other platforms, for different subject areas, that are also not performing adequately.
The core platform needs replacing and at the same time many of the fundamental subject areas require new systems that match the growth and ambition within the business. All functional areas can radically improved with new technology but the driver is the real pain and problems that are holding the whole company back.
Conventional Agile methodology as it applies to software development is not relevant to the scenario above. The example involves a program of change and implementation across workstreams and the business as a whole. It involves hardware, infrastructure and new ways of doing things. Agile methodology alone is unable to cover the whole program.
The fix is not to scale Agile teams but to scale the projects by adding teams and including them within the wider Lean program. This does not mean that Agile principles get thrown out. Instead they sit in alongside the Lean toolbox to reduce disruption, waste, cost uncertainty and delays.
A business, or even a business product will consist of a number of technologies and applications. Together these form a system which is made up of interrelated and dependant sub systems. A typical business will have a number of functional areas, for example: Finance, Sales, Operations, Marketing, HR. Each subject area will have a set of technologies specific to their needs but will also need access to cross-functional systems.
A subject area will have its own subsystems, finance for example may have: invoicing, purchase, accounting and payroll. These may be subdivided, invoicing may have a ledger that links to the accounting functionality, it may also have an application that provides invoice support breakdowns derived from the operational sales application. Work products from the accounting function will interact with other business systems and company wide reporting.
In the transformation program a subsystem could have its own agile teams delivering internal or external vendor supplied applications. It may have started with a need for one team delivering the features to support invoicing, this team may begin to grow in size to add additional functionality. At this point the opportunity occurs to split the team into two. Another agile team may then be assigned to provide payroll functionality.
The teams need to be connected while remaining distinct design and production units. This will lead to a number of feature based teams. The teams are agile. The business needs a way of connecting these, and other, teams to deliver a coherent system. The business/customer also needs a method whereby these teams can be managed efficiently and their outputs integrated into a rational whole.
Empowered teams deliver quickly, reduce waste and build integrity into their products. Lean management principles allow this. The arena where; the visions are shared, feedback is communicated, insights are revealed, value recognises and waste avoided; scales accordingly. The scaled structure for the transformation is not hierarchical but is centered around the program.
In the example above the main sets of tasks are born from the need to replace the core platform. It is too big a task for a single team, agile only works well with small teams (5-12 members). One definition of the ideal size for agile teams is if it is too big to share two pizzas it is too large.
Any team that gets bigger than 11 members becomes inefficient. This does not mean program is confined to projects of less than 11 workers. It just means that teams need to be split before they become inefficient. The team of ten can exist but it is also an opportunity to become two teams of two.
We do not want the teams going in opposite directions or duplicating work effort so someone needs to coordinate them. This is the role of project management. The role assumes the link between the team and the client (or for internal teams the business). The agile practices of:
- Deliver early and often to satisfy the customer
- Trusting motivated people to do their jobs
- Face-to-face conversation being the most efficient and effective method of conveying information
- Working product being the primary measure of progress
still apply. Alongside them sits the Lean principles of:
- Deliver as fast as possible
- Empower the team
- Build integrity in
- See the whole
The difference is they are now reside within an implementation framework rather than a pure production framework.
Big projects will spawn many teams. If it is too many for a single project manager to handle it will require someone to sit over the project managers and pull them together. At this point the project has in fact become a program and requires a program manager. The program manager will possibly be involved in other transformations as well. These other transformation will also be able to benefit from agile principles of;
- ‘Deliver working software frequently’
While this does not apply to transforming the hardware or network infrastructure the principle of
- ‘Eliminating waste’
applies. The larger the program the more waste there is to eliminate.
As complex systems are made up of subsystems and components. Frequent product releases, prototypes or demos of the subsystems and components gives flexibly and agility. This is achieved by specifying performance and functional requirements at a high level. An early release allows the product design and implementation to respond to feedback.
Delivering value early through iterative and incremental processes provides opportunities for feedback. Feedback is critical to allowing design options to be made later in the project lifecycle. Keeping options alive and open is a way to continuously reduce uncertainty. Risks and errors can be identified and mitigated early in the project lifecycle.
Options will be influenced by the iterations and the result of testing releases. Aside from providing incremental improvements this gives additional ways to reduce risk and cost. Faults detected early in one project save waste as they can be fixed before they affect too many other projects and parts of the system.
Parts of the system may have functionality that is replicated in other parts. It is useful to find duplication of features early. As an example; issue tracking, project tracking and task tracking use similar functionality and could be broken out into a single service rather than being built into separate sub-systems.
Agile principles are not just for software
Delivering working software frequently eliminates waste by delivering defects early and providing the opportunity to fix them. This allows fault recovery to be built into the process. Having a prototype for soft configurations of hardware allows settings to be fine tuned. Providing early access to hardware installations will provide an opportunity to test the hardware, simulate load and performance recognise weak spots and build in recovery.
Applying a ‘test early’ approach to hardware could involve deploying a demo version of the hardware. This would allow it to be load tested and run as a prototype installation. In this incarnation it may suggest the design for a ‘failure recovery’ plan. As hardware is more prone to breaking more than software identifying in a strategy for recovery gives value.
For example if the hardware was a mobile phone. Having taken delivery of a particular model as a demo would allow the assessment of how to:
- Configure it.
- Test its performance in the field.
- Evaluate the options for backing up data.
- Time the sequence required to mirror the configuration.
- Run a test of reinstating the backup.
This gives procedures and information that will could save future waste. The performance may also suggest a additional infrastructure or a different model at a stage where changing specification does not have a significant impact on the integrity of the whole transformation. Having demonstration becomes similar to the agile software practice of delivering early. It also allows a window to respond to change.
In the above example the capability of the hardware demonstrated early may reveal other capabilities that hold value such as having: a built in voice recorder, voice recognition, the capability for conferencing or navigation. Discovering this would prevent duplication within other program work streams. Capabilities that without the demo would also be invisible to other workstream and project managers.
Using the Agile principle of ‘Deliver early’ when adopting hardware or provisioning network provides the opportunity for sharing insights across the business. This could lead to an awareness of other options early enough to prevent waste. Again as an example of how waste could be reduced, if a network solution allowed distributed data storage then the need for an internal backup would be superfluous.
Decide as late as possible.
It would be foolish to decide and then research but again this is a common sequence of events. A technology is acquired, installed and then the business starts to learn about it. Technology should be trialed before decisions are made about deploying it.
There are many sales department that have gone through many different CRM applications because they decided on which CRM they wanted before they had decided how they were going to use it. All subject areas, not just sales, are prone to this syndrome.
The advantages of delivering early can be negated if deliveries are not shared. We have all heard “If I knew it did that I would not have bought this”. I have seen complex ERP systems put in place with overlaps of functionality to parallel systems. Yet the overlaps were not apparent until the end of a very long; specification, configuration and implementation chain.
The number of times I have seen a business where the same thing is done in different ways by different people is amazing. Subject areas will frequently commit to a technology and not implement it for some time. This can lead to instances of functional areas having solutions that never get out of the box due to some other area already having a package with the same functionality.
It is not unknown for projects to overlap, deciding late does not prevent this but it can help by giving more time to share the knowledge, achievements and pitfalls. Procuring a system that is dependant on a second system relies on both systems being coordinated to arrive at the same time and to arrive as expected.
A system may change quite a bit between specification and delivery. The decision to procure a system should only be made once the system it is dependant on is in a stable form. Implementing a system that interfaces with other systems is a case when pushing back decisions is a good strategy.
Dependencies are not confined to technology. If it is possible that a work stream is likely to go in more than one direction before settling on a final place, the technology to support that work stream should not be decided until there is certainty. There is no point implementing a help desk system if support is going to be outsourced.
Agility is also required here to prevent the tail wagging the dog and a willingness to embrace change is required. Deciding as late as possible should not lead to, or be confused with, indecision. The extent of discovery will point out when is the correct time to decide. This is usually when dependencies are in place and a good domain knowledge has been acquired.
Managing the project managers
As projects get subsumed into programs and makes yet another Lean principle ‘seeing the whole’ ever more important. Project Managers tend cover one type of activity e.g. production or implementation. The role of a Program Manager may cover production alone but more often than not it will cross production and implementation. On large programs the Program Director will cover whole transformation.
The success of programs is not dependent on micro management but on;
- ‘Empowering the teams.’
- ‘Incorporating integrity.’
- ‘Seeing the whole.’
Attempts to run plan based programs are on most occasions doomed and will lead to a lot of wasted effort. Requirements will change and they will change often as more insight into the new platforms are discovered. This does not mean there should be no planning, just that planning should be a continual process carried out throughout the life of the program.
As new possibilities emerge.
- ‘Deciding as late as possible’
- ‘Welcoming change.’
are the Agile and Lean principles that will encourage success and reduce wasted effort.
Trying to create a static plan for a large change to the technological ecosystem of a business is neither possible nor desirable. On the other hand planning processes to:
- Implement change.
- Mitigate the risks of change.
- Form strategies to simplify into small manageable steps.
is very possible. Enabling individuals and teams to become completely engaged in very desirable.
Being agile is not about being blind and not planning it is about looking down the road for potential disasters and planning the steps to avoid them. Being Lean is recognising values that can be measured and evaluated. Planning paths that can be adjusted and re-routed as the impact of implementation is assessed.
It is important to have strategies in place to deal with the unknown. Every implementation of an enterprise level systems is different, the outcomes are unknown but this does not mean they should be leaps into the dark. Plan the first iteration, test it, measure outcomes, learn from the test.
On the subsequent iteration apply the knowledge gleaned from the first iteration, change and improve. The first iteration may or may not need to be refactored. Either way, once completed, released and evaluated, it will cast some light on the next step. It will show if variations from the path are needed, it will illuminate potential collisions.
Instead of swallowing the elephant whole the program should be seen as natural divisions that are interdependent and can each be controlled. Each step should independently provide value and avoid waste. If it does not provide value it can be abandoned. There is less waste in abandoning a small step than a large step or a lot of steps. Small iterations require and less effort needed if the course needs to be redirected .
Some of the tools used to manage large programs may be different to those used on smaller projects. The main aims are for simplicity, visibility and that to encourage collaboration. It is impossible for a single person to know what 100 developers are doing and what progress they are making towards the desired and result.
It is possible to know that there is interaction between 20 agile or lean teams. It is possible that communication channels exist to share knowledge and enable best practice. It is possible to see milestones being met, the road map being traversed and to view achievements via hard data and metrics. It is possible to encourage, obtain and allow responses to feedback.
There are many methods within the total framework of agile methodologies and in the main they encourage responsibility, engagement and communication. The most important aspect of an agile program is to enable interaction between individuals. This interaction can be scaled to almost any size but it is not achieved by taking a small team and making it bigger.
The military have discovered optimal team sizes to be between around five t0 11 and recommend breaking large teams (7) into two.
The Agile/Lean methodology begins by scaling communication.
The instigation of a program is from the businesses managers and executives. It is they who plan change and who understand the need for improvement. They will have the clearest view of the difficulties that their existing systems are causing. They will have a vision of where the business can succeed most once the constraints of the old systems removed. They have the measures of success.
It crucial that the vision of the business and the program are aligned. This is where whoever is leading the program will establish the benefits of scaling Agile/Lean methods. The initial process involves lines of communication, in the first instance these should come from the users of the system. Finding their problems and discovering where improvements can make is the root of all specification.
Sharing the users problems with the project teams gives them an opportunity to propose solutions. This is the point when the business as a whole becomes involved, the executive can now be shown a problem, the source of the problem, a solution and an estimate of the effort that is needed to provide the solution. The business is then in possession of the information required to judge the value and form decisions based in information.
The sub projects will begin to fit together as number of simple functional features. The business may decide that a feature set would give distinct commercial value if it was changed to include different or additional features. This is a point where many technologists and engineers on the teams will groan.
Despite having subscribed to ‘welcoming changing requirements’ many project teams dislike having their work to date altered, they may see it as waste. Iterative methods allow value to be clearly communicated back to the teams. The advantages of fast delivery and quick feedback are; reduced waste and a visibility of the reasons for changing requirements.
Teams thrive on seeing improvements and delivering value. As they were involved in designing the solution and delivering an realease they are motivated. Early feedback will show them how a change in requirements improves the product and increases value. It also gives the business an early chance to evaluate the success of the program as it progresses rather than waiting for it to end.
Agile and danger
Are there dangers in using an Agile and a Lean approach on large scale programs? Any large transformation brings risks and danger but using an Agile/Lean methodology to approach big changes allows risks to become evident early on, giving opportunities to mitigate and reduce them as the projects develop.
The big danger in having a large plan is that it takes a long time to get to a point where errors and mistakes are evident. It takes a lot more effort to change the direction and it is harder to respond. The more people involved the harder it becomes. On the other hand flexible and adaptive teams who have devised their own plans can respond to both need and change.
The program manager leading the big change is not there to instruct those delivering the change. They are not there to tell the business what direction it should take. They are there to assist the teams to deliver value to the business. They are there to communicate to the business the ways in which problems can be removed and where new solutions lead to.
They discovered that the role of a leader is to serve those they lead. This is reflected in the motto of Sandhurst – ‘serve those they lead’.