Agile Software Development is more than just Sprints and Scrum meetings

These days, most of the companies I've either consulted, contracted with, or been employed at claimed to do Agile software development. By this, they mean that they have regular cycles in which they plan their software delivery and have regular meetings in which they discuss their status. But many of them miss the true point of Agile development, which is to focus on just-in-time development and continuous improvement.

To see why so many companies get Agile wrong, let's first delve into what Waterfall is and how it differs from Agile.

Waterfall

Waterfall management focuses on making sure that each step in the software development process is complete before moving on to the next step. This diagram shows the typical Waterfall process:

1. Gathering Requirements – During this stage, the software team would get together with the business leaders users to determine what the business need is. All portions of the business need at hand are explored, including (but not limited to) overall need, exceptions to various rules, competitors’ methods, current method to ease the pain, etc.
2. Design Software – At this stage, the software team will take the business requirements and create documentation in the form of descriptions, mockups, and prototypes to describe what the software should do after completed. The business team is not allowed to change the overall strategy or direction of the project.
3. Architect/Build Software – At this point, the team knows what features will be in the software and so it is ready to be built. The development team creates a plan, then creates the software.
4. Testing – Once the software is deemed complete, the completed product is sent to a team of testers, who verify whether the software does what it is supposed to do. The development team is allowed to fix any issues, but they are not allowed to add any features.
5. Deployment – The software is pushed to end users. If features need to be added or changed, they go through the same process as the original project.

What's Wrong with Waterfall

The Waterfall approach appeals to managers, project managers, and businesses because it feels predictable and controllable. But in reality, business users are bad at knowing how to ask for what they want, and IT professionals are bad at asking questions that get at this information. The result is that the software requirements have some fatal flaws, and this can be extremely expensive to fix:

	Time Detected
Time Introduced	Requirements	Architecture	Construction	System Test	Post-Release
Requirements	1	3	5-10	10	10-100
Architecture	-	1	10	15	25-100
Construction	-	-	1	10	10-25

Table from Code Complete by Steve McConnell, page 31

In a typical Waterfall environment, the technology consumers might not see the finished product until the system goes live. Therefore, if a requirement is missing or misunderstood, it might not be found until the “Maintenance” phase. As you can see, these are the most expensive bugs to fix.

Agile

Some software engineers saw the persistent failures present in software projects driven by Waterfall methodologies and came up with a different approach. They called it "Agile", the backbone of which can be seen in this quote from the Agile Manifesto site:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

(http://agilemanifesto.org/)

The result is a much more iterative product development process:

As you can see, the basic steps for Agile are the same, but instead of expecting that each step is complete before one can start the next one, all of the steps in the development cycle are more or less completed simultaneously. One team may be testing a portion of the system that is deemed to be complete at the same time that another part of the system is being developed at the same time that yet another part of the system is being designed. In true Agile, the business users are encouraged to work with the product long before it is ready to be deployed to a wider audience. While this means that the technology consumers have more work to do after the initial requirements have been gathered, they have greater opportunities to give feedback and fix misunderstandings that arise.

To put the difference between Waterfall and Agile a different way: Waterfall, when done correctly, is about managing the process so you can control timelines and scope, and deliver software as efficiently as possible. Agile, when done correctly, is about being flexible and reactive to changes to ensure that whatever is delivered is both needed and used.

But in the real world…

The Agile development model is antithetical to the entire mindset of managers in general. Managers' jobs are to make processes more efficient, where Agile, by design, prevents this type of control. Perhaps the most common way teams try to implement an Agile process is to gather all of your requirements and complete your design at the beginning like Waterfall, but then provide portions of the project in short iterations. The process looks a bit like this:

The result of this process typically is that you get some of the benefits of Agile, primarily the early feedback, but you get some of the benefits of Waterfall, such as budgeting and project planning. Make no mistake, though, if you have a target date or a budget, you are not doing Agile. In taking this approach, you still have the problems related to poor requirements up front, and by insisting upon tracking time and budget you are still taking focus away from the success metrics that really matter. In some cases, especially when you lack trust between the software delivery team and the team that will be using the software, you may want to take this approach. Few things build trust as easily as making tangible promises related to time and budget. But once you are ready, you will operate more smoothly and efficiently if you adopt a true Agile methodology sooner rather than later.

If you're serious about using Agile

If you want to reap the benefits of using Agile, you really should do most of the following:

• Getting feedback early and often is essential - It is a frightening proposition to give access to incomplete software to your users. The risk that they will react negatively to missing functionality is high. And the truth is, if your customers aren't used to Agile, the first iteration or two will involve some difficult conversations. But as your customer sees that their concerns are being addressed, the conversations will get easier, and you won't implement features that users hate.
• Forget about traditional project management - The purpose of project management is to ensure that software is delivered on time and on budget. However, the true value of software should be measured via other metrics. Focus instead on value and customer satisfaction. Sometimes scope will expand and you will end up spending more. Other times, though, you'll realize that you didn't need all those fancy features and you will save money. Either way, customer satisfaction will be high and you will have received value for your money and effort. Isn't that what really matters?
• Strive for better, not perfect - Too many times a good idea on paper turns out to be a horrible idea in software. If you try to make the idea perfect, you will probably put in something that is too convoluted to use. Instead, make the software better, implementing what you can of your idea. This way, you can adjust the design as you get feedback from users.
• Communicate, communicate, communicate - If the business team is communicating frequently with the software developers, the business team will get a general sense for what is easy (and hard) to change, while the developers get a sense for why certain functionality is used. This significantly reduces the amount of work per business need met since business users have a better starting point and developers can anticipate issues more easily.
• Finally, if you're purchasing a tool to implement Agile, you're doing it wrong - Agile is a mindset, not a tool. You can't purchase software that will change the hearts and minds of your employees.