Agile Software Development is more than just Sprints and Scrum meetings

These days, most of the companies I've either consulted, contracted with, or been employed at claimed to do Agile software development. By this, they mean that they have regular cycles in which they plan their software delivery and have regular meetings in which they discuss their status. But many of them miss the true point of Agile development, which is to focus on just-in-time development and continuous improvement.

To see why so many companies get Agile wrong, let's first delve into what Waterfall is and how it differs from Agile.

Waterfall

Waterfall management focuses on making sure that each step in the software development process is complete before moving on to the next step. This diagram shows the typical Waterfall process:

1. Gathering Requirements – During this stage, the software team would get together with the business leaders users to determine what the business need is. All portions of the business need at hand are explored, including (but not limited to) overall need, exceptions to various rules, competitors’ methods, current method to ease the pain, etc.
2. Design Software – At this stage, the software team will take the business requirements and create documentation in the form of descriptions, mockups, and prototypes to describe what the software should do after completed. The business team is not allowed to change the overall strategy or direction of the project.
3. Architect/Build Software – At this point, the team knows what features will be in the software and so it is ready to be built. The development team creates a plan, then creates the software.
4. Testing – Once the software is deemed complete, the completed product is sent to a team of testers, who verify whether the software does what it is supposed to do. The development team is allowed to fix any issues, but they are not allowed to add any features.
5. Deployment – The software is pushed to end users. If features need to be added or changed, they go through the same process as the original project.

What's Wrong with Waterfall

The Waterfall approach appeals to managers, project managers, and businesses because it feels predictable and controllable. But in reality, business users are bad at knowing how to ask for what they want, and IT professionals are bad at asking questions that get at this information. The result is that the software requirements have some fatal flaws, and this can be extremely expensive to fix:

	Time Detected
Time Introduced	Requirements	Architecture	Construction	System Test	Post-Release
Requirements	1	3	5-10	10	10-100
Architecture	-	1	10	15	25-100
Construction	-	-	1	10	10-25

Table from Code Complete by Steve McConnell, page 31

In a typical Waterfall environment, the technology consumers might not see the finished product until the system goes live. Therefore, if a requirement is missing or misunderstood, it might not be found until the “Maintenance” phase. As you can see, these are the most expensive bugs to fix.

Agile

Some software engineers saw the persistent failures present in software projects driven by Waterfall methodologies and came up with a different approach. They called it "Agile", the backbone of which can be seen in this quote from the Agile Manifesto site:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

(http://agilemanifesto.org/)

The result is a much more iterative product development process:

As you can see, the basic steps for Agile are the same, but instead of expecting that each step is complete before one can start the next one, all of the steps in the development cycle are more or less completed simultaneously. One team may be testing a portion of the system that is deemed to be complete at the same time that another part of the system is being developed at the same time that yet another part of the system is being designed. In true Agile, the business users are encouraged to work with the product long before it is ready to be deployed to a wider audience. While this means that the technology consumers have more work to do after the initial requirements have been gathered, they have greater opportunities to give feedback and fix misunderstandings that arise.

To put the difference between Waterfall and Agile a different way: Waterfall, when done correctly, is about managing the process so you can control timelines and scope, and deliver software as efficiently as possible. Agile, when done correctly, is about being flexible and reactive to changes to ensure that whatever is delivered is both needed and used.

But in the real world…

The Agile development model is antithetical to the entire mindset of managers in general. Managers' jobs are to make processes more efficient, where Agile, by design, prevents this type of control. Perhaps the most common way teams try to implement an Agile process is to gather all of your requirements and complete your design at the beginning like Waterfall, but then provide portions of the project in short iterations. The process looks a bit like this:

The result of this process typically is that you get some of the benefits of Agile, primarily the early feedback, but you get some of the benefits of Waterfall, such as budgeting and project planning. Make no mistake, though, if you have a target date or a budget, you are not doing Agile. In taking this approach, you still have the problems related to poor requirements up front, and by insisting upon tracking time and budget you are still taking focus away from the success metrics that really matter. In some cases, especially when you lack trust between the software delivery team and the team that will be using the software, you may want to take this approach. Few things build trust as easily as making tangible promises related to time and budget. But once you are ready, you will operate more smoothly and efficiently if you adopt a true Agile methodology sooner rather than later.

If you're serious about using Agile

If you want to reap the benefits of using Agile, you really should do most of the following:

• Getting feedback early and often is essential - It is a frightening proposition to give access to incomplete software to your users. The risk that they will react negatively to missing functionality is high. And the truth is, if your customers aren't used to Agile, the first iteration or two will involve some difficult conversations. But as your customer sees that their concerns are being addressed, the conversations will get easier, and you won't implement features that users hate.
• Forget about traditional project management - The purpose of project management is to ensure that software is delivered on time and on budget. However, the true value of software should be measured via other metrics. Focus instead on value and customer satisfaction. Sometimes scope will expand and you will end up spending more. Other times, though, you'll realize that you didn't need all those fancy features and you will save money. Either way, customer satisfaction will be high and you will have received value for your money and effort. Isn't that what really matters?
• Strive for better, not perfect - Too many times a good idea on paper turns out to be a horrible idea in software. If you try to make the idea perfect, you will probably put in something that is too convoluted to use. Instead, make the software better, implementing what you can of your idea. This way, you can adjust the design as you get feedback from users.
• Communicate, communicate, communicate - If the business team is communicating frequently with the software developers, the business team will get a general sense for what is easy (and hard) to change, while the developers get a sense for why certain functionality is used. This significantly reduces the amount of work per business need met since business users have a better starting point and developers can anticipate issues more easily.
• Finally, if you're purchasing a tool to implement Agile, you're doing it wrong - Agile is a mindset, not a tool. You can't purchase software that will change the hearts and minds of your employees.

Leadership vs. Management

Why do projects go wrong? There are a lot of causes, ranging from a poor understanding of what the business need is to poor execution from the delivery team to lack of communication between the two groups. A lot of these problems could be alleviated through effective leadership and management of software developers and their projects. This is the first in a series of posts that describe how this came about and what we can do about it. This month, I’m going to kick off the series by talking about the differences between leadership and management and why both are important.

People often confuse leadership and management. Or rather, they assume that with one comes the other. Except the two are quite different – and getting it wrong can cause problems for your project, team, or company. To see why, let’s compare the two concepts. (The ideas of leadership vs. management come from John Kotter in his book Accelerate. The explanation of how that applies to projects in general, and software projects in particular, are my own.)

Leadership

The good leader persuades, shapes, molds the people around them, regardless of the job title or organizational hierarchy of those around them. Leaders are the ones that have a vision for how the project should be completed and what it takes to implement that vision. In other words, leaders are not only the ones to make decisions, but they are also the ones that persuade people that their decisions are the right ones.

Management

To contrast, a good manager’s primary skill is to communicate the goals of the leadership team to their employees or team members and to remove obstacles so those employees have what they need to achieve those goals. Instead of adapting their team to the ideal future, their job is to make the processes of the present more efficient and reliable.

Why it matters

Teams that function well are both led effectively and managed well. Here is a table that depicts what often happens when teams or companies are lacking one or both skills:

	Unskilled Leaders	Skilled Leaders
Skilled Managers	A well-run and consistent company, but lacks the ability to respond to changes when needed	A well-run company that is poised to succeed in the 21st century
Unskilled Managers	Doomed	A company that can adapt to new opportunities and threats, but is chaotic

As you can see, without significant skill in management, it’s tough to ensure that people know what they need to do and difficult to confirm that they are doing it. Without leadership, though, the company will be ill-prepared to take advantage of new opportunities or adapt to new threats.

Why it matters for Software Development

Many of the programmers I've worked with in my career had great leadership skills, but I've not met many (me included) who also had good instincts for management. They are often good at managing the team to ensure high code quality (which doesn’t directly add value to the project as a whole), but they don’t manage the people involved. Their view of management seems to be that since it does not directly provide value to a project, team, or company, they focus on leading the team and letting everyone manage themselves.

Since virtually any non-trivial project usually need some sort of management, companies compensate for this by adding a project manager to most software projects. These project managers don’t often have development experience themselves, but are tasked with communicating goals to the team and removing obstacles like a good manager should.

But since companies confuse leadership and management, project managers are often asked to lead projects as well. Effective leadership usually requires one to have a deep understanding of the subject matter at hand, and it is extremely difficult being an effective leader of software projects without significant hands on experience. As a result, companies can choose either a software developer to lead their projects, which often leads to a project that is adaptive to change but is extremely chaotic, or they can have a project manager lead projects, which leads to a project execution which is ill-suited to adjust to fast-moving changes in today’s business world.

And we wonder why software projects have such an abysmal track record of successful delivery?

To delve further into the problem of leadership within IT, and to discuss solutions, I’m planning the following blog posts:

June: An introduction to Leadership vs. Management

July: Is a project manager the right person to put in charge of your software project?
August: How do you manage an Agile software project?
September: How to leverage a project administratoreffectively
October: Getting programmers to see the value in management
November: The Peter Principle and software architects
December: Creating a leadership team to make your software project a success

Integration vs. Out-of-the-box

If you read my blog on a regular basis, then you know I somewhat recently had a discussion (if you can call it that) on Twitter about whether integrating Yammer in with Office 365 was a good thing or not. While I found approach of “integration is bad” too simplistic a thought process for the real world, it does encourage a larger discussion as to when it is best to integrate software together and when it is best to purchase a tool that meets all of your needs.

When you might choose to integrate multiple software products

There are two primary benefits to integrating software products:

1. You can choose best-in-class solutions for each need you are trying to fill. Let’s face it, you may see a shiny new software package that promises to replace your CRM system, your project management system, and your invoicing system, but you’re probably not going to find a single software product that has a sophisticated lead management system that also has a robust invoicing system. If you want software that does it all, but does it all well, you’re probably going to have to purchase multiple products.
2. If you are looking to integrate multiple disparate business processes into a single software product, implementing multiple systems separately at different times can be less daunting and less risky than implementing a giant system at once. No, you don’t need to use all of the features of a gigantic system at once, but sometimes the pressure is there to get the most from your money and effort. Having multiple systems can relieve that pressure.

In summary, you probably would choose to integrate products when you have business-critical systems involved that have needs above and beyond what a simple do-it-all system can provide, or you want a lower-risk method to upgrade a large portion of outdated infrastructure.

When you might choose to purchase a single product

You may choose to purchase a single software product when you run into one of these two typical integration issues:

1. Multiple products from a single vendor that have integration points that come out-of-the-box. These types of integrations typically don’t work as well as advertised and cause you to alter the way you’d use one or both products to get the integration to work. These problems can cause your team to lose enthusiasm for the project, if not for the project as a whole.
2. Multiple products from separate vendors that don’t talk to each other, so your development team needs to build a bridge between each of the systems. These bridges can be time-consuming and costly to make, in no small part because the documentation for creating these bridges are almost always poor or out-of-date.

To avoid the possibility of missed expectations or costly maintenance, you may choose to purchase a single system to meet all of the needs of your company. You might also choose to utilize a single system for all of your needs if you want to simplify your vendor management.

Other considerations

Most companies I’ve worked for and contracted with have become dependent on strange idiosyncrasies within their business processes that don’t truly provide value to the business as a whole. Purchasing a system that doesn’t support these idiosyncrasies can be an impetus to change. You really should only be deviating from standard industry practices when this deviation gives you a strategic advantage of some sort. Otherwise you’ll probably decrease costs and variability by adopting industry best practices as a standard.

When companies are trying to replace old systems with new ones, their instincts are to choose systems that have all of the functionality the old one provides, except with a few improvements in reliability, cost of maintenance, features, etc. The problem with that thinking is that companies will often think that they are dependent upon existing features that don’t really provide value for the organization. So when you are trying to determine what your new system should do, try to be open-minded about which features you need. This will help prevent you from pursuing best-in-class systems and integrating them when a single system will do.

Beware of systems that claim to be extremely configurable/flexible. The right software for your business will force rogue employees from following their own process and can help guide new employees in forcing them to follow what is “supposed” to happen. The more flexible your software is, the less likely that it will be able to enforce your company-specific business workflows.

What does your existing infrastructure look like? If you are looking at a software system that has a feature you need, but has three you already have elsewhere, will you save money by getting rid of these other systems and moving the three business processes over to the new software product? Is the migration worth it?

Finally, you should ask yourself whether the systems should be integrated at all. I personally hate the idea of having the same information stored in multiple systems (such as customer information duplicated in the CRM system and the invoicing system), but sometimes the costs of integrating these disparate sources of information aren't worth the benefits gained from pulling from a single source.

Why Stages 1 and 3 are not viable long-term stages

Last month, I wrote about how there were different levels of interaction IT could have with the rest of the business. As a summary, here are the stages:

• Stage 1: Internally Neutral (Minimize IT’s Negative Potential)
• Stage 2: Externally Neutral (Parity with Competitors)
• Stage 3: Internally Supportive (Provides Credible Support to the Business Strategy)
• Stage 4: Externally Supportive (Provides an Important Contribution to the Competitive Success of the Organization)

Despite the fact that there are four stages, there are only three stages that are sustainable for the long term, and only two can reasonably be used by companies expected to compete in the 21st century.

To start, Stage 3 is not a stage that is sustainable for the long term. To be expected to consistently provide support for the business strategy without being expected to help define and shape that strategy isn’t reasonable. It’s too ambiguous, and I would see the best IT personnel leaving to be full partners in shaping the strategy. Even Wheelright and Hayes observed that most companies that achieved Stage 3 fell back into their old Stage 2 habits after one or two significant initiatives.

Unlike Stage 3, Stage 1 could conceivably be used by a company for the long term. However, it is not a level that allows an organization to get anything out of IT. Any organization attempting to build software (either for internal or external use) needs to partner with the people who do it for a living. A company that stifles its best IT talent will drive away its best IT employees, making it even more difficult to deliver software, which makes it more difficult for the leaders to give up control, and so on. It’s a nasty cycle that’s hard to stop. In today’s world, every company needs to be competent in technology. They cannot afford to be significantly behind their competitors in the way they create and consume information and technology. Otherwise their competitors will pass them by in virtually every aspect of business: marketing, operations, employee satisfaction....

To give you an idea of the dysfunction that exists within a Stage 1 environment, I’m going to tell you a story about a project I did with what was undeniably a Stage 1 company. On one particular project, we were tasked with creating a website for one of our clients that gathered information about prospective customers. The information we were gathering was relatively straightforward, but we needed to pull the participants in the program from their system and push the results back to them.

Since the leaders of the project had a Stage 1 mindset, the development team did not have any opportunity to ask for the information we needed, and instead we relied upon the sales person to get this. He was not very technical, so he asked for the wrong information more often than not. Because the development team only had a vague idea what the needs of the business were, we couldn’t help him ask for the right information, we could only say that what we had was very clearly wrong.

Once we did get information that was good enough where we could get started, we found out that the sales person made promises that we would struggle to keep. We did the best we could, but ended up making a system that only barely worked and only barely had the functionality that the client requested. The worst part of it was, though, that as I was implementing the system, I discovered that the functionality that the clients requested was not the functionality that the clients really needed. But since we were on such a compressed time schedule, I did what I could and moved on. But if someone had some knowledge of the business pain and knew the data truth and not just the façade, we could have delivered software with much more useful functionality at about the same cost.

In the age of search engines, companies simply cannot afford giving their customers anything less than the best service. Word of mediocre service gets around. What would happen to your business if it consistently treated its customers like this?

One final thought – if you need proof that Stage 1 companies drive away good employees, this is a good example for you. I’d like to think that I was a good employee for this company, but when I discovered that the next phase for this client would be managed about the same way as this one was, I shortly thereafter informed the company that I would no longer be doing work for them. Life is too short.

Other posts involving the four stages:

May: Levels of function (and dysfunction) IT can have with business

June: Why Stages 1 and 3 are not viable long-term stages

July: What a Stage 2 organization might look like

August: What a Stage 4 organization might look like

September: How IT professionals can be their own worst enemy in achieving Stage 4

October: How your Stage would affect your management practices

November: How your Stage would affect your hiring practices

December: How can an organization reach Stage 4?

Blind spot among IT professionals: My approach is the right one!

A couple of weeks ago, in response to the blog post about Yammer's uses I wrote, I got into what started as an interesting conversation with @JoshDeax. Long story short, he suggested that Yammer's functionality was lacking and there were other products that had more features. Apparently it wasn't enough for him that Yammer will easily integrate with other Office 365 products, though, as this was his response:

@scottnorberg It's been two years so far. Plus integration=wrong. Out of the box=right.
— Enterprising Josh (@JoshDeax) May 20, 2014

I suspect I'll be writing about integration vs. out-of-the-box at some point, since that's a fairly interesting discussion in itself. Instead, though, I'm going to write about the culture of IT, specifically as it relates to comparing one solution/approach to another.

(Disclaimer: Since you're only allowed 140 characters in Twitter, it's a hard place to hold a nuanced discussion. Plus, Josh may have been trying to get attention by posting a strongly worded opinion. Therefore, I am not trying to argue that Josh necessarily has this blind spot. This is a problem with IT in general, though, and as a result should be addressed.)

If you search the web, you will find a large number of stories of IT professionals who don't want to sacrifice quality for price or convenience. Like Josh above, too many people in IT don't understand (or don't want to understand) when sacrificing quality to reduce costs may be appropriate in some situations. In this case, Josh didn't ask questions about existing infrastructure or whether the more sophisticated (but integrated) Office 365 was needed to meet specific business needs. No, instead he focused on "integration=wrong, out of the box=right".

Aside from being dumb and annoying, this all-or-nothing approach alienates the very people that we are being paid to help. They just want to get their problem solved - they don't want to hear about the messy details about how we'll get it done. And they certainly don't want to hear people creating unnecessary obstacles to getting the "real work" done. And make no mistake, assuming you're a typical medium-sized business using the Microsoft stack, abandoning several existing Microsoft products, such as Lync, Outlook, and SharePoint, to take full advantage of an enterprise social network is creating artificial obstacles to implementing a solution.

So what should we as technologists do?

1. Focus on solving business problems. Always remember you're being paid to make someone's life easier, not write code. 
2. Keep in mind that guidelines are just guidelines, not hard-and-fast rules. Clean code, elegant solutions, and staying current with product releases are fine things to strive for, but sometimes a small but messy problem needs a small but messy solution.
3. Focus on meeting needs, not fulfilling desires. Most business users aren't sure how their needs can be solved using technology. Therefore they often ask for a lot of things that aren't really needed. If you find out what is actually needed, then you're in a better position to know what corners to (or not to) cut.
4. Communicate! Sometimes implementing a new solution really does involve upgrading a product or replacing a technology stack. But you'll be much better off if you communicate the difficulties early and talk about pros and cons rather than assuming that you already know the correct approach.
5. Admit when you don't know something. This is related to #4 - if you're unfamiliar with a product, then the risk of using something new should be communicated and addressed. Too many IT professionals (and people in general) hide what they don't know for fear of looking ignorant or incompetent. If people are used to you delivering solid products, they'll appreciate your honesty. (If they're used to bad products and missed deadlines, then you have other problems to address first.) As a bonus, your co-worker or client will see you more as a person, not a "magic IT guy".

Can anyone think of anything else? Please comment below!