We have helped small startups and enterprises such as AWS, Magnacare, New York Ophthalmology, Mt Sinai Hospitals etc. In my experience, not many understand the agile mindset inherently, nor are ready organizationally for agile development.
Here are our thoughts on how you could look at agile development, whether your organization is ready for it, what you need to do to be prepared etc.
Who should read this
Enterprises still consult with us on what exactly agile development is, how exactly they could implement agile methodologies within their organizations, how scaled agile can solve all their problems, how they would deliver software reliably on time and on budget using agile etc..
This is mostly a post for managers who are trying to get a better handle on agile development technologies and how their team could get better at development and delivery of software using agile methodologies.
From what I have noticed most of the managers are, as expected, a little out of touch from daily development. They have been there, done that, have delivered software reliably in their past without even using agile methodologies; so what exactly does this newfangled way help them and their team? More importantly how would it help them deliver on time and on budget?
I am not going to attempt to write a book on Agile development as there are WAY too many already.
Rather, this is just a quick read for managers, product owners, non technical entrepreneurs etc to understand what it is, how to use it, whether agile would benefit them, how it will benefit them (if at all) and most importantly what the risk profile of using agile development is.
Basic stuff first – What Is Agile Software Development?
It’s not a silver bullet, a specific formula, some specific technique or anything of that sort. It is simply an umbrella term that is used by various software development methodologies.. (various methodologies to develop software in an iterative and incremental manner).
You’ve probably already heard some of these – e.g Extreme Programming (XP), Scrum, Crystal, Lean Development, DSDM, and Feature-Driven Development etc.
These are all talking about the same thing – agile software development.
At some point in time, google “Agile manifesto” and read up on it.. It’s a quick read and here’s the basic stuff you need to understand about that manifesto
“Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan”
You can choose any of the agile methodologies that you want – the underlying concept and principles remain the same… developing software collaboratively, responding to and embracing change + frequent feedback, preferring working software over boat loads of documentation.
Note that agile development is an approach that’s nowadays used even beyond software development.
How’s that possible?
Because agile development is more about a mindset and approach to developing “stuff” – whether it is software, a market, a product etc..
Just let that sink in first.. Then proceed.
Remembering waterfall development methodology
So, as an old timer, you are surely used to waterfall methodologies. That’s what we used to do before…
We’d get marching orders to develop *something*.. We would get together and start the process to uncover and gather all the possible requirements.
This would give us reams of project requirements documentation (about what needs to be done).
Then we would go ahead with the analysis – therefore producing our models, actors, schemas, business rules etc.
Then we would also start putting the design together. We would break this into high level and low level designs and would also get the system architecture.
Finally we would start development of the product – i.e the actual coding.
Then we would go through a lot of testing and validation..
We would produce reams of product documentation and support the system etc..
That worked fine for us.. That worked well for the market back then. Even with inherent issues and the massive project failure rates, we didn’t know any better.
Then the business would see our work, validate that the product meets all the requirements we had been given, then go to market with what they intended to… 6-12 months ago (sometimes years ago).
More often than not, the market didn’t stay where it was when the requirements were written.. The market had already moved on.. Potential customers of the product now wanted something more or something different..
And the “success” of the project was.. Well.. dubious at best.
That worked fine.. So why change now?
What are the benefits of agile software development
Markets and opportunities are coming up fast, and competitors are coming up faster..
These days it has become a “you snooze, you lose” game. Customers have too many options available to them.. There are too many competitors and even small, agile product companies are giving the big guys a run for their money.
The challenge back in the day was that the software development team was almost never in direct touch with the business team. The business team was always in touch with the end customer, but the dev team not being in touch with the business, was, in effect, never really building a system / product directly for the customer.
So, let’s think about this.. The product management team might have been talking to customers that would “some day, potentially” buy a product that they envisioned. They spoke to these potential customers, pitched them their product idea, got an agreement and then the software development process took over.. Using waterfall…
In other words.. Now the product management folks didn’t have any real good reason to get back in touch with or continue to stay in daily/weekly touch with the end customers.. Why? Because they have nothing to show..
Nothing to collaborate on.. Nothing to get feedback on.. The only feedback they could get from the customer would be .. well, 6-12 months away.
Even if the business was in regular touch with the customers, and found that the requirements changed due to some reason (whatever be that reason).. Getting those changes done in the middle of a large project was.. Well.. arduous.
That’s the problem – customer development was hard and only kept getting harder. You couldn’t go to a set of ideal customers, show and tell what your idea was, get their nods and get back to them quickly with some progress.
Meanwhile, the customer either moved on or put that project in the back of their minds.
That had to change.. And so it did.
Why use agile software development methodology?
Experienced software development professionals saw the issues in this kind of development approach. So did the business.
- What if there was a way to accelerate the delivery of initial business value?
- What if there was a way to show constant progress & get feedback on that progress? We all know that as soon as a customer sees their vision come to life in any shape or form (i.e. tangible), their minds start working faster.. They start thinking about how else the system could solve their problems.. What would be a better way to solve their problem etc.. So, what if there was a way to get feedback and be able to incorporate those changes immediately instead of having to put a huge risk burden?
- What if we didn’t have to collect reams of requirements, analysis and design documentation for months before starting the project? What if we could talk about a product at a high level, figure out the bare minimum to get started and show value in that week itself? Of course, we don’t want to throw planning out the window, but what if planning was done in an iterative way because we got feedback on business value delivered iteratively and continuously?
- Would that result in a software system that much better addresses the business and customer needs?
This is not to say that it was a brand new concept.. Even back then, extreme programming was already into the picture.
That’s how agile was dreamt of and started being adopted.
Differences between agile and waterfall software development methodologies
If nothing, keep at least these few things in mind
Agile software development has releases and iterations.
Each release usually has many iterations.. And each iteration itself is like a micro project.
So, what you did was to take a giant waterfall project, break it down into many, many smaller chunks of features and releases.. Then you took each release and broke it down into many iterations..
You put work items, enhancement requests, bugs, features etc into an iteration. After an iteration, you get feedback, collect those and prioritize for the next iteration(s).
And most importantly – each such iteration produced tested, working software.
Is this cruel to the development team? A little 🙂 yes.. But it’s in the best interest of the overall team (ah.. That includes business as well, in case you are wondering)
Advantages of agile methodology over waterfall model
Quite a few actually ! You can google for this and get a ton of information.. But keep these few things in mind.. (and don’t get carried away with the purists’ propaganda)
Working, tested software each time
That’s pretty much a hallmark of agile methodology. A giant project broken down into baby steps. Success shown in baby steps.
Working, tested software in each such baby step.
Think about your risks – you just brought it down to a single iteration.
Does this mean that there are no bugs or errors in each iteration/release? Sure there is.. But instead of discovering a giant bug at the end of 12 months that makes the development team do heart surgery to fix it..
You find those bugs in smaller chunks, address it immediately, in smaller chunks.
It delivers value, constantly
Why? Because your development team is focused on delivering a feature (or features) to you in every iteration and release.
You see tangible business value in every release.
You measure progress and assess its value in every iteration or release.
That’s the point of it.. Break your risks down to smallest chunks possible..
It helps you plan, constantly
As you are seeing incremental progress and are validating your idea(s), this allows you to plan iteratively.
It allows your development team to plan iteratively instead of having to sit through all the planning way up front in the project.
It allows the business to plan iteratively about their activities and customer development. Is the customer asking for something else or something different now? Well, no problem. In waterfall, you’d find out the changed customer needs at the end of 6-12 months.. In agile, you find that out in 2-3 weeks.
And you incorporate those changes in the next releases.
See the difference?
Discovering, learning, collaborating
Developers develop for end users, customers. In waterfall, we had customers in one building, business (product management, marketing etc) in another and developers in yet another building.
If you really think about it – solving that business problem with the help of software was the goal for the *entire* team. So why were each sitting in their own silos?
That’s what agile software development allows you to break down. Breaking down those walls and really allows you to collaborate across teams.
To achieve the same goal.. While collaborating.. As a team.
Instead of the customer having the entire onus of figuring out everything up front, this allows the broader team to discover features and functionality together… as a team.
The development team can come up with various ideas of solving a problem because they have worked across verticals and business problems.
The business can advise the development team that the feature that they are saying will cost $5,000 to build might actually not be such an important one.
The development team can constantly provide feedback to the business with the costs associated with their choices.
And so on..
Real collaboration to produce business value within a reasonable budget.
Disadvantages of agile development methodology? Risks?
Everything has risks. Agile development does as well.
Experts say the core elements of agile development such as frequent customer interaction to refine scope, frequent delivery incremental software capabilities, product backlog can be effective in some software projects but can be ineffective or suboptimal in others.
If your team or your organization is not ready for agile development, then you are taking on a risk.
If you are going the route of “agile development” cos that’s the latest fad.. You are at risk
If you have been tasked by your boss to become an “agile development” enterprise.. Well.. depending on how you responded to your boss.. You *might* be at risk.
In other words, your mileage may vary (YMMV).
The good thing is … you can actually evaluate whether you are actually ready for agile or not.
And hint.. You are not ready if you just had your entire team sit through some darn scaled agile course.. No, sorry to say that you wasted your money.
Agile needs commitment and it needs a cultural change..
And that change needs a strong sponsor – usually at the top of an organization. If you don’t have sponsorship, don’t try to boil the ocean. Start with a small committed team on a small, low risk project… show progress, get buy in for the next bigger project.. And that’s how you bring about that change in your organization.
How to decide if your team is ready for agile software development methodology
There are several considerations when deciding which methodology is best for a specific project and a specific organization.
First thing to ask yourself would be, how does my team communicate?
Does your team really have open and face-to-face communications?
Even if it is not face-to-face communications (i.e. remote, VOIP based communications), does your team discuss pros, cons and everything else in between, openly?
You might be tempted to answer this question flippantly by saying “yes of course!”.
Stop right there and ask yourself again – does your team really do this?
If your team is not the best at communication then agile is not for you.
Keep in mind that agile (e.g. Scrum) has only 3 roles – product owner, scrum master, development team member.
This means that the overall team is going to communicate everyday, multiple times a day. They are going to discuss issues and development approaches, pros and cons etc.. each day. They are going to white board whenever necessary.
Be prepared to break down the barriers of formal communication structures. Be prepared to forgo giant documents to communicate between team members. That’s an anti-pattern and is simply not going to work in agile.
You can be a distributed agile development team (if that’s your only option) but that doesn’t negate the need for openly flowing communications, the need for stand up meetings (short), the need for discussions on a daily basis, the need to collaborate between development, support, marketing, product management etc.
Next, how is your team organized and structured?
Most enterprises I have worked with, are organized in a very top-down way. It’s not a bad thing – a top-down approach has its own benefits as well.
But don’t try to fit agile within a “top-down heavy” kind of organization.
Agile development needs cross-functional teams.
Understand that.
Being agile means that you are going to need a team of developers, testers, scrum managers, business folks, user-experience developers, user interface developers & hopefully even marketing people all together in one room.
Can you do that? Can you truly get all these folks in one room to discuss and develop a product iteratively? If you cannot, agile development is going to be really painful for you.
Look at your team’s composition. Most organizations have a pyramid structure with senior and junior team members. Keep that in mind and understand that very well.
As I mentioned, agile teams will have just *developers*. So, if you have senior people mixed with very junior people, you are going to face problems. The team will only be as fast as the slowest developer. I believe in the book “Mythical man month” it is reported that the productivity difference between best and worst programmers is to the tune of 10:1.
This doesn’t mean that junior developers are a no-no in an agile development team. It just means that you need to be cognizant of your team mix.
Again, go back to the 3 roles in scrum/agile. If your organization cannot thrive in such a flat structure, you are going to fail. The managers are no longer managers, the lead developers are no longer lead developers, the rockstars are no longer rockstars.
One of the benefits of agile development teams is that silos of information and silos of skill sets are reduced and bus factors are increased. Your frontend dev will learn from your backend dev and vice versa. You need to ensure that your organization can thrive in those situations.
Can you really handle frequent delivery of working software?
No, it’s not a trick question.
In my experience with most organizations, especially larger ones, I have found that many (if not all) are not ready nor have the bandwidth to accept frequent delivery of working, tested software.
Why? Simply because they don’t have the bandwidth nor the team needed on their end to validate what we have delivered.
This is demotivating for the agile team. If you are not truly ready to work hand in hand with your agile development team – don’t take the agile route.
Agile works best when feedback is frequent. If business and possibly customers cannot provide feedback on a frequent basis, there’s no point in doing agile development.
Think about it. Basically when you are developing a product, there are 3 main functions – developing it, operating it, selling it.
What good is it if these three sets of people are not on the same page?
What good would it do anyone if development produces a release and marketing cannot give feedback – “hey, this is going to help us sell” or “hey, our customers won’t understand this/that”?
What good would it do anyone if operations folks cannot give feedback like “wait a minute, if I get a call from a customer that X or Y is not working, where do I go look for it?” or “How do I answer question A from a customer when they call me?”
You get the picture?
Are you ready to measure progress by features and working software rather than phases?
In my experience, even if an organization starts off by saying that they are ready to work in an agile manner, they truly are not.
Most organizations (larger ones) need reams of documents, progress reports, requirements traceability matrices, etc to see and report progress.
Agile development works around features and delivery of working features.
Can you live with that? If you cannot, don’t go into agile development.
Many developers initially look at agile development as a way for managers to “micromanage”. They see it as a way for managers to say “what are you doing on a daily basis?”
Why? Because they are used to meeting with managers, probably, once a week to discuss progress. In agile, they will be meeting with them on a daily basis.
This is also where managers need to have a fundamental change in their behavior. Instead of barraging the team about why a feature took longer to implement, a manager needs to work towards removing obstacles from the team’s path. The manager would need to spend this time with the developers understanding why the delay occurred.. So that they can address removing those obstacles for the next iteration.
Your organization needs to be able to measure progress in terms of feature development, not in terms of gantt charts.
Do you need to the ability to change directions during the project?
If you really don’t need the flexibility of changing directions of a project while the development is underway, don’t necessarily jump into the agile bandwagon.
If the product that you are developing has set features, set functionality, will not change (yes, there are MANY such projects) then don’t bother with agile. Waterfall will work just fine and will will actually be easier.
Are you comfortable with “just enough”, iterative and continuous planning?
As we mentioned before – that’s a hallmark of agile development. If you cannot handle the fact that the entire product and project will not be planned for up front, then don’t do agile development.
If you are not comfortable proceeding with short, iterative planning, intermediate steps of code hardening, adjusting plans based on prior learnings.. Then don’t go into agile development.
This is where upper management has most of their issues.
When can we say that the project is “done”? When can we promise what to customers? I need a better handle on the budget.. How do I do that?
Yes, that’s understandable as well. It’s hard to get upper management comfortable when you tell them that you are going to keep iterating on the project as long as the customer continues to identify high priority/high value work.
You are going to have to come up with some ways to allow project tracking and project reporting. Your development team is going to have to come up with a general idea of a timeframe that they would need to implement the product. That could very well be a range between 6-9 months.. (or whatever they feel comfortable with).
At this point, you can ask management to budget for at least 6 months and get an agreement that at the end of these 6 months, you could be asking for more budget if you have been demonstrating quantifiable value.
Allow upper management the satisfaction that in agile development, they can also yank the budget much easier if customer stops seeing the initial perceived value in the project or product. Tell them that instead of canning a project and its budget AFTER completing it in 6-12 months, they have the ability to plan iteratively and based on customer feedback, have the ability to can it in the 2nd month itself.
Give management that – they deserve to define their own boundaries of risk and be comfortable with it.
Do keep in mind that unless your management believes that your organization’s main goal is to delight customers (rather than increase shareholder value), it is, ultimately, a difficult sale.
Of course, your mileage may vary 🙂
How to adopt an agile approach in enterprise change management
Change management is a process, techniques and set of tools to manage the people side of change to accomplish required business results. It comprises use of organizational tools to help individuals make successful personal transition that results in realization and adoption of change.
Challenges in agile software development
Agile software development is designed to flourish even in the most dynamic business and technical environments. Since the concept of this methodology is based on “adaptiveness and response to change”- the most essential concept in change management.
However, all agile methodologies are only limited to the integrated practices and process required in developing a continuous stream of new software. Agile methodologies do not address the changes related to enterprise support for the agile task and processes that fall outside the scope of the project work. Some of the challenges that agile methodologies cannot address are
- How to ensure appropriate stakeholders are available throughout the development cycle?
- How to manage organization’s internal personnel to get things done?
- How to gather and prioritize important features wanted by the organization throughout the ongoing development cycle?
- How to get approval of new technologies that a development team likes to gain?
Enterprise change management is the answer
Enterprise Change Management (ECM) offers a framework that addresses many of these challenges. We will discuss how organizations can use ECM practices in conjunction with their agile development teams to promote and strengthen IT delivery adoption.
Convey the vision of change
In any organization the pattern of success in change follows the core pattern – see-> feel -> change. To ensure the stakeholder do not carry any inhibition, feelings or negative thoughts, the enterprise change management should communicate the vision of change that will help stakeholders to get over negative preconceptions and also motivate them for change.
Enterprise change management demands of agile software development
The change management challenges are greater in magnitude when a successful agile project develops new capabilities. Instead of stakeholders adapting to a single release that brings number of changes within the multiyear software release cycle, the agile methods require stakeholders to accustom themselves to incremental, small releases every month.
An ECM program is necessary for every organization which is shifting from traditional software development methods to agile. Corporate cultures which are habitual to traditional development release cycles can be stressed by shift to frequent releases and ongoing iteration and participation required throughout the development cycle.
Agile development process requires higher level of participation from the stake holders and the impact cuts across technology and functional groups. An ECM can breakdown the feeling of burden that is brought by persistent demands of Agile teams for day-to-day customer involvement.
Introducing Enterprise Change Management to Agile teams
Applying basic concepts of ECM can foster positive change in existing agile practices. For example customer focused stories and acceptances can be communicated by ECM considerations. This allows the side of IT and business to work in a coherent manner.
This approach also helps in building high level of trust and provides measure to track quality of software. Bringing ECM personnel in agile team is a good idea. This will allow the agile team to anticipate potential change management issues and help teams to synchronize their efforts.
The basic approach to integrate ECM into agile development process is by including ECM progress through the same process as technical requirements such as user stories, acceptance tests and iterative development of deliverables.
Product engineering vs IT projects – how do you bridge that gap?
Most companies these days are “software companies” in the sense that their leverage over the competition is via automation and software tools.
Doing more with less overheads.
This translates to IT having a much bigger impact on the success of any organization (especially so with healthcare organizations).
Not many IT departments we work with, seem to understand the differences in product engineering vs IT projects mindsets.
Arguably one side of the fence was, and still is, struggling with the mandate of “innovating for business advantage”. The other side of the fence was, and still is, struggling to bring “unprecedented value” as their buyer is asking them to – so they can gain a “trusted advisor“ status.
Based on our observations + experiences, my question to the question above, has been the same.
“Why are you trying to bridge the gap”?
In more cases than not, we haven’t received a convincing answer.
Our expectation in each case was that senior IT leaders would understand that their need to bridge the gap is really stemming from the fact that they need to innovate and they need to innovate faster.
Our expectation was that, more fundamentally, they would know why they need to innovate better, cheaper, faster.
The need to survive.
Over the last few years, the fundamentals of doing business have started shifting. The consumerization of almost everything has been happening for the past few years and the demographics of a large chunk of their addressable market has changed.
Leaders in the traditional businesses in various industries are certainly feeling the pressure of consumerization. Their customer base is changing, along with their customers’ expectations of how they want to experience each brand.
Most organizations are simply not even prepared to make this fundamental shift.
Today’s customer is always ON, always connected and leaves digital footprints everywhere. They want to access your services anytime, anywhere and in their own terms. They want to feel that they can tweet/email/chat/call customer support anytime, anywhere and they’ll get a response. They don’t want to wait for 3 seconds for your web page or app screen to load. They don’t want to place and order and not be getting push notifications on where exactly it is.
Customer loyalty is declining and a vast majority would switch brands at the drop of a hat.
How each organization is choosing to respond to this (either as an opportunity or a threat), however, is up for grabs (our findings).
So how does product engineering fundamentally differ from IT projects engineering?
One quick way to understand this is to look at the differences between IT “in-house” products vs the consumer products.
“In-house” software operates under a very, very controlled environment (even if you have 100K employees). You can provide user manuals, guides, provide training, force your users to use the software etc.. and can control the environment (and the times) that it is being used.
Consumer software / customer facing software doesn’t have that luxury.
Even with consumer facing software that these organizations produce – it still is an interface to the fundamental service that the organization is providing.. So, even if the software is not the best, consumers still do use it (yes, attrition happens because of bad experience with that as well).
As an example – you don’t buy software from a bank. You use their banking interface to access my money. Sure, if it is crappy, you will switch banks but you are not fundamentally buying the banking software that your bank created.
Never would this banking software be in a situation where millions of users are using it everyday.
The scale is different, the resiliency requirements are different, the user experience expectations are different (you must have noticed vast changes in your own bank’s user interface lately as well).
Traditionally, an in-house IT mindset worked just fine, until consumerization threatened everyone’s existence.
But with majority of such talent being used to creating software for pre-consumerization, captive customer base.. Will they be able to make this fundamental shift to creating consumer software?
The question is – Do they actually need to?
These companies are not in the business of software. They are not in the business of selling software to consumers. They solve a non-software related consumer need and the software they create, augments that need.
Think about it – you probably use banking software. But what’s it that you went to the bank for?
To handle your money.
Traditionally, you didn’t go to the bank to get a mobile app that keeps and moves your money around. You went to the bank to keep your money.. Safe..
However, if banking is banking.. And there’s no real, tangible way to differentiate yourself from another bank from a business function point of view.. How do you compete as a bank?
You can’t run your rates higher than the next guy.
So, what do you compete with?
Sure, you can choose to answer with “customer service”, “customer relationship management”, “ease of access”.. etc
But, what’s underlying all of these?
Software?
Of course, every competitor of yours can also provide the same “customer service”, “customer relationship management”, “ease of access” via software.
So, what’s next?
Innovation. … business model innovation, customer service innovation, customer access innovation.. Whatever your choice be, for that competitive edge and to stay relevant.
Without innovation, no matter the value you create for your customers, new customer acquisition and current customer retention is going to be a challenge.
But, how do you innovate in a traditionally, IT minded organization? Is that even possible? More importantly, is that even feasible when business continuity still overrules innovation?
Can a traditional, IT project mindset really produce business model innovation, customer service innovation, customer access innovation?