The development process Part 1

Have you read the previous blog yet?: Creating ethical software, part 2?

The Booking System team is happy that Alex will be spending more time with them and the application. A few weeks ago, a product owner was added to the team. Even though she and the team get along well, her introduction has led to an increased demand from management to deliver new features. 
“Lately, we’ve been missing a lot of deadlines for new features,” she says to Alex. I’ve spent hours upon hours trying to write plans for the application. I’ve discussed with the team each time whether the planned deadlines were realistic, and they assured me each time that they were. I don’t see what we’re doing wrong.” 

Alex nods. “That seems frustrating. Are you following any specific software development methodology?” 

The product owner laughs. “I guess we’re following two different ones at once. The team wants to work with scrum – well, most of it anyway – but my manager expects me to plan out releases with specific features so that we can inform our other development teams about upcoming changes.”  

Alex thinks for a bit. “Let me just get back to work, picking up some user stories and getting back into it. When I notice something about our process that might be relevant, I’ll send you an e-mail about it. Does that seem like a good idea?” 

“Oh, it’d be great if you could start coding right away,” the product owner says, smiling. “You see, there’s an upcoming deadline…”

“We’re going to miss our deadline!” This is a phrase well-deserving to be on any software project bingo card. So why do we do it? Why do we set deadlines? Why do we try to plan the unplannable? 

A software development plan serves several crucial purposes, even though the exact duration of development may be challenging – frankly, impossible - to estimate accurately, especially at the start of a project. But the plan gives us guidance and direction, helps us allocate resources (including how many developers are needed), and manages risk. A plan communicates intent to both the team as well as others, helps manage time, encourages task breakdown, and helps set priorities and goals. Better plans will also set up quality expectations and prevent scope creep. And finally, a plan will allow for continuous improvement of the estimations and even development processes. 

Of course, many people will assume that the main point of a plan is to provide an estimated time frame for a project. When do we begin, and when will we be finished? That is – in my opinion – completely and utterly missing the point. I have written this blog to illustrate the truth of a statement I read long ago on the internet: “Plans are worthless, but planning is everything." 

While software development plans provide structure and direction, it's utterly obvious to anyone who works in software development that flexibility is essential. Agile methodologies, for example, emphasize adapting to change rather than strictly adhering to a rigid plan. Plans should be seen as living documents that evolve as the project progresses and new information becomes available. The goal is not to have a perfect plan but rather to have a strategic guide that helps the team navigate the complexities of software development. Even traditional Waterfall methodologies like Prince2 can help a team, if everyone acknowledges that it is the planning involved that is paramount, not the estimated deadline.

Speaking of Waterfall: in the annals of software development history, there was a time when the traditional Waterfall methodology ruled the roost. Imagine a manufacturing assembly line, meticulously planned from start to finish.  

A great example of a Waterfall methodology is Prince2 (PRojects IN Controlled Environments). It is the equivalent of a scientist's meticulous lab notebook. It's a comprehensive framework that outlines key project management principles and processes, ensuring that no crucial element is overlooked during the project lifecycle. 

Software projects, especially complex ones, can sometimes become labyrinthine. It's easy to miss out on important steps, leading to delays, scope creep, or subpar deliverables. Even in complex environments, Prince2 can serve as a handy checklist to guide your project from inception to completion. It provides a structured approach, including defined roles, stages, and processes, ensuring that nothing falls through the cracks. 

Consider a software development team tasked with building an e-commerce website. They decide to adopt Prince2 to keep their project on track. At the end of each sprint, they review their progress against the Prince2 checklist: 

1. Initiation: They confirm that project goals, scope, stakeholders, and risks are well-defined. 
2. Planning: They assess whether the sprint plan aligns with the overall project plan and whether resources are allocated appropriately. 
3. Execution: They verify that the sprint's tasks have been completed, the code is tested, and features are integrated. 
4. Monitoring and Control: They ensure that progress is tracked, risks are managed, and adjustments are made as needed. 
5. Closing: They review whether sprint deliverables meet the defined criteria and whether documentation is up to date. 

By using Prince2 as a consistent guide, the team ensures that they haven't overlooked any critical aspect of their project. It's like performing a thorough check of your experimental apparatus before each trial, ensuring accurate results and reliable outcomes. 

However, using Prince2 as the sole methodology is bound to lead to frustration. As software systems have become more complex and the pace of technological change has quickened, we have learned that Waterfall is not suited for software development. Agile methodology has become the norm, with various related methodologies helping teams to be transparent and efficient. 

Agile was born out of necessity, a response to the perceived heaviness of Waterfall. Just as a scientist adapts experiments to changing conditions, Agile champions flexibility, iteration, and collaboration. Instead of meticulously mapping out the entire project upfront, Agile encourages breaking it into smaller, manageable chunks - think of them as mini-experiments. Each iteration results in a tangible output, fostering a culture of reflection, adaptability, and continuous improvement. In this way, Agile also brings many of the benefits of a plan. But in the same way, Agile is just replacing the one, singular plan with many, many smaller plans. Each with its problems, depending on the specific flavor of agile. 

While Agile brought about a much-needed shift, it's not immune to becoming a tad weighty itself. In a way, it's like an experiment that becomes too complex, with variables competing for attention. The key lies in finding the right balance - just as a scientist fine-tunes their apparatus. 

Scrum ceremonies, like Daily Standups, Sprint Planning, and Retrospectives, offer invaluable structure. However, if these meetings are done even slightly “wrong” (or at least in a way that does not suit the team and context), these meetings will inadvertently lead to time-wasting and information overload, causing your team's output to stagnate. The Scrum Guide will stress that it's not about following rituals blindly but about nurturing a dynamic environment where processes enhance, not hinder creativity. But it’s still hard to find that balance.

Here’s where I think all methodologies tend to work against the team: meetings. Overlong, unnecessary meetings or meetings with silent attendees are a common plague in software development. These meetings eat away precious time that could be better spent coding, debugging, or collaborating.  

Now, as you can tell by Alex’s story, I do think communication and discussion are important. But it should be as short as possible with as few people as possible.  

Here’s what you should do. Embrace the principle of quality over quantity when it comes to meetings. Ensure that each meeting has a clear purpose, relevant participants, and a focused agenda.  

Most importantly, all people present should be treated as equals. People “higher in hierarchy” should host the meeting but not join the discussion, or not join the meeting at all. Otherwise, meetings tend to be in person e-mails for many of the participants. 

For instance, in Scrum, the Daily Standup is a prime example of an efficient meeting. If done well, it is a meeting of equals taking up as little time as possible to bring structure and transparency. Each team member provides a brief update on their progress, roadblocks, and plans for the day. This meeting is short, to the point, and ensures that everyone is aligned without wasting time. It is the reason, I think, that many companies that do not work Scrum (or do Scrum In Name Only, SINO) adopt the Daily Standup anyway, as it is a prime example of a meeting done well.

Okay, so we can use planning over plans and efficient meetings to create more efficient and transparent development. But the scale of that transparency is small. It won’t help us keep to deadlines months or even years in advance. So, are deadlines doomed to be missed? The very essence of software engineering defies a rigid, one-size-fits-all plan. Instead, it demands the developers (and their managers) to fight with uncertainty. An ever-lasting dance. Or not? 

Before we continue: what do I mean by a deadline? Take a look at the image below. You may already be familiar with it, but if not, let me explain it. 

The project management triangle, also known as the "iron triangle" or the "triple constraint," is a fundamental concept in project management that illustrates the interrelationship between three key factors in any project: time (schedule), cost (resources), and scope.  

Time refers to the project's timeline or schedule—the duration required to complete the project. In software development, time considerations include milestones, development phases, testing, and deployment. Project schedules are often established based on estimates of the time required to complete various tasks and activities. When a team misses a deadline, they often think they had too little time. 

Cost refers to the financial resources allocated to the project. This includes costs associated with human resources, equipment, software tools, infrastructure, and other expenses. Proper budgeting is crucial to ensure that the project remains financially viable and does not exceed its allocated resources. If a company wants to make a deadline, they can often throw more money at the problem if they think it’s worth it. 

Scope refers to the work that needs to be done to deliver the final product, including features, functionalities, and requirements. The scope encompasses the specific features, modules, and user interactions that the software will have. It's important to clearly define and document the scope at the beginning of a project to ensure a shared understanding among stakeholders. A team can choose to increase or decrease a project’s scope, which will change pressure on the cost and/or on the time. 

Finally, in the middle, is quality. We’ve discussed it before. Quality refers to the level of excellence and fitness for the purpose of the final product. A team agrees on a level of quality that is the bare minimum and aims to achieve that. If the level of quality needs to increase, the time, cost and/or scope will need to be adjusted. A team should never sacrifice quality below that bare minimum to “make a deadline”. Again, we’ve discussed why. 

The interactions among these factors form the project management triangle. The triangle illustrates that changing one factor (e.g., scope) will inherently affect the other two factors (time and budget). For example, if you increase the scope by adding more features, you may need more time and resources, which could impact the project schedule and budget. 

The team should strive to find a balance among these factors. The challenge is to optimize scope, time, and budget while delivering a product of acceptable quality. For instance, if a project has a tight deadline, the scope might need to be adjusted, or additional resources may need to be allocated to maintain quality. This balance, the final equation, is what I mean by a deadline. A deadline is not just a date; it is a combination of time, cost, and scope. 

So, given the definition above, are deadlines meaningless? At the start of a project: yes. It can never be much more than random guesswork if even that. But watch what happens when a project continues, especially a project where the team and the requirements stabilize a bit. 

In the early days of a software project, the team embarks on a voyage of discovery. The first phase of a software project is tumultuous. New members come together, personalities clash, features are still unknown or unclear, and ideas collide in a tempest of creativity and conflict. When a team can and is allowed to confront these challenges head-on, bonds are forged, the goal of the project becomes clearer, and a sense of shared purpose emerges. 

As the dust settles, a remarkable transformation takes place. The team transitions into the norming and performing stages. Communication becomes more fluid, conflicts are resolved constructively, and roles are defined with clarity. In this harmonious phase, the team becomes a well-oiled machine, working cohesively towards shared goals. 

Planning slowly becomes a plan as the project progresses. On the one hand, the amount of work that needs to be done will decrease over time, and with it, complexity can decrease as well. At least, if you’re building quality software, it will. On the other hand, the project and team will have created an amount of transparency that makes determining a deadline – well, not easy, but possible. However, for this magic to take place, the project mustn’t reenter that first tumultuous phase. Such a fallback can have many reasons, some of which we’ll discuss in part 9 of this series. If a fallback happens, the planned deadline will immediately lose value, and the team will have to resort to planning over a plan again, to attempt to set a new deadline in the future.

A few weeks later, Alex had written many e-mails to the Product Owner. In them, Alex recommended looking at possible changes in budget and scope, instead of only focusing on time and looking at why the team could not develop as fast as they expected. Alex quickly learned that the e-mails had been shared with other product owners and project managers. Alex's thoughtful e-mails sparked a series of discussions across E-train. The various product owners and project managers began to realize that they were all facing similar challenges and were intrigued by Alex's suggestion to reevaluate their approach. 

Alex's idea of addressing the root causes behind the missed deadlines resonated with the Product Owners. They recognized that constant changes in team members, project focus, priorities, and other sources of chaos were preventing their teams from solidifying into well-structured and transparent units. This lack of stability was leading to confusion, inefficiencies, and, ultimately, missed targets. 

Today, the Product Owners have decided to come together for a collaborative meeting. “Not too long, I hope,” Alex jokes when the invite comes in. The product owners and project managers gather in a room, ready to discuss the issues they’re facing and to collectively devise a strategy to address them. 

"I want to thank Alex for bringing up this crucial point," one of the product owners begins. "It's clear that our projects and teams are struggling with a lack of stability, and this is impacting our ability to deliver on time."  

A project manager chimes in, "I've noticed that every time we have changes in team composition or shifting priorities, it takes a toll on our productivity and overall performance." 

Alex speaks up to set a clear goal for the meeting. "I believe that if we can prioritize creating stable and transparent projects and teams, we can greatly improve our outcomes.” 

"Well, it’s a great analysis,” another project manager says, before adding, somewhat cynically: “But what can we do differently? It’s not like I like how management keeps changing up my priorities and my team members.” 

The room falls silent for a bit. Then, people start discussing their options.

An overview of what has already been posted and what is still to come, here is a full overview:

• Don't Skip, Hot Take
• Writing Quality Code (part 1)
• Writing Quality Code (part 2)
• Building Quality Software (part 1)
• Building Quality Software (part 2)
• Creating Ethical Software (part 1)
• Creating Ethical Software (part 2)
• The Development Process (part 2)
• The Software Engineer Oath

As always, I ask you to contact me. Send me your corrections, provide suggestions, ask your questions, deliver some hot takes of your own, or share a personal story. I promise to read it, and if possible, will include it in the series.