Not a lot of people know this about me, but I've almost exited programming at three different times in my career...because I wanted to get into project management at a high level. Ever since my first project management class in grad school, it just made sense to me...but after about 15 years in software it doesn't anymore. Let me explain how to fix it.
// Share.on([hacker news,
linkedin,
twitter,
facebook,
reddit])
Full discussion on Hacker News.
When I was in undergrad at Clemson, there were certain classes that just clicked with me immediately and other classes that I forced my way through because they were required. Computer Science clicked with me.
Only two other subjects in all of undergrad did that: Psychology and Economics.
When I went to grad school I enrolled in an experimental Masters program at Clemson called eCommerce. It was run by the College of Business and only a few courses different from the MBA...except you could take 800 level Comp Sci classes. At the time, that was appealing to me.
In hindsight, the name of the degree mattered a whole lot more because nobody has any idea what a Masters in eCommerce actually means. Eventually, the department also realized that problem and changed the program to an MBA "with emphasis in eCommerce." Unfortunately that didn't apply retroactively. I always chuckle when I see professional signatures with "My Name, MBA" wondering what people would think if I started writing "Barry Jones, MeC" on mine.
Despite the name, I gained a lot from the curriculum, but one class stood out above all others: Project Management. Project Management is essentially Psychology + Economics when you break it all down, so in hindsight the appeal is more understandable. I'm also a big fan of poker, as it happens.
Before I was hired for my first major programming job at a big business telecom company I was attending Project Management Institute (PMI) meetings, signing up for membership, learning more about the subject and trying to figure out how to count my hours so that I could take the test to get my Project Management Profession Certification (PMP). For my entire career, I’ve paid careful attention to how projects were managed through the lens of eventually wanting to shift my career that direction.
I came very close to finally making the time to get my PMP about 2 years into that first job. An interesting opportunity came up at a smaller contract web development company, so instead I focused on the work for multiple clients...at the same time...with competing deadlines.
When I left that job to start Brightball, a huge part of it was wanting to structure a company to only handle large projects. The prior company was doing both small, design focused CMS projects and big web projects but the resources weren't managed any differently...and it was wearing the developers out.
About a year into Brightball, the recession hit and things got harder. I've written the story of the company elsewhere already if you're curious. After we shut down, I went back to the big telecom again, in a production support team role this time. It was a term limited contract, so I was constantly looking at other opportunities.
I was about to re-enroll in PMI to pursue my PMP when I was presented with another big opportunity that resulted in the most chaotic learning experience of my career until it was clear I had to move on...largely because of issues created by project management.
Finally, I re-enrolled in PMI and started actively pursuing my PMP. It was time for a career change. The only way to fix project management in software was going to be for me to become a project manager. From the programmer side...it just wasn't happening. No matter your background, you get into an organization with a label of your job title and you become only that job title
But then I ended up in another great opportunity and ended up dropping out of the PMP course again.
That was five years ago and I have no plans to pursue my PMP at any point in the future. Because what you learn to get your PMP and what you need to manage software projects are not the same set of skills...and nobody seems to realize it.
That core problem creates so much friction between software teams and project managers...and the business people who those project managers report to...
It's not the project manager's fault. They know exactly what they were taught in regard to how project management is supposed to work. Those principles actually do work extremely well in many, many industries. PMP focuses almost entirely on Waterfall methodology and gaining the certification is largely driven by terminology and math.
But software is a special snowflake in the project management world and this is why countless books around managing software continue to be written. It is an unsolved problem that nobody has managed to get correct because understanding the psychology of how software developers work plays a much more significant role. It’s not as simple as organizational business rules, estimates and dependencies when how people work can sway timelines dramatically.
I'm going to do my best for the rest of this post to translate 15 years of software and project management experience into a simple, cohesive formula that can make your company better. There's no need for a book. No need for consultants. If you'd like to pay me to email this link to your boss, I have a contact us form.
Read from the next paragraph down a couple of times until it makes sense.
The primary forms of project management that we see used in software are Waterfall and Agile. Under the Agile umbrella, we have Scrum and Kanban.
Waterfall thrives in environments where everything can be planned up front. Construction is a good example, where you can have architects draw up plans for a building complete with resources needed. The cost of resources, time to acquire them, order of assembly, time to construct each phase, licensed professionals and necessary equipment can be predicted with a reasonable degree of accuracy.
Once you have all of that, it becomes a problem of optimizing concurrency against the budget to minimize the timeline. When is Bob going to finish with X so that he can complete Y before Tim needs it to start Z? That shortest point on the timeline is dictated by the critical path, on which any time increases will affect the time for the entire project. There are software tools that excel in managing this model.
From a business finance perspective, this is the dream. This thing will cost $X and be ready in a specific amount of time. It's accurate enough that entire contracts are entered into around these types of numbers with penalties for not hitting the deadlines. You can plan multiple projects and accurately decide to execute based on the expected Return on Investment.
Unfortunately, in software...everything is not known up front, in fact…most things are not known up front. Acknowledgment of that is usually where the transition to Scrum begins.
The goal of Scrum is to isolate chunks of work into sprints. Usually a sprint is a 2 or 3 week time period. The development team will meet, look over the backlog of work to be done, estimate how long those tasks will take to complete and then "commit" to what will be done. In some cases, dependencies will be known ahead of time and project managers can still nurture the critical path through this process.
At least that is the ideal. The reality of Scrum in many organizations is that it's a concession just trying to hide an attempt at Waterfall behind the scenes. If you are developing a system that has not yet been publicly launched, this approach can sometimes do well for a while even despite the problems that begin to surface quickly.
We will talk about those in more detail soon.
Next in the methodology timeline is Kanban, which in the world of software is a very simple approach. Your developers are generally working on one thing at a time, potentially a second item that is in progress but is on hold for some reason. The backlog is prioritized by management and developers will pull things off of the top of the prioritized queue to work on once they finish the prior task.
The prior task will move to a review phase until it's approved/deployed/done. One of the hard rules of Kanban is a limitation on how may stories can be In Progress or Reviewed at any given time. If things are waiting to be reviewed and haven't been yet, you can't move anything new to the Review phase.
The strict limits to both force teams to address bottlenecks. If a single senior person or team is responsible for "reviews" but are also otherwise occupied by their own work, forcing the rest of the team to stop to wait on them will make it clear quickly that more people need to be able to Review items to keep the team moving. Limiting In Progress items helps developers avoid context switching that significantly negatively impacts productivity.
Kanban's focus is on Cycle Time. How long does something take from the time work begins to the time it is delivered. This creates a focus on organizational efficiency and barriers to it.
The very real problem faced by software when it comes to project management is estimating. That's before we even factor in what estimating means to different people.
Project management demands timelines. In Waterfall, if we were able to lay out the plan for our entire project from start to finish with all of the task completion dependencies rolled up...we would still have to estimate the time of each task.
When you are estimating tasks that are done over and over, you can look to your historical execution time for insights. We average chopping X logs per hour.
If I hand you a Rubik's Cube and ask you to tell me how long it will take you to solve it...what is your answer going to be? If you've done it before you might have an idea. Maybe you just kept plugging away until you finally solved it? Maybe you've watched one of those videos about how to solve a Rubik's Cube, have practiced it and are confident that you could do it in 3 minutes flat? Maybe you have no idea?
Complexity and experience will dictate your answers, but then those estimates still only reflect your perception of the task...not everyone else on your team.
At best those numbers are a guess.
Every step in a software project is solving problems. You don't know how long it will take to solve the problem. You don't know how many new problems you might run into along the way. Can the new problems be solved by you or do they need to be solved by other people?
When your project plan is a collection of tasks with highly variable time estimates, where each task can sprout new tasks that may be assigned to other people at any point during the process...it becomes very clear that clinging to that Gantt chart isn't helping.
This is before we've even begun factoring in non-project related tasks in support of existing live systems. Or that different people will have a different concept of what "complete" looks like. Or over-engineering a solution that far exceeds the requirements because someone believed that was the "right way."
And here's the thing...as a business person that is hard to accept.
When you are working with budgets, marketing, press releases, investors...you need that information to be able to make decisions.
So we shift our expectations and start implementing Scrum. We are now embracing the Agile way of doing things. This seems like a reasonable compromise.
We will plan the work to be done in 2 week sprints. We'll only estimate a few stories and we will look to past performance to measure our velocity. As long as developers can make a Sprint Commitment then we should have a reasonable expectation of consistent development pace. You lock in the stories for the sprint and focus on getting them done.
Early on, the team is excited. Look how much better this is! You're even finishing things early and pulling in new stories to far exceed your commitments.
And then you find out something was underestimated. The person who is working on it doesn't want to ask for help, so their daily stand up report becomes remarkably vague. As you get close to the end of the sprint, this story still isn't done...and then it happens. You miss your sprint "commitment."
Is there a consequence? Does management notice that the sprint commitment was missed? Did the team miss its commitment because of you?
This is where commitments become such a loaded issue, especially for teams. If there is no consequence for missing a commitment, then it's just an arbitrary deadline. It becomes clear to the team that it's just an arbitrary deadline too.
If there is a consequence...did management forget about all of the extra stuff we did in the last sprint because we were ahead of schedule? Did that mean anything? Do we get rewarded for exceeding estimates?
As soon as any of these questions get introduced, you get a complex social dynamic. If missing a commitment means something but exceeding it means nothing, then teams become incentivized to under commit to ensure that they can hit the commitment. If exceeding the commitment isn't rewarded then you create a situation where the team is steered toward setting a low bar that they know they can hit. If exceeding the commitment is rewarded, then the team is still incentivized to undercommit so that they can outperform.
The other dynamics that creep in to this model are bad for the business as well. What if there is an issue in production that demands your attention immediately? Does the business still care that you missed the commitment because that was clearly higher priority? You'd be shocked to know how often somebody in management will still care about the commitment.
What about cross team communication? Do you make any new requests for collaboration, no matter how small, wait until you've hit ALL of your commitments first? What if somebody needs help on your own team? Do they go to the back of the line so you can cover your stuff first?
As the importance of hitting those commitments rises, you'll also see people start taking shortcuts. Do you just rip off the stickers and put them back on so it looks done? This is where technical debt comes from...taking a short cut to hit a deadline for something you'll fix later. Except somebody will build on that broken part before you get to it...or the fix will be deemed lower priority and problems will spread farther and farther. The best term I've seen for this phenomenon of measuring the spread and impact of technical debt is contagion. If the sprint deadlines are arbitrary and meaningless...why do we bother with them if we know all of the problems they introduce?
"When a measure becomes a target, it ceases to be a good measure."
- Goodhart’s Law; Charles Goodhart, Economist
Now...I've spent a lot of time talking about just a few of the issues with sprint commitments, so you should probably know that back in 2011 Scrum actually removed "commitment" in favor of "forecast" because of the sheer number of problems that stemmed from it. If you work for a company who's still doing sprint commitments...now would be a good time to let them know that it's wrong. Not a preference...wrong. Broken. Hurting your company.
The simple change to the word forecast fixes many of those problems because it removes the arbitrary deadlines that create problems. If something comes up in production, you fix it because it's clearly higher priority...and your forecast was a little high. If something isn't ready every single, 2 week period...it just continues with no reason to rush it out.
But all of those issues that we talked about happen because we were making commitments based on estimates. Estimates are guesses. Every 2 weeks you can't ask professionals to make commitments based on guesses and have any shred of hope for accountability.
The other point that we haven't even discussed is what estimates really are. Agile methodologies come with Story Point estimates rather than time estimates. You'll often see Waterfall trying to creep into your Scrum when, inevitably, somebody develops a formula to convert story points to time.
Story points are a measure of complexity, not time.
You and I can look at a Rubik's Cube and both of us can acknowledge that it's a complex problem to solve. Let's make it a 5 point story. If I know how to solve it already and can complete it in 15 minutes or so, but you don't and may take closer to 16 hours...that complexity estimate is still accurate. Translating it to time would be wildly incorrect.
My experience and existing knowledge of the problem allowed me to complete the task more efficiently. It’s exactly the same in software, where knowledge of existing libraries, algorithms, systems, permissions and even organizational politics will have a tremendous impact on completion time.
The moment that story estimating becomes time estimating...stop. That's not what it is there for and you're not trying to assign out every minute for a person.
The Scrum model actually fits well with contract development shops. They can commit to completing a certain level of work for a certain price and simply not be paid until the work is complete. They are both rewarded for finishing early and penalized for finishing late by that arrangement, with no incentive at all to expand the scope of work beyond the commitment. In order to make to the commitment, your client has to agree to pay you for what you're promising to deliver. The daily stand ups becomes an opportunity to report progress to a potentially eager client on a regular schedule too.
As it turns out, one of the more highly rated books on Amazon called Software Estimation: Demystifying the Black Art actually breaks down how easy software estimation really is...in a mere 300 pages or so. It's a very good book with a lot of very practical advice, such as creating clear terminology separation between estimates, targets and commitments or providing three estimates for each story like best case, realistic and worst case numbers...then re-evaluating the estimate once work as begun. It also highlights looking a past work to get a feel for how long a task will take to complete.
Past work as a measure of future time works well in a contract development shop as well, where you're likely doing the same thing over and over for multiple clients. There are a number of areas where that model fits. Sprint commitments or forecasts based on three estimates per story will remove the prediction goals that many project managers assume will come from the combination of estimates and velocity.
Which brings us to the last one...
If you're about to read this section and think that the result of this long journey is to "just use Kanban", you're about to be disappointed. While Kanban has a lot of strengths for software, it is not without faults either.
Kanban is much more closely aligned with reality than any of the other methods we have discussed so far. Context switching is bad. Developers work better when they can focus. Whether that focus is split because you're trying to complete too many things at once or because you were pulled into a scheduled estimating meeting while you were in the middle of working through a problem, the switch is the switch.
The problem from a business perspective becomes accountability. Many people see Kanban as a total lack of deadline or any incentive to complete things in a timely manner. This is also a bit unworkable from a perception standpoint, even though it’s not accurate.
I've worked in places where some employees did as little as possible until the company fired them. I've known people who actually wanted to get fired. The business folks are not out of line by showing concern over consistent performance goals. Usually things are called Key Performance Indicators (KPI). A KPI should only focus on goals that make the organization better. See Goodhart's Law.
Kanban is an ideal software project methodology when you remove accountability concerns from the equation. The best developers I know are obsessed with solving problems. If they are pulled in too many directions, when the end of the day comes they'll just turn their brain off and mentally check out for the evening.
If they are focused on a single problem...they will obsess about that single problem after hours. They will sleep on it. They'll be distracted while watching TV. They'll run to the computer at 10pm to test an idea that they had about the problem.
The next thing I'm going to say is important, so please read carefully.
As a business owner, you want those extra mental hours spent solving your problem. At the same time, you have to identify when you have people who do this for you and insistently make them take extra time off every now and then. Whether for their family, their kids or their health the last thing you want to do is burn out an asset like that by letting a person push themselves hard enough to create problems outside of work.
Developers aren't paid for lines of code. They are paid for solving problems. Often just working out the solution to a complex problem will end up resulting in only a few lines of code. Getting the lines correct is the task and extra brain power is devoted to that task when developers are focused. The net effect is that you create a longer work day without keeping the developer at their desk.
So...how do you take the continuous software lifecycle of Kanban and solve the accountablity problem?
A little bit of Pair Programming. But only a little.
Emphasizing "only a little" when talking about pair programming is important, because there are some development companies out there who utilize full time pair programming...and honestly that sounds miserable to a lot of people (myself included).
If you're not familiar with Pair Programming, it can be a very loaded term. At it's most basic level, it means two developers working on the same problem at the same time on the same screen. One of them writing code and verbalizing what they are doing while the other observes, looking for problems and offering suggestions.
Some people who adhere more strictly to it will go as far as having two developers at a desk, with an egg timer taking turns with the keyboard every time the timer dings. You can also do the same thing with tools like Screen Hero that allow two people to control the same machine at the same time.
I'm not suggesting you go that far, as you can get all of the benefits that we are after by simply having one person fill the observer role without the need to switch keyboards. It also gets complex as many developers prefer different environments or tools and may have muscle memory built up around keyboard shortcuts and workflow. In order to keep trading keyboards you have to have both developers utilize the same development environment, which isn't really needed.
Before we talk about the how, let's talk about the benefits of Pair Programming first. Developers don't often get to learn from each other beyond shared libraries and blog posts. Pairing solves by that making you take time to watch somebody else work and discuss what they are working on. As you see people do things that you didn't know about, you can ask how. If you see someone repeating a process over and over, you can suggest a way to improve it. You learn from how others work, you share knowledge of previous problems solved and you share knowledge of other related systems at the company.
This isn't the same thing as a meeting where your developers are looking to exit as fast as possible so that they can get back to work. It has the dual benefit of allowing your team to cross train and share knowledge while working through difficult problems. In many cases, developers don't want to ask for help. Whether that's from pride, desire to prove oneself or just trying not to bother people...it happens. If somebody is pairing with you, staring at the screen or constantly Googling for the solution isn't really an option. You can't help but discuss the problem. Additionally, when things are difficult, it's natural for your brain to find something easy...which is dangerous when you're sitting in front of the internet. That doesn't happen when you're pairing.
And all that you need to accomplish that is an expectation of about 2 hours a week worth of pairing. When somebody finishes a story, the expectation should be that they pair with somebody who is still working. Shake up who, randomize it, allow the project manager to decide it's the person who is earliest into the biggest story already. Let that little bit of pairing become the norm and it won't feel strange. If you start handing out pairing assignments only when someone is behind, it becomes a penalty. If you only do it when somebody feels it is necessary, it will be avoided. When you start pairing, it's an adjustment. It's different and there will absolutely be people who do not want to do it...and that is why keeping it to a minimum is so important.
Two of 40 hours is doable by everyone and the benefits still far outweigh zero of 40 hours. If you get somebody who is adamantly opposed, a simple concession of letting them focus on the observer-only role for a while can help to get them used to it.
It doesn't have to be a hard formula. Just try to spread things out reasonably. If somebody is working on a big story and hasn't had a chance to observe others, suggest they start one day by taking a break from their task to observe someone else, just to clear their head and get the time in. If something really is a difficult problem, the lone developer will have somebody who can vouch for the unexpected complexity.
I told you this was going to boil down to a simple solution. Kanban combined with about 2 hours per week of pairing gives you a very productive and workable solution for an accountable, productive software development process that allows your developers to maximize their focus and still swiftly adapt to the needs of the company.
The process still needs to be monitored. 15 minute, hard time limited daily stand up meetings are a simple way to do that without interfering with people. The only thing to be careful of here is that when you have a distributed team in different time zones, those meetings will interrupt people. Scheduling at the beginning of the day, end of the day or just before/after lunch where possible. If you can’t find a time that doesn’t interrupt people, allow those folks to just check in on the team chat room on most days of the week. Stand up meetings should be quick: What did you do yesterday? What are you working on today? Any blockers? The end.
Project managers and business people still need to work on prioritizing the backlog. You need to watch for slanted incentives coming from the business side of the house, such as sales commissions that accent acquisition but ignore retention. Often this will manifest as a prioritization of "New Feature X" and backlog "Existing Support Ticket Y".
If responding to support inquiries is high on your priority list, you have to be careful not to constantly interrupt existing work. In many cases it makes sense to dedicate specific people or teams to absorbing those production support requests to allow fast response while preventing developer interruption.
But bottlenecks, whether process based or people based, are the number one impediment to organizational efficiency. Kanban makes spotting those particularly easy. Without diving into it much farther, I'll leave you with this excellent piece from the Harvard Business Review called Collaborative Overload. It breaks down 3 different types of bottlenecks and how they can be addressed. The most dangerous kind are the ones most easily exposed with Kanban, so that you can get them resolved.
Kanban and light pairing will give you a flexible, reality driven approach to achieving the real goal that's driving your business: delivering as much value as possible as fast as possible.
// Share.on([hacker news,
linkedin,
twitter,
facebook,
reddit])