Why We Wait in Lines

It sounds like such a simple question--we wait in lines because there are people ahead of us--but why are there so many people ahead of us? And why can't they processes us faster? Didn't they plan for this? What's taking so long? Few things makes us lose our patience more than waiting in line.

The study of lines is called queue theory--'queues' like the British call them. The British make things sound more refined. Queue theory is important in computer programming, manufacturing, traffic flow, buying groceries and making your next visit to Disney World happier. Recent news about container ships stuck in port is an example of queue theory in action. Unfortunately, the science of lines is not very intuitive. I hope some examples will make it easier to understand. Perhaps this will make you more patient, or help you set up your own lines more effectively.

A toll plaza for example has to take tolls from all the cars that travel on a road. If there are 600 cars each hour, then 600 tolls need to be collect in an hour. Again, this sounds obvious, but it's more subtle than it seems. Imagine if they can only collect 599 tolls every hour. Then at the end of that hour, one of the cars wouldn't be processed and would go into the next hour's pile. On hour two, there would be 601 cars to process, but only capacity for 599, leaving 2 extra cars. After a day, there would be a permanent line of 23 cars, getting bigger ever day forever. Eventually, there'd be no room for new cars to get on the road.

How long is a 23 car line, or any line for that matter? The math is simple: the number of cars waiting (or people, or shipping containers) multiplied by the average time to process a single car. In this case, 599 cars an hour is just over 6 seconds per car. So a 23 car line is about a 2 minute wait.

You will rightly object that my example is too simple: Lines don't work that way. Sometimes there are 600 cars, and sometimes there are 500 cars or even 0 cars. The line goes away when traffic is lower. And, yes, that's true. We have intuition for that from commuting. There are lines (and traffic, another subtle queue problem) at rush hour when the road is at capacity, but that all goes away when the commute is over.

A funny thing happens with lines that struggle to empty out. Let's change the numbers on our toll example. Imagine the toll plaza is set up to move 300 cars an hour (7200/day), to handle the average flow of 250 cars per hour (6000/day). Sounds like plenty of extra capacity. During rush hour there are 600 cars gumming up the road. The system sounds like it can handle it--everything should be fine. Sounds like about an hour delay, but the rest of the day, the road has plenty of capacity for the extra 350 cars. Just saying it out loud makes that obviously false. Let's walk through what really happens.

During that rush hour, the first 300 cars get through. At the end of rush hour, we have a line of 300 cars, but an extra 250 have come along. The second hour, every one of those 250 cars waits in line for an hour for the 300 cars ahead of them to pay their tolls. The third hour, another 250 cars show up making 500 cars in line, but only 300 get through. The fourth hour another 250 come in, and wait behind the 200 in front of them for about 40 minutes each. That leaves 150 cars at the start of the fifth hour joined by another 250. In the sixth hour, the toll plaza processes 300 of 350 cars waiting. Finally by the end of the seventh hour, the line is cleared up and everything runs smoothly until the next day's rush hour. The line lasted for seven hours!

See how if you run closer to the edge of your capacity, you have less room to catch up. The result is lines that can last a long time.

The easy solution is to plan extra capacity for increased demand, which is fine, but can be expensive. You can guess when rush hour happens, so perhaps you suggest doubling the staff for that one hour so the line never forms. People don't tend to like to have jobs that last for just one hour, so you'll have to be creative and figure out how balance the workload in ways that extra toll-takers are willing to come work for you, but you can do that.

Ideally, you'd have enough toll lanes and toll collectors to process rush hour, and shut those down to save money during the rest of the day. Say that a toll lane can process 100 cars per hour, you might have 6 lanes during rush hour and 3 lanes the rest of the day. Unfortunately, this plan falls apart too, because systems don't run evenly enough: a collector in one lane needs a bathroom break, another toll collector works slower than the others, cars don't evenly distribute across all the lanes, one driver has to hunt for change and takes 30 seconds to process, another car gets pulled over for speeding causing a clump of traffic to rubberneck and all show up at the same time. You get the point. Each of these is a mini rush-hour causing it's own mini-traffic jam, the effects of which can take hours to clear up if you are running too close to the capacity of the toll plaza.

Lines happen, fundamentally, because demand is not steady. The little bumps of life cause demand to temporarily exceed capacity. Short times where capacity exceeds demand cause little lines. Lines take longer to dissipate than our instincts would tell us, then another short bump comes along, making those lines longer ... and longer ... and longer.

Lines happen when two things are true: First, the little bumps of the system temporarily exceed the capacity of the system, and second, when those bumps happen faster than the previous bump's line is cleared out. In other words, lines happen when you plan to be too efficient, and when systems are less predictable.

As a rule of thumb, you can plan for about 25% extra capacity than you need to account for this. When capacity is set right, there is almost never a line for long and it looks like you've got way too much capacity: people are sitting around, doing nothing for a lot of the time. Managers like their people to look busy, so their boss doesn't cut their budgets. Systems either look inefficient, or they have long lines all the time. There is no way to balance these things perfectly.

You have an example of the need for excess capacity with your computer. If you've ever looked at it's processor utilization, you will have noticed that it usually is running about 5-10% of capacity. When you notice it starting to slow down, you will be running at around 70% capacity, and in the snap of a finger everything stops working and you're using 100% capacity. Time to reboot!

As much as we love the idea of of an assembly line, the approach of having stations to do part of the work in a series has an impact on queues in manufacturing. Manufacturing engineers work hard to make sure that each step in a process takes about the same amount of time as every other step. The speed of parts through an assembly line is limited by the time it takes to do the slowest step. We see this problem play out in the container ships stuck in port. To get products to market, first there is a limit to how many spots there are for ships, and a limit to the cranes to unload the shipping containers, and a different limit to the number of trucks available to move those containers to their destination. If the truckers don't move the first shipping containers out of the port, there isn't room for new ones, which blocks up ships in the harbor--a line caused by capacity issues in an upstream process.

Disney has an interesting approach to lines. They make them part of the attraction; part of the fun. If you are fortunate enough to not wait in line for one of their big ticket attractions, walk through it slowly anyway. You don't want to miss all the fun details they've added to make the line more tolerable.

If you would rather avoid your systems having long lines, there are a number of things you can do. First, avoid the tendency to run at the peak of efficiency--allow yourself to have excessive capacity. You need it to deal with the variability of demand. Second, reduce variability in demand. Typically, that means looking for ways to reduce errors either by you or the customer. For example, don't put the ice cream flavor sign where only people at the front of the line can see it. It can help to have separate lines for different kinds of transactions. It's why grocery stores have fast lanes. Overall, the store has variability, but each type of line has less variability.

One final thought about lines. Don't make them up for yourself. Companies often make this mistake with things like repair services. You ask how long it will take and they say something like, "Just an hour or two, but I've got a bunch of people ahead of you. Let's schedule you for next week." They've set up a backlog of work that doesn't need to be there. Every day, they have to do one day's worth of work. That is, the shop has to have the capacity to handle all the work, or the line will grow forever. If they could get rid of the line, by adding a little extra capacity, they'd give the customer a better experience. As an added benefit, they wouldn't have to pay for space to store all the work waiting in line, or pay for a system to track all that work. When processes can finish jobs in under a day, the best process does today's work today.

Be an Engineer

Engineering is the process of applying science and math to solving real-world problems.

Coding is to a software engineer like a hammer is to an architect. Coding isn’t what makes us engineers, it’s the problem-solving methodology we use. Too often we are falling into the trap of thinking that our job is writing software--coding. It’s not. Our job, alongside other teams, is to solve the problems of the business.

Sometimes I see people behaving more like order takers, coding up what is asked of them, without clear understanding of the problem they are trying to solve or whether the solution proposed will be effective at solving it. This approach relies on the assumption that the requester has already fully understood the problem and has done the engineering work. This approach isn’t optimal for the business. If you’ve ever been pushed to solve a problem without understanding it, or you think you have a better solution but stakeholders won’t listen, or you think you’re working on something that isn’t important, you know how unfulfilling that feels.

Of course, our product management partners and our internal stakeholders often have done some of the engineering work. The line between who does what should be wide and fuzzy. That doesn’t absolve us from doing our own work. You can’t lend your perspectives to the team if you don’t understand the reasons behind what you are asked to do.

One of the reasons we behave this way is that some of us have never been taught how to be an engineer. Colleges and boot camps lean toward teaching software methodologies, but barely gloss over the engineering skills needed to do our jobs well. It’s like an architecture major only having courses like “basic hammer technique,” “saw sharpening” and “making buildings square and level.”

So, let’s try to fix that!

Engineering tasks fit into a broader context of visions, roadmaps and multi-stage projects. But at every abstraction level, engineering methodology has the following common steps:

    1. Understand the problem statement
    2. Determine the variables that influence the problem
    3. Analyze effective ways to manipulate those variables
    4. Make a plan for the most cost-effective way to fix the problem
    5. Implement a solution (Do the coding if required)
    6. Check that the change solved the problem

If we think of ourselves as merely software developers or coders, we will naturally skip important steps. We will tend to focus on step 5--perhaps because we don’t understand how to do the other steps. Take the following as a quick introduction to engineering. Study harder where you are weakest. Practice and get coaching. Don’t just be a coder--be an engineer.

Understand the Problem

Requirements often come to us framed as solutions: “We need to add +4 to zipcodes,” “We need performance to be 10% faster,” “We need a tool to manage compensation changes.”

While these may be the correct solutions, you shouldn’t accept requirements framed this way. Push back on your PM and stakeholders until you understand the underlying problem these solutions are trying to address: “Our tax charges are wrong for some jurisdictions,” “Customers abandon orders if response time is too long,” “Managers spend too much time adjusting compensation and we still have unexpected errors.”

Asking for details about the goal of the change--the business value--helps you understand if your specific solution is helping that and it helps you consider the right level of effort for fixing the problem. It is also the only framing that gives you a chance to consider other, better ways of fixing the problem. Without the underlying problem statement, you can’t participate as an engineer.

Consider using the “five whys” technique: When given a requirement, ask “why is it needed?” When you get that explanation, ask “why” again, peeling back the onion for deeper understanding. Keep going until you feel like you fully understand the root problem. Don’t give up too early. There’s no magic to five; keep going until you feel like you’ve found the root cause of the problem.

Another key aspect to understanding a problem is to understand it mathematically. Engineers apply science and math to problem solving. If you don’t have the numbers, you don’t yet understand the problem with sufficient depth to solve it. Words and phrases like “wrong,” and “too long,” “too much,” and “unexpected” are insufficient for problem solving. If you get these kinds of answers, push for clarity on the numbers behind it.

Remember that most engineering problems are statistical problems. Order rates for example have statistical variation month to month. If you don’t have a grounding in concepts like standard deviation, modes, medians, and statistical significance, learn those things. Otherwise, you’ll find yourself chasing phantom problems like “Why did two people leave my group this month?” For example, most of our numbers don’t make sense as averages--modes and medians are usually more interesting.

Finally, it is important in this first step to know what good looks like. Usually this means you need a numerical measurement of what you are trying to achieve. Armed with a measure of good, you can test your results to see if you have solved the problem. This helps you focus and can even stop you from delivering more than is needed.


Determine the Variables

Now that you understand the problem, the next step is understanding the factors (variables) that influence the problem. For example, the speed of database calls depends on the structure of the database, the size of the data set, the power of the hardware, the format of the database call, etc.

If the problem you are trying to solve is that we get application errors whenever a database call takes longer than ten seconds, knowing those variables for database speed gives us a range of options for fixing the problem including pruning stale data, beefing up the hardware and restructuring our longer-running database calls.

Thinking about the same problem in terms of variables might lead us to asking some more “why” questions, “Why do we get an error at ten seconds,” leading us to think of the other variables to our problem.

There is a technique called a “fishbone diagram” that can help you map out the variables of your problem. You start with your problem statement as the head of your fish then draw a spine off that. Then you draw bones of that spine for each cause/factor/variable of that problem. You can further breakdown those causes with “why” questions to get more detail and insight into the root causes. Think of this as a fancy brainstorming session.


Manipulating the Variables

Once you have a clear picture of the variables impacting the problem statement, you are ready to consider which ones are worth attacking. There are a few factors to consider for each variable:

• How easy is it to change?
• Is this a variable I’m permitted to change?
• How big an impact does the change have?

As an engineer, you need to understand not only the problem, but how effective your tools are at interacting with that problem. We often measure that as a cost in either time or dollars. Implementation time--story points--is often sufficient for our needs, but there are times when you need to think about the problem in dollars. (For example, when considering the impacts of adding hardware, or improving performance.)

Unfortunately, the ease of changing a variable is not just measured by its cost. You also need to account for the risk that your solution won’t work or will cost more than you planned. In these cases, you need to express your costs with these extra factors: 5-8 story points--10% chance of failure.

You also need to recognize the limits of the variable's impact on the problem. For example, pressing the gas pedal on your car makes it go faster only as long as the limit of the engine’s RPMs can support it. And in some cases, you don’t have permission to change certain factors, like when the code is owned by another group. In which case you have to add in the costs of getting permission or getting help. One caution though, never create a sub-optimal local fix because it seemed too difficult to work with another group to fix it correctly. That is, don’t forget upstream and downstream impacts of each approach.

Besides the cost of changing each variable, you need to consider the impact of each change. You may be able to get 20% improvement by uprevving hardware, but 50% improvement by tuning a database query. On the other hand, you might get a 100% improvement by removing the need for the query altogether.

Remember, your solution space doesn’t need to be limited to solutions that can be implemented with software. As an engineer, look for the best solutions to each problem, not just the solutions that can be implemented with your bag of coding tricks. For example, if the problem you are trying to solve is the amount of time it takes to fix a customer data entry error, you might be better served by making a change that helps the customer avoid the error in the first place. Consider using better documentation, better training, process improvement, or even removing a low-value, error prone feature.


Make a Cost-effective Plan

Finally, after understanding the problem and the variables that impact it, pick the set of variables you need to change to get the impact you need. That plan should consider not only the cost, but the likelihood of success, the time to value, and any other factors that would help us make the right choice.

Of course, you can’t make this plan unless you understand the factors that make us successful, that is, unless you understand the problem and all the business needs. That’s where you should leverage your business partners. Present your plan in terms of options, not absolutes. Use the skill of your PMs and business partners to help pick the best options based on factors you might have missed or misunderstood.

For example, you might present a solution that delivers 50% of the value in half the time, but will need to be reworked in two years and another solution that delivers 90% of the need in double the time, but will continue to scale with the business.

The key is that you can’t do this work in a vacuum.


Do the Work

When you decide on the work to be done, do it. If your plan has some uncertainty or some unexpected hurdles come up, pull in your stakeholders and replan. Replanning will happen, but check yourself if it’s happening every day. Agile methodologies always have a retrospective (replanning) step at the end, which is a good reminder to re-evaluate progress. Finally, as a rule of thumb, hope is not a plan. Feeling anxious about a deadline is your body’s way of saying it’s time to raise the red flag and ask for help--it gives stakeholders the most room to help.


Check the Results

The final step in the engineering process is to check the results. This is not a testing or QA step. Those are designed to confirm that you implemented what you said you would implement. This step is the engineering step to confirm that what you coded actually solves the problem you set out to solve. It goes back to the measurements you took when you set out to understand the problem statement.

For many problems you can tell right away whether the solution worked. But for a huge number of interesting problems, the results are harder to see. For example, if you are trying to reduce the variability of a margin report, it may take many cycles to get enough sample sets to tell if your changes had the desired impact.

Reserved Executive Parking Leaves a Bad Impression

I pulled into the lot of a large consumer products company a little before 9:00 for a networking meeting. The lot was crowded with employee's cars. I took a visitor spot near the front, although I prefer to park further out. I noticed there were four empty spots close to the door labeled "Reserved Executive Parking."

That left an amazing first impression for me. First I was struck by the wording: "Reserved Executive Parking." What was wrong with simply "Reserved Parking?" What was the benefit of adding "Executive?" I am still trying to imagine the series of meetings or the considerations that went into that decision. Someone made a point of adding the word I assume with some purpose of highlighting rank. Either someone thought that the executive needed the extra respect, or the executive asked for the rank designation and nobody was comfortable saying it would make them look arrogant.

The second thing that struck me was that the executives spots were closer to the door than the visitor spots. It paints a clear picture of who is most important in that culture. Imagine the impression this leaves on visiting customers.

It didn't strike me at first that the spots were empty. I talked to an employee who shared how busy he had been, working late most nights. I remembered that the parking lot was crowded when I came in. Each employee who came in early would walk by those four empty spots. Imagine the impression those employees got.

I had a nice, long meeting and left around 10:30. When I walked out, two of the spaces were occupied, one with a Mercedes and one with a BMW. I certainly don't begrudge a successful executive the reward of a nice car, but no, these weren't nice cars. These were the kind of cars that Mercedes and BMW owners are envious of: top of the line in every way. They were beautiful machines. They didn't also need the label "Executive."

Bad Meeting Math

Some people use their meetings as a way to gather status, in part because it's easier than getting it one by one. Consider the math of that. One leader gathers nine team members for a one hour meeting, for a total of 10 staff-hours spent, with about six minutes per person of value.

The alternative: one leader has a 15 minute 1-on-1 with each of the nine team members. That takes the leader two hours and 15 minutes with an equal amount of time from the team members, for a total of 4.5 staff-hours.

15 minutes is more than double the six minutes available for each person in the one hour meeting with less than half the time cost to the team.

If the leader is optimizing for their own time, the meeting works best. If they are optimizing for team effectiveness, 1-on-1 is the clear winner.

Of course, you might argue that everyone benefits by the shared meeting experience. That is true for some meetings, but unlikely for the typical round-table status meeting. If in doubt, ask your team if they get value from your status meetings, and ask yourself if you are holding them for your convenience or for the greater good of the team.

Earning Respect as a New Manager

I was recently asked to advise a new manager who was having trouble getting respect from her new team. She is now managing the group she used to be a member of, as many first time managers are. From the start, she could feel the resentment from some of her new staff that she was promoted. They would show up late to her meetings and sometimes not show up at all. One of her staff dismissed her in a group meeting, "Your idea for this project isn't important. You've never worked in this area before." Wow!

A lot of new managers struggle with this transition, particularly when they come from the group they are assigned to manage. This can be even worse when the new manager is younger than some of their new staff, which is also the case here. Fortunately, it is not too difficult a problem to deal with. I'm a bit disappointed that she didn't get more support and coaching from her own manager.

The first key to solving this problem is an attitude change ... on the part of the new manager. While respect is important, it is not respect for authority that is at issue here. Everyone should expect basic respect independent of role, rank or authority. A new manager needs to have the attitude that their job is just a role on the team like everyone else on the team. Authority comes into play at times, but largely, managers have a role on the team to coordinate work, administer various company processes and to support their team members doing their roles. Occasionally, that means holding team members accountable sometimes by exerting authority.

I coach new managers to review each member of their new staff with an eye to understanding who might have issues with the change. We sit down and review their analysis together, and make a plan of action before we announce the new role. But, it is never too late for this analysis or to make a plan to fix respect issues that you didn't catch early.

For each such person, schedule a one-on-one meeting and address the issue head-on using the standard "When you"/Result/"I need" pattern, "When you show up late to our team meetings, the whole team is less effective. I need you to come on time in the future." For some that will be all you need to do. You may get a half-hearted agreement and perhaps even an eye roll. If you do, that's fine, it's part of the process.

The next step is to bring the disrespect into the open, "Do I understand correctly that you are unhappy with me as your manager?" After either "yes" or "no" explain your role with a little bit of an appeal to authority, "Our manager gave me the role of ... I play that role to help you and the whole team be effective. You have the role of ... I need you to respect my role on this team in the same way you need me to respect your role." Keep this emotionally neutral. You are not angry, you are fixing a problem.

The next part is key, a challenge with some commitment to change, "Can we work together respectfully, or do we need to deal with this issue in another way?" In many cases this will be enough. You may need to reinforce the agreement to make it stick. Next time they act disrespectfully, call them into your office as soon as possible, "That was not how we agreed you would behave. Will we continue to have a problem?"

In most cases, this will solve the problem enough for the new manager to build new cultural norms. If the problem persists with a particularly difficult person, escalate fast to put them on a plan and help them move on. If the team sees that you will tolerate the problem, all hope is lost, and your manager will be putting you on a plan. When you let these problems fester, your team will be ineffective. I need you to fix the problem for your team. That's your role.

How a Moderator Can Deal with a Bully

Watch some of this video from the second 2012 presidential debate. Candy Crawley has to moderate (that is, lead) a debate between two people who are used to being the leader in their own worlds. Pay particular attention to the body language of Mitt Romney.



There is a key manipulation technique being used here by Romney and to a lesser extent by Obama. Notice how Romney walks forward toward Crawley as he tries to take over the conversation. That is a bullying technique. It is aggressively asserting power including a subtle threat of assault. Walking toward someone like this is a push for the other person to back down. And, as he gets closer, he makes himself appear both larger and higher in Crawley's vision as she is seated. This is another way of asserting power.

So, Crawley is put in an awkward position as the leader of the debate. She is being bullied by a presumptively powerful man. In most situations, he would expect to demand the leadership role in a room. But, Crawley is the leader in this situation, and she must retain the leadership in order to make the event successful. Part of the awkwardness comes because she can't rudely stand up and tell him to back off and follow the rules. But, she still needs to back him down. You can decide how good a job she did.

We find ourselves in these situations when we have to lead a meeting that includes managers above us in hierarchy. When I find myself in this situation, I emphasize the role I've been asked to play, and use the magic phrase "I need." It works wonders. "My role here is to keep us on track. I need to ask you to hold that thought for a few minutes." It's tough, but it's your role as the leader.

Women and Office Politics

As the father of two daughters, I'm keenly interested in the role of gender in business. This Harvard Business Review blog article caught my attention: Three Ways Women Can Make Office Politics Work For Them.

I particularly liked the Mary Matalin quote, "This business about politics at work being sleazy drives me crazy. Virtue can be the essence of politics. The reality is that politics can be just as virtuous or as sleazy as you are." That's a lesson that breaks the gender barrier.