Coding is to a software engineer like a hammer is to an architect. Coding isn’t what makes us engineers, it’s the problem-solving methodology we use. Too often we are falling into the trap of thinking that our job is writing software--coding. It’s not. Our job, alongside other teams, is to solve the problems of the business.
Sometimes I see people behaving more like order takers, coding up what is asked of them, without clear understanding of the problem they are trying to solve or whether the solution proposed will be effective at solving it. This approach relies on the assumption that the requester has already fully understood the problem and has done the engineering work. This approach isn’t optimal for the business. If you’ve ever been pushed to solve a problem without understanding it, or you think you have a better solution but stakeholders won’t listen, or you think you’re working on something that isn’t important, you know how unfulfilling that feels.
Of course, our product management partners and our internal stakeholders often have done some of the engineering work. The line between who does what should be wide and fuzzy. That doesn’t absolve us from doing our own work. You can’t lend your perspectives to the team if you don’t understand the reasons behind what you are asked to do.
One of the reasons we behave this way is that some of us have never been taught how to be an engineer. Colleges and boot camps lean toward teaching software methodologies, but barely gloss over the engineering skills needed to do our jobs well. It’s like an architecture major only having courses like “basic hammer technique,” “saw sharpening” and “making buildings square and level.”
So, let’s try to fix that!
Engineering tasks fit into a broader context of visions, roadmaps and multi-stage projects. But at every abstraction level, engineering methodology has the following common steps:
1. Understand the problem statement
2. Determine the variables that influence the problem
3. Analyze effective ways to manipulate those variables
4. Make a plan for the most cost-effective way to fix the problem
5. Implement a solution (Do the coding if required)
6. Check that the change solved the problem
If we think of ourselves as merely software developers or coders, we will naturally skip important steps. We will tend to focus on step 5--perhaps because we don’t understand how to do the other steps. Take the following as a quick introduction to engineering. Study harder where you are weakest. Practice and get coaching. Don’t just be a coder--be an engineer.
Understand the Problem
Requirements often come to us framed as solutions: “We need to add +4 to zipcodes,” “We need performance to be 10% faster,” “We need a tool to manage compensation changes.”
While these may be the correct solutions, you shouldn’t accept requirements framed this way. Push back on your PM and stakeholders until you understand the underlying problem these solutions are trying to address: “Our tax charges are wrong for some jurisdictions,” “Customers abandon orders if response time is too long,” “Managers spend too much time adjusting compensation and we still have unexpected errors.”
Asking for details about the goal of the change--the business value--helps you understand if your specific solution is helping that and it helps you consider the right level of effort for fixing the problem. It is also the only framing that gives you a chance to consider other, better ways of fixing the problem. Without the underlying problem statement, you can’t participate as an engineer.
Consider using the “five whys” technique: When given a requirement, ask “why is it needed?” When you get that explanation, ask “why” again, peeling back the onion for deeper understanding. Keep going until you feel like you fully understand the root problem. Don’t give up too early. There’s no magic to five; keep going until you feel like you’ve found the root cause of the problem.
Another key aspect to understanding a problem is to understand it mathematically. Engineers apply science and math to problem solving. If you don’t have the numbers, you don’t yet understand the problem with sufficient depth to solve it. Words and phrases like “wrong,” and “too long,” “too much,” and “unexpected” are insufficient for problem solving. If you get these kinds of answers, push for clarity on the numbers behind it.
Remember that most engineering problems are statistical problems. Order rates for example have statistical variation month to month. If you don’t have a grounding in concepts like standard deviation, modes, medians, and statistical significance, learn those things. Otherwise, you’ll find yourself chasing phantom problems like “Why did two people leave my group this month?” For example, most of our numbers don’t make sense as averages--modes and medians are usually more interesting.
Finally, it is important in this first step to know what good looks like. Usually this means you need a numerical measurement of what you are trying to achieve. Armed with a measure of good, you can test your results to see if you have solved the problem. This helps you focus and can even stop you from delivering more than is needed.
Determine the Variables
Now that you understand the problem, the next step is understanding the factors (variables) that influence the problem. For example, the speed of database calls depends on the structure of the database, the size of the data set, the power of the hardware, the format of the database call, etc.
If the problem you are trying to solve is that we get application errors whenever a database call takes longer than ten seconds, knowing those variables for database speed gives us a range of options for fixing the problem including pruning stale data, beefing up the hardware and restructuring our longer-running database calls.
Thinking about the same problem in terms of variables might lead us to asking some more “why” questions, “Why do we get an error at ten seconds,” leading us to think of the other variables to our problem.
There is a technique called a “fishbone diagram” that can help you map out the variables of your problem. You start with your problem statement as the head of your fish then draw a spine off that. Then you draw bones of that spine for each cause/factor/variable of that problem. You can further breakdown those causes with “why” questions to get more detail and insight into the root causes. Think of this as a fancy brainstorming session.
Manipulating the Variables
Once you have a clear picture of the variables impacting the problem statement, you are ready to consider which ones are worth attacking. There are a few factors to consider for each variable:
- How easy is it to change?
- Is this a variable I’m permitted to change?
- How big an impact does the change have?
Unfortunately, the ease of changing a variable is not just measured by its cost. You also need to account for the risk that your solution won’t work or will cost more than you planned. In these cases, you need to express your costs with these extra factors: 5-8 story points--10% chance of failure.
You also need to recognize the limits of the variable's impact on the problem. For example, pressing the gas pedal on your car makes it go faster only as long as the limit of the engine’s RPMs can support it. And in some cases, you don’t have permission to change certain factors, like when the code is owned by another group. In which case you have to add in the costs of getting permission or getting help. One caution though, never create a sub-optimal local fix because it seemed too difficult to work with another group to fix it correctly. That is, don’t forget upstream and downstream impacts of each approach.
Besides the cost of changing each variable, you need to consider the impact of each change. You may be able to get 20% improvement by uprevving hardware, but 50% improvement by tuning a database query. On the other hand, you might get a 100% improvement by removing the need for the query altogether.
Remember, your solution space doesn’t need to be limited to solutions that can be implemented with software. As an engineer, look for the best solutions to each problem, not just the solutions that can be implemented with your bag of coding tricks. For example, if the problem you are trying to solve is the amount of time it takes to fix a customer data entry error, you might be better served by making a change that helps the customer avoid the error in the first place. Consider using better documentation, better training, process improvement, or even removing a low-value, error prone feature.
Make a Cost-effective Plan
Finally, after understanding the problem and the variables that impact it, pick the set of variables you need to change to get the impact you need. That plan should consider not only the cost, but the likelihood of success, the time to value, and any other factors that would help us make the right choice.
Of course, you can’t make this plan unless you understand the factors that make us successful, that is, unless you understand the problem and all the business needs. That’s where you should leverage your business partners. Present your plan in terms of options, not absolutes. Use the skill of your PMs and business partners to help pick the best options based on factors you might have missed or misunderstood.
For example, you might present a solution that delivers 50% of the value in half the time, but will need to be reworked in two years and another solution that delivers 90% of the need in double the time, but will continue to scale with the business.
The key is that you can’t do this work in a vacuum.
Do the Work
When you decide on the work to be done, do it. If your plan has some uncertainty or some unexpected hurdles come up, pull in your stakeholders and replan. Replanning will happen, but check yourself if it’s happening every day. Agile methodologies always have a retrospective (replanning) step at the end, which is a good reminder to re-evaluate progress. Finally, as a rule of thumb, hope is not a plan. Feeling anxious about a deadline is your body’s way of saying it’s time to raise the red flag and ask for help--it gives stakeholders the most room to help.
Check the Results
The final step in the engineering process is to check the results. This is not a testing or QA step. Those are designed to confirm that you implemented what you said you would implement. This step is the engineering step to confirm that what you coded actually solves the problem you set out to solve. It goes back to the measurements you took when you set out to understand the problem statement.
For many problems you can tell right away whether the solution worked. But for a huge number of interesting problems, the results are harder to see. For example, if you are trying to reduce the variability of a margin report, it may take many cycles to get enough sample sets to tell if your changes had the desired impact.
