Disruptions to the Release Process, Pt. 1: Critical Bugs in Production
Photo by You X Ventures on Unsplash
As a Product Manager, my greatest wish is to be able to run off to an isolated mystical land filled with rainbows and sunshine and crank out release after release without any interruption. Unfortunately for Product Managers, releases do not occur in a vacuum. They occur while millions of other things are happening simultaneously across all other departments. With every release you manage, there is a dark cloud looming on the horizon made up of disruptions that can stop any or all progress towards your product. Here is Part 1 of Disruptions to the Release Process and how to handle them:
A critical bug appears in production that requires a hotfix
Regardless of how stable your product is, shit happens. Between software and hardware updates, accumulating tech debt, and general mistakes, you are bound to eventually experience the need for a hotfix in the middle of a release sprint. Timing matters.
Problem 1: A critical bug is found in production while your release is still in development.
Solution: It is “easier” to have development start on a fix and get that issue resolved before returning back to the scheduled product release. However, you should keep in mind that switching projects takes mental effort. One cannot easily bounce back and forth between projects with no time wasted. To switch from a focused project to another project that came from left field is a huge distraction to developers and distractions cost you time. You also need to consider that your QA Engineer is likely busy testing applied changes to the development environment, so they need to reallocate their time towards completing a full regression test or smoke test on the hotfix. This time spent on projects other than your product release eventually impacts your release date and pushes out other projects on your roadmap. It is not the worst case scenario, but it is important that you recalibrate your release date and communicate any changes to the company.
-
Problem 2: A critical bug is found in production while your release is in staging and in the QA phase.
This could be a huge blessing or a terrible curse.
Solution A: A huge blessing looks like finding the critical bug in production right before QA starts their regression testing on a product release that will likely only require one round of testing (AKA this is not a huge product release that requires multiple rounds of QA and bug fixes). You can easily slide a fix for the critical bug into the product release and push to production the same day.
Solution B: A terrible curse looks like finding the critical bug in production during the middle of your massive regression testing cycle for a product release that is large and impactful. This is a nightmare because in order to fix the critical bug, you have to first push the code changes into staging for testing. However, you already have the entire product release in your staging environment that has been updated and tested multiple times. You simply cannot release the hotfix without deploying the entire product release.
-
If you are able, you can try to remove the product release from the staging environment without causing too much damage. If the hotfix is that important to the health of your product and you can afford a delay for the product release, this is probably the path of least resistance. However, if you are too deep in the product release process and cannot remove it from the staging environment, you may have no choice but to apply code that hides the new product or quickly smoke test the entire thing so you can push the new product and hotfix into production together. While this choice can result in the introduction of new bugs into the system, you still maintained the path for your product to be released and fixed the critical bug in production. You can always follow up on this release with another hotfix to resolve any bugs you introduced with the rushed release.
None of these solutions work for me…now what?
You know your product and dev team best. You must know and be able to define the difference between truly critical bugs that block access to your product versus nasty bugs that block certain facets of your product. Give consideration to maintaining trust and credibility with your teammates while weighing your options. Can you mea culpa your way out of not fixing the critical bug for a few days? Can you influence developers and QA engineers to work after hours with you to get the product release out the door sooner so you can resolve the critical bug? Can you better explain the release process to customer-facing colleagues so they can hold down the fort while you try to get your product release out the door? Can you persuade your sales people that they can live off of vaporware while you take care of the hotfix?
Political influence is the big player here, especially when the solution to your problem is not clear.
Besides, at the end of the day, no one solution will satisfy every department the same way. It will always benefit you to keep politics at the forefront of your mind while you strategize on the details of the release.
