Why Executives Should Care More About Software Delivery

Note from Elias: Kudos to Andrei Vasilescu – one of my favorite engineers and my partner in crime in my YouTube days – for the peer review on this article.

We talk about product discovery a lot (as in too much). Let’s talk about delivery, and by delivery, I mean the actual deployment of your software. In recent years, people have started to think of delivery as things like writing tickets, burndown graphs, and managing backlogs.

When I think of delivery, I think of the actual deliverance of the software: release management. As a Product Manager, this is something you should pay attention to. We want to release at high quality as often as possible and minimize the risk when introducing something new.

The Current State of Software Delivery

Let me be clear: this is mostly an engineering topic. However, I observe a lot of confusion and antipatterns in the wild that prompt me to write this article:

• Executives and Product Managers do not know and care about how the software is built and released.
• Quality assurance is not part of the due engineering process.
• Software is released in batches (monthly, 6-month periods, etc.).

Why Should Executives and Product Managers Care?

Software products are awesome because we can make anything happen; it’s just bits and bytes, right? Sort of. I meet many executives who are annoyed that their teams are practically unable to react when things get ugly. A major bug or customer issue seems to take months to fix. A competitor seemingly makes leaps in no time with a smaller team. The company’s ability to make changes quickly is literally business-critical.

Why Have a Release Process?

Yes, you heard me right! I am a strong advocate against process-overload in software development (especially in Europe). But, and that is a capital BUT, some processes are critical to have and are critical to get right. Release Management is one of them. Top teams are on top of this – I am no engineer, but here are some simple process steps that you and your team should care about (there is a lot going on under the hood, typically your engineering team will take care of that).

1. QA/Staging Instances to Test Release Candidates
2. Proper Unit and Integration Tests
3. User Tests Along Defined Journeys
4. Thorough Tests of New Features
5. Ability to Rollback
6. Ability to Cherry Pick
7. Ability to Flag-Flip New Functionality
8. On-Call Rotation
9. Release No Less Than Weekly (for most products)
10. Launch Checklist for Larger Launches

That sounds like a lot, but well, that is the core technical competency of your company, and you better get good at it. The benefits will manifest, especially over long periods of time, making you faster, not slower, and you will have substantially fewer screw-ups and technical debt. Some refer to this concept as CI/CD, continuous integration, continuous deployment.

Staging and Testing

We want to stage our release candidates on our staging or QA instance, which is basically a non-public version that mirrors the latest version of the code so you can test the difference from the current production version. For new code, the bar should be high in terms of code tests. I know it is frustrating if something is ready, but engineers tell you they haven’t done the test yet. As the PM or exec, you should strongly encourage that. Proper test coverage will make your life easier, especially over longer periods of time. If a new release has side effects, those tests will put them on your radar before the users find them.

Release Automation

The next step is release automation, i.e., capturing the critical user journeys as click-by-click instructions and letting those tests run without human involvement. I still recommend an actual user clicking through your bread-and-butter journeys; they will find an issue more often than not. The journeys can be defined by the Designer, PM, or Tech Lead on the team and should be conducted as part of the release process by a diverse group of people, including engineering, customer success, PMs, etc.

A Note on Major Launches

Every now and then, you will have a release coincide with a launch, the release of a big update or important feature. That often coincides with a bit of a buzz in terms of PR or announcements. These should be rather rare and should be given special attention and care. I recommend checklists for bigger launches, in which key stakeholder groups give their sign-off (Execs, Product, Engineering, sometimes Legal or Data Privacy). I also recommend thorough testing, including end-users, to ensure things work as expected.

A Note on On-Call Rotations

I am shocked that many software companies do not seem to have this concept. My teams always had an on-call rotation, which essentially means one or two engineers are running point throughout the week for releases and anything else that goes wrong. That process creates clear ownership in the engineering team for excellent release management, but it is also a good way to ringfence resources for technical debt. Aside from release management, on-call engineers deal with incoming ad hoc user issues, refine the process, automate time-consuming engineering tasks, and refactor old code that is known to need an overhaul. Over long periods of time, this will immensely help to keep your code quality decent. Product Managers have no business being part of the on-call rotation, but I highly recommend at least participating in the conversation when teams hand over to the next – it is a great opportunity to learn about the way the software works and regular user issues and bugs.

The Pitfalls of Batch Releases

First of all, you want to be able to react quickly. We hear companies try to be agile all the time. How agile can you really be when it takes you weeks or even months to get something out the door? At one point in my career, we noticed a major bug resulting in us not applying the targeting advertisers had paid for. Within one hour, we were able to find the issue. Within about three hours, we had fixed all affected campaigns, informed customers, and fixed the issue going forward, including test coverage for these kinds of issues.

Second, even a weekly release is already complicated, as many different code contributions need to be tested in isolation and in concert. Imagine you cram many months of work into a single humongous release. Your engineering team will crumble under the weight of dealing with the dependencies alone, and this is bound to introduce delays. Even when you finally get it out the door, I can guarantee that your customers will find many bugs and issues – but hey, it will only take a few months to fix them…