Why Does it Often Take Nearly a Month to Ship a Few Lines of Code?

Ever wondered why it takes nearly a month to ship out a few lines of code change to the star and reliable customers of your company? When the changes made are formally aligned with the product, marketing and application managers, what stops it from being shipped immediately? Why do managers quote timelines for a maintenance release that seems so "unrealistic" to you? These were the thoughts that were going through my mind during my first few months of writing production-quality code.

Back in college, I always thought that completing the project was all about development and writing never-ending column lines of code. And once an initial draft of the feature is done, we wrap it up. All done. No peer code reviews, no documentation, nothing. Just raw files of code written with some comments here and there. It still works, serves the purpose. We never think about maintainability, readability, scalability, etc. Duh! You are the one who came up with the feature requirement, you are the one who designed it, you are the one who developed it, you are the one who tested it, you are the one satisfied with it.

“More importantly, our software worked. I don't just mean that it didn't bump, or that it performed according to the written specifications, or that it was efficient in producing reports. It really worked”

But, things seem to be different in the corporate world.

There is something called "process" and the success/outcome of the process depends on the team. The teams should have enough patience and discipline in following it. We will be talking about the so-called process in this article. Different companies follow a different process, I will try to make this article as generic as possible for you guys to relate and connect with the content. Please feel free to share about the process followed in your company in the comments section.

Step-1: Software Product Proposal

The marketing and the sales division of your product line would analyze the market trends of your product offering and would briefly understand the customer's requirements at a high level (some customers tend to give more details and insights, but a general lot just give high-level details as you have still not shown any signs of commitment to them. It also depends on a broad market/narrow market that you are targeting). The marketing team's job is summarized by the below quote.

"In programming the hard part isn’t solving problems, but deciding what problems to solve."

Once the marketing and the sales team find a promising ROI on it based on the previous analysis, they will go ahead and create a Software Product Proposal that briefly explains the position of the company in the market, the product's specifications, the potential customers, the expected revenue, etc. Once a green light is given, the leadership team identifies the respective owners for each module - business owner, product manager, S/W manager, S/W developer, S/W test lead, Quality manager and the Applications/Customer manager.

Step-2: Software Product Specification

Once the high-level marketing requirements are in, the associated product manager chimes in and creates a comprehensive S/W product specification. It generally includes an executive summary, product description/classification, product distribution, customer-specific requirements and product safety requirements(applicable to Automotive and related industries). The product manager creates block diagrams explaining the high-level picture keeping in mind the diverse audience.

Once the draft is ready, the product manager circulates it among various stakeholders for review and a review meeting is held which discusses the possible timelines in which the execution will be done, the external dependencies and the product availability to the customer.

Step-3: Software Architecture, Design & Test Plan

Once the R&D team is aligned with the business/marketing team on the features and the specifications, they go ahead and create the S/W architecture and design documents. They take inputs from the product manager and create these documents so that each marketing feature is well mapped to a requirement. And each requirement will map to respective Architecture and design specifications. In this way, the development execution can be made seamless.

"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies."

The S/W development manager will be on top of all the mappings and will assign requirements to the S/W developer/s (based on the resources available during the bandwidth). In the meantime, the S/W test lead is made aware of the specifications, and he/she has to come up with a test plan and a test strategy that can test the features. Any waivers that are required for the product execution is taken now.

A risk management plan is also created at this phase by the product manager and the S/W development manager which documents all the dependencies, resources/bandwidths available and the possible slip in the timelines if things don't go the way they are expected to. (Very valid as generally, things don't go as expected). Basically, it is a document that tends to suppress all possible surprises, but I have experienced surprises every single day. :)

Step-4: Feature & Test case development - Unit tests

Once the Software Product Specification, Requirements, Architecture and Design documents are well-reviewed, it marks the start of the development phase. Phew! We are finally here. Something relatable, eh.

The development is completely owned by the R&D team taking inputs from the Apps and the program management intermittently. The feature development is generally in sync with the test case development. The test case development, however, is done keeping in mind that the feature is a black box and has no insights into the feature development what-so-ever. The entire input to the test plan is the Software product specification. This helps in keeping the testing unbiased. As the development of the feature is partially done, most developers write unit test cases to check the intended functionality. Generally, unit tests are highly modular and are tested at the function level. This gives enough confidence to the developers that the function's response is always deterministic. The below statement is my favorite form of debugging.

"The most effective debugging tool is still careful thought, coupled with judiciously placed print statements."

The development is closely monitored by the S/W development manager who provides updates on the execution progress (weekly/monthly) to the program manager.

Step-5: Some more testing - Integration Testing & Static/Dynamic Analysis

As the feature and the test case development is almost completed, the developers hand over the code in a testable fashion to the test lead. As the testing team performs rigorous iterations of tests over the code, the developers aren't kept quiet :) The developers are encouraged to perform integration tests that happen at the requirement level. They test the feature/requirement by integrating various unit tests together. Once, they have fair confidence that things will work as they are, things aren't complete yet. The test lead will report some in-deterministic behavior which obviously surprises/frustrates the developer. Generally, the developer and the tester will agree to disagree. You can't really blame the tester as they are doing the job they were asked to: Find bugs.

Another important aspect of testing is performing static and dynamic analysis. This form of testing is under the scope of the developer. They basically check the code for proper coding guidelines, how much statements and functions are covered by the unit tests performed previously, etc. From my personal experience, I have fixed a lot of undiscovered bugs by performing this analysis. This ensures sanity and makes the code even more deterministic and maintainable. The code is generally frozen once the developer based reports like Unit tests, integration tests, static analysis are in place. In the meantime, the test report is generated by the test lead.

Step-6: Getting things together

The code is frozen. Check. PASS all the test cases. Check. You think we can hand off everything and move on right? Hang in there, buddy.

Documentation. I cannot stress how important this one is.

"Documentation is a love letter that you write to your future self."

The product is right now comprehensible only by two sets of people - the developers and the testers. The applications/customer engineers have absolutely no idea on how to effectively use your feature. The developers need to write crisp documentation on how to use the feature. Not too long - boring. Not too short - they will definitely come back to you on more queries. Documentation is generally underestimated in terms of bandwidth. It does consume a good chunk of your time explaining how to use the feature.

The S/W development manager hand-offs the code, the application team provides their inputs on how the product is, the test lead provides the reports and the quality manager validates the documents and scores based on how disciplined was the process followed. The program manager pulls all the things together and calls for a final hand-off meeting.

Coming back to the question asked at the beginning of the article. Why it takes nearly a month to ship out a few lines of code?

Assuming we are targeting a maintenance release, we only perform the steps of developing, testing and documentation (Steps 4-6). What might seem a 2-day effort of making the code changes for an S/W developer is actually several weeks to nearly a month's efforts considering the above steps. Let me illustrate it through a diagram.

The stick-human like figure was the one who asked the question. I hope he is clear now.