Everyplace I have worked over the years had a rhythm. The pace of development, releases and problems feels unique. My shower thought of the morning enlightened my thinking about the cadence of work for a big company versus a small company.
You’ve read it all over the web, all the cool kids are doing startups. Everything is continuous delivery and late night coding behind a wall of pizza boxes and diet code empties. It sounds like a small startup is hectic and busy. Truth be told, that piece of conventional wisdom is backwards.
Small startup has an easier, slower cadence than a big company.
How do I know? Well I have done both from 3 guys in a garage to over 37,000 people at my current employer. Today’s epiphany showed my the why behind this unexpected truth.
Big feels fast
In a big company, everyone complains about the slowness of decisions, and nothing changing. It has to be slow to work in the same place as Wally from Dilbert. Not so at all.
My current employer is an American semiconductor company that makes memory chips. (You can guess who it is if you know the industry.) Everyone is busy. It was always a mystery to me why. Some people seem to be busy because they are showing off for the managers, or like the drama. But, let’s dig into the reasons for the hurry.
First, pretend our product is semiconductors. They are built in a fab. The modern fab costs billions to build, has to be retooled every couple of years to keep up with Moore’s Law, and never sleeps. There is no way to turn off and reheat ovens, chemical baths and all the machines used to build chips. Even if you could shut it all down for the holidays, why would anyone have a billion dollar fab not making money (chips) 24/7.
The weird thing about a fab is, the throughput is huge, but the latency is long. If you put a lot of wafers in one end, a whole lot of finished parts come out the other end, numbers like tens of millions in a month. The parts coming out this week were clean silicon wafers 3 or four months ago. The cycle times for processor chips are even longer. Do you see the problem.
This means when a change is made, or required, to actually see that change from a customer point of view it takes months. We can change things at different steps in the process, but still, even a simple fix will take months to reach the customer.
Slow, right? Maybe from the outside, but you as the person making those changes are working fast as you can. Make sense?
No? We can dig deeper.
Assume there is a minor problem that can be fixed by a hardware change. We will modify the metal layer to fix some timing issue or something. Not huge, just a bug fix.
First, we had to find and isolate the problem. So get your group, the customer test group, and people working on the test fixtures all digging in independently. If the issue is a big one, something blocking a customer, more eyes are better.
Once you know what is wrong, if it is possible, the fix has to be prototyped. An existing part needs to be modified to be sure we fixed the right thing. Then other people make the actual changes to the part. Those changes involve the design engineers, simulations, and the layout group. You did not think one lone ranger could would do all that stuff?
The fix is in, right. Well… a few more steps. The design is made into reticles, the photo masks used to generate features on the chip. Those take time and are done by a specialized team. The reticles go to the fab, on the far side of the world, of course.
The fab then makes a split batch. Half the original way, and half with your changes. Everything is the same, just the latest set of changes. All this costs a lot, so it is rarely one change but a group going in together. When done, there are about 10,000 parts new and old.
That change, the simple fix, does it change any of the functional test steps that come next in the manufacturing process? The way the bare wafers are tested, the package chips, or high speed tests? If it does, those teams have to modify test programs, commit the changes, and release the changes to production.
After weeks of fab time for a metal change, the new parts are tested as bare wafers, then packaged, tested again for functionality and infant stress, then tested at speed. Now you get one to verify and validate the change fixed the problem.
OK, now that the fix is complete, it can be released. New parts will start to flow, but like turning the hotel shower from cold to hot, it takes some time to flush out the line. The fix won’t hit in volume for X weeks again.
I can hear you thinking “that took forever. What do you mean work faster?”
Think about the time it took and the number of people involved. Directly the number is close to 50, if you count the fab people, test technicians and all the others, the number is in the hundreds. An extra day or two delays all those people. Add on to that the fact your small bug could exist in other parts. Yep, there is a system and a procedure to inform them and check if other designs have the same issue.
All of this has to be communicated. The reasons why something failed go up the chain. The details go out to the test groups affected and the design groups. Schedules, inventory, and capacity have to be managed. Other groups are informed, data is collected, new tests added. So, we have meetings. Lots and lots of meetings. And Power Point. Lots and lots of Power Point.
The longer it takes to get the fix implemented, means the more parts built without the fix. Remember, the fab does not shut down, not today, not ever.
Every day your small part of the project needs to be tracked in case something is breaking. Problems found have to be attacked and fix correctly. All of this has to be checked and watched for quality. You have to work fast.
Small feels slow
Contrast all that with 3 guys in a garage office by the highway.
“Hey, Jim, I found that bug in the FPGA, wanna try the new code?”
Jim, from the next desk away says, “Sure”
That afternoon the auto-update and new code goes up on the web site for the customers.
We work without a net. There is no test or quality group to catch a mistake. Slow, sure and methodical is the only way to survive.
If your head is fuzzy from looking at the problem too much, go home, get some sleep. It will wait.
The cadence is slow. The response is fast.
“Go slow, slow is smooth, and smooth is fast” – Navy Seal Wisdom from Jocko Willink
Some people prefer the big company that does big things. With all those people and momentum, doing something new is like turning a huge oil tanker. Others prefer small and nimble. A startup can pivot to a new idea over lunch. Think about cadence and how you want to work.
The lesson is “do every thing you can to reduce cycle times”. Use automated tests, unit tests, and release often. Make the product automatically update with new code. Get the time for a fix to any part of the product approach zero from fix to the field. When the cycle time is fast, you can slow down, maybe even relax a little.