One of my pet peeves with the usage of complex(ity) out of the traditional time/space in computer science is that most of the time the OPs of several articles over the internet do not make the distinction between boundaried/arbitrary complexity, where most of the time the person has most of the control of what is being implemented, and domain/accidental/environmental complexity, which is wide open and carries a lot of intrinsic and most of the time unsolvable constraints.
Yes, they are Google; yes, they have a great pool of talent around; yes, they do a lot of hard stuff; but most of the time when I read those articles, I miss those kinds of distinctions.
Not lowballing the guys at Google, they do amazing stuff, but in some domains of domain/accidental/environmental complexity (e.g. sea logistics, manufacturing, industry, etc.) where most of the time you do not have the data, I believe that they are way more complex/harder than most of the problems that the ones that they deal with.
I’d wager 90% time spent at Google is fighting incidental organizational complexity, which is virtually unlimited.
The phrase thrown around was “collaboration headwind”, the idea was if project success depends on 1 person with a 95% chance of success, project success also had a 95% chance. But if 10 people each need to succeed at a 95% chance, suddenly the project success likelihood becomes 60%…
In reality, lazy domain owners layered on processes, meetings, documents, and multiple approvals until it took 6 months to change the text on a button, ugh
Another side of this coin is that the expected payoff from a project depends on how many unrelated projects your organization is engaging in, which is deeply counterintuitive to most people.
Every project carries with it three possibilities: that of success, where the company makes money, that of failure, where the company does not, and that of a "critical failure", where the project goes so wrong that it results in a major lawsuit, regulatory fine or PR disaster that costs the company more than the project was ever expected to make.
If you're a startup, the worst that can happen to your company is the value going to 0. From an investor's perspective, there's not much of a difference between burning all the money ($10m) and not finding product-market-fit (normal failure), or your company getting sued for $3b and going bankrupt (critical failure). The result is the same, the investment is lost. For a large corporation, a $3b lawsuit is far more costly than sinking $10m into a failed project.
You can trade off these three possibilities against each other. Maybe forcing each release through an arduous checklist of legal review or internationalization and accessibility testing decreases success rates by 10%, but moves the "critical failure rate" from 1% to 0.5%. From a startup's perspective, this is a bad tradeoff, but if you're a barely-profitable R&D project at big co, the checklist is the right call to make.
This problem is independent from all the other causes to which bureaucracy is usually attributed, like the number of layers of management, internal culture, or "organizational scar tissue." Just from a legal and brand safety perspective, the bigger your org, the more bureaucracy makes sense, no matter how efficient you can get your org to be.
"Nothing breeds conservatism like having something to conserve" was a recurring line from a founder-type, complaining individuals had become less dramatically innovative as the "startup" succeeded and prospered. Though they had colleagues who thought the line misapplied. IIRC, fuzzily.
This is really insightful, I hadn’t considered this dynamic before.
I wonder if a related intensifier is that as a company grows larger it tends to follow the letter of the law rather than the spirit of the law, which results in less buffer space between the behavior and the law (and hence higher lawsuit risk).
I like the ideas here but I think the actual chance of getting sued for $3b is so small as to be negligible in the context of costs. It's also questionable how much the additional process/overhead moves the needle on that chance. Larger companies also have various "shields" against these sorts of lawsuits. E.g. they lobby politicians, they employ lawyers, they have legal and IP protection.
Just like anything else in life you want to look at the present value and then get insurance for huge risks.
That said agree that a startup can take more risk but I don't think that is the major factor explaining why larger companies tend to be process heavy and slower.
This explanation wasn't just about lawsuits.
Other things in a similar category are:
- negative media attention (media scrutiny increases proportionally to organization size)
- doing something that upsets an influential group and may have consequences for the rest of your business (think how big the outrage would have been if Apple, Google or Microsoft tried making an "Uber" app before Uber existed)
- bringing down the service which is being worked on, potentially breaking SLAs
- failing to meet customer / legal commitments, particularly in regards to internationalization, accessibility etc.
- security incidents, which are presumably a bigger deal, as your project is connected to the rest of your infrastructure
- getting cancelled online, which causes employees (unrelated to the project) to quit
- natural, random and serious consequences that result from the fact that your project needs the company to hire additional employees. E.g. there's a certain number of people willing to commit sexual assault or financial fraud in the population, and the more people you hire, the more likely it is that you get one of them.
I work for "big tech" and have worked for others. Other than some CYA training there really isn't a lot of cross talk between engineering productivity and this sort of risk aversion.
Process, complexity, inefficiency is just something that happens for big companies and big software. Things slow down and then there's a negative feedback loop and things just go down hill. Innovators dilemma sort of stuff.
Getting sued for $3B maybe, but what about getting fined $3B? Such as by the EU.
> lazy domain owners
Interesting. As a consultant for the most of the last 25 years, my experience is the domain owners are typically invested and have strong opinions on the things that impact their jobs.
Executive leadership, on the other hand, doesn't want to actually know the issues and eyes glaze over as they look at their watches because they have a tee time.
There's a culture of "I won't approve this unless it does something for me" at Google. So now changing the text on a button comes with 2 minor refactors, 10 obvious-but-ignored bugfixes, and 5 experiments that it is actually better.
While this sounds pretty frustrating, there is at least a small upside: at least you get to the obvious-but-ignored bugfixes.
Most smaller places don’t have the bandwidth and many larger ones don’t have the desire.
I’m not sure if that makes up for bugs potentially introduced in the refactors, though.
Well, when the owner asks for a whole test suite that didn't exist to get a fix in, what most likely happens is that you just wasted your time in a draft CL that will get lost.
Do you mean the relevant code area(s) didn't have (sufficient) tests? You're being asked to backfill those missing tests in addition to your fix?
Yes. I've experienced pushback from obvious fixes with requests to formally test their code for the first time.
All because it may break someone. Even when I presented a real defect based on docs/comments and fixed it. You'd think that if they truly cared about breakages they'd already have some tests for it from where I can easily start.
I don’t think that is necessarily unreasonable. The team may have the same constraints on themselves in that they wouldn’t touch the code either until tests are written.
> All because it may break someone. Even when I presented a real defect based on docs/comments and fixed it.
It's great that you found a bug and fixed it.
The problem is, how do you know that there are no other regressions?
Code is a liability. Once you check it in, the team that owns it is responsible for it. Untested code is a liability of unknown scope. It's quite understandable why they don't want to accept someone's contributions, when the contributor isn't the one who will ultimately be dealing with any of the consequences. If you think they are being mean and lazy, imagine if the tables were reversed.
I don't accept puppies or elephants as gifts for similar reasons.
It's unfortunate that existing test coverage sucks. In this case, the best way forward should be for the team in question to improve coverage, and for you to then submit your fix + a test for it. And if they don't have budget to do this, then that sucks, but that's their call to make, and that's a signal that the project in question is abandonware.
And it's fine for a large company to have a bunch of abandonware. If it works, and produces value, the optimal amount of ongoing development effort to invest into a piece of software may, depending on the circumstances, be near-zero.
They aren't asking for you to write tests because 'it benefits them', they are asking you to write tests because as a professional engineer, you should write tests, and not just yolo it.
Look, sometimes you may have good reasons for why a test is impractical. You are allowed to push back, or look for a different reviewer. There's a hundred thousand people in the firm, you should be able to find one or two that will let you submit literally anything that compiles.
But most of the time, the reviewer is giving you advice that you should take.
If you are turning a button to a slightly different shade of blue and it's not a button you own, the owner of the button should not be asking you to write tests for the button.
It's as good an opportunity as any to improve things.
They're acting as selfish demanding you do something for them, as you are for refusing.
If the case is as simple as you describe, surely there's more than one owner of the code that can approve this, if one guy is being unreasonable.
If there is actually only one owner that can approve changes to the package, there's something really weird and unusual about that project setup, or it's someone's internal hobby project that they wrote five years ago and semi-maintain, in which case, I have to wonder why you submitting one-liner changes to it is all that important.
We're all adults, we all work together, we can all work this out. If someone absolutely insists on being an asshole, escalate. It's why you have a manager, and why they have a manager.
My experience is that very few people are unreasonable assholes.
There's always plenty of organizational, vision, strategy, and execution problem in any billion-dollar company, but 'people are unreasonable in code reviews' is not one I'd put in the top 10. It might be something that ruins your day once or twice, but that doesn't make it systemic.
> If someone absolutely insists on being an asshole, escalate.
That's doubling down on time spent on contributing back. It's usually cheaper to workaround the issue once you notice it'll be way harder than it should be (not hard at all).
I would think the person is more interesting and more relevant than the button. One doesn't create hurdles when I'm trying to work. You just don't do it.
I've allowed people to build a whole obstacle course one time. Decades later it stil has me randomly burst out in laughter. It's like hoarding technical debt until nothing in the code base makes sense or even looks familiar. You just don't do that...
Coordination Headwind: https://komoroske.com/slime-mold/
The old "If you want to go fast, go alone. If you want to go far, go together."
Also why the optimal business strategy seems to be to go as far as you can alone and then bring on other people when you're running out of steam.
Well, good management/tech leadership is about making sure that the risks coming from individual failure points (10 people in your example) are recognized and mitigated, and that the individuals involved can flag risks and conflicts early enough so that the overall project success probability does not go down as you describe...
The assumptions in that math are wrong anyway. Once you depend on 10 people, the chance that they each achieve "95% successful execution" is 0.
This is only partially down to the impossibility of having every staff member on a project be A++ players.
There is coordination RISK not just coordination overhead. Think planning a 2 week trip with your spouse with multiple planes/trains/hotels, museum/exhibit ticket bookings, meal reservations, etc. Inevitably something gets misunderstood/miscommunicated between the two of you and therefore mis-implemented.
Now add more communication nodes to the graph and watch the error rate explode.
That's what the math is reflecting. Project succeeds if all of 10 people does their job well. Each person has a 95% chance of succeeding. 0.95^10 ~= 60%, and so the chance that all 10 people do their job successfully is ~60%.
Those jobs also include things like management and product design, and so the coordination risk is reflected in the 5% chance that the manager drops the ball on communication. (As a manager, I suspect that chance is significantly more than 5% and that's why overall success rates are even lower.)
That's what I mean "only 5%" encapsulating all failure modes (comms/implementation/coordination/etc) is very low.
And that under-estimation compounds to make the top level 60% much higher than it should be.
A 7.5% rate takes top-level success odds below 50% - 46%. A not unrealistic 10% takes the top level down to 35%.
Etc.
And when you’re at a smaller company 90% of your time is fighting societal complexity, limit of which also approaches infinity, but at a steeper angle.
No greater Scott’s man can tell you that the reality is surprisingly complex, and sometimes you have resources to organize and fight them, and sometimes you use those resources wiser than the other group of people, and can share the lessons. Sometimes, you just have no idea if your lesson is even useful. Let’s judge the story on its merits and learn what we can from it.
Look, I've never had to design, build or maintain systems at the scale of a FAANG, but that doesn't mean I haven't been involved in pretty complicated systems (e.g., 5000 different pricing and subsidy rules for 5000 different corporate clients with individually negotiated hardware subsidies (changing all the time) and service plans, commission structure, plus logistics, which involves not only shipping but shipping to specific departments for configuration before the device goes to the employee, etc.
Arbitrarily, 95% of the time the issues were people problems, not technical ones.
You are right but it misses the flavor of the problem. I was a consultant in infosec to F500s for many years. Often solving a problem involves simply knowing the right person that has already thought about it or toiled on the problem or a similar one. But when there are 100,000 engineers it becomes an order of magnitude (or two!) more difficult and that puts forth unique challenges. You can still call them “people problems” and they often may be. However if you try to solve them the same way you might solve it at a smaller engineering org you will get and be nowhere and be poorer for the time you spent trying it. Ask me how I know lol. The technical problems are also like that. Almost everything has an analog or similar thing to what you are probably familiar with but it is scaled out, has a lot of unfamiliar edges and is often just different enough that you have to adjust your reasoning model. Things you can just do at even a typical f500 you can’t just do at big tech scale. Anyway, you are directionally correct and many of these wounds are self inflicted. But running a company like Google or Facebook is ridiculously hard and there are no easy answers, we just do our best.
Fair, but just in case, the system I used as an anecdote is operated for a company that has 45,000+ direct employees and $25 billion annual revenue.
I have a similar perspective. I think after a few years, it's the people things that have always been the hardest part of the job. That's probably why in the interviews, we always say things like: communication is key, culture fit, etc.
On the other hand, the good part of the job is solving complex technical problem with a team.
Equally important is the amount of time they save because of available abstractions to use like infra, tooling etc
I think I understand what you are saying, and I agree.
Google has all sorts of great infra projects that simplify complex domains. Those are solved problems now, so nobody notices it.
The existence of incidental complexity isn’t evidence that the counter factual is less complexity.
I think this is addressed with the complex vs complicated intro. Most problems with uncontrolled / uncontrollable variables will be approached with an incremental solution, e.g. you'll restrict those variables voluntarily or involuntarily and let issues being solved organically / manually, or automatisation will be plain and simple being abandoned.
This qualify as complicated. Delving in complicated problems is mostly driven by business opportunity, always has limited scaling, and tend to be discarded by big players.
I don't think this is adequately addressed by the "complicated vs. complex" framing—especially not when the distinction is made using reductive examples like taxes (structured, bureaucratic, highly formalized) versus climate change (broad, urgent, signaling-heavy).
That doesn’t feel right.
Let me bring a non-trivial, concrete example—something mundane: “ePOD,” which refers to Electronic Proof of Delivery.
ePOD, in terms of technical implementation, can be complex to design for all logistics companies out there like Flexport, Amazon, DHL, UPS, and so on.
The implementation itself—e.g., the box with a signature open-drawing field and a "confirm" button—can be as complex as they want from a pure technical perspective.
Now comes, for me at least, the complex part: in some logistics companies, the ePOD adoption rate is circa 46%. In other words, in 54% of all deliveries, you do not have a real-time (not before 36–48 hours) way to know and track whether the person received the goods or not. Unsurprisingly, most of those are still done on paper. And we have:
- Truck drivers are often independent contractors.
- Rural or low-tech regions lack infrastructure.
- Incentive structures don’t align.
- Digitization workflows involve physical paper handoffs, WhatsApp messages, or third-party scans.
So the real complexity isn't only "technical implementation of ePOD" but; "having the ePOD, how to maximize it's adoption/coverage with a lot uncertainty, fragmentation, and human unpredictability on the ground?".
That’s not just complicated, it’s complex 'cause we have: - Socio-technical constraints,
- Behavioral incentives,
- Operational logistics,
- Fragmented accountability,
- And incomplete or delayed data.
We went off the highly controlled scenario (arbitrarily bounded technical implementation) that could be considered complicated (if we want to be reductionist, as the OP has done), and now we’re navigating uncertainty and N amount of issues that can go wrong.
I was very centered on the software part of the problem. A complex problem can be solved with a complicated chain of small technical solution. At implementation level, its complicated, not complex, e.g. you mostly need knowledge of the general problem to understand the solution, and many added things make things complicated.
My take is that if your complex problem is only solvable by complex software (e.g. not a combination of simple small parts), and _cannot_ be reduced to simpler things, you are in the complex space.
Maybe it's too reductive, it's just my opinion, but it's a good way for me to determine predictability on ability to solve a problem with many unknown, at the engineering level. The dangerous blockers are in complex space, identifying them early is critical. Complicated stuff can be worked around and solved later.
I don't think it is, because the intro gets it wrong. If a problem's time or space complexity increases from O(n^2) to O(n^3) there's nothing necessarily novel about that, it's just... more.
Complicated on the other hand, involves the addition of one or more complicating factors beyond just "the problem is big". It's a qualitative thing, like maybe nobody has built adequate tools for the problem domain, or maybe you don't even know if the solution is possible until you've already invested quite a lot towards that solution. Or maybe you have to simultaneously put on this song and dance regarding story points and show continual progress even though you have not yet found a continuous path from where you are to your goal.
Climate change is both, doing your taxes is (typically) merely complex. As for complicated-but-not-complex, that's like realizing that you don't have your wallet after you've already ordered your food: qualitatively messy, quantitatively simple.
To put it differently, complicated is about the number of different domains you have to consider, complex is about--given some domain--how difficult the consideration in that domain are.
Perhaps the author's usage is common enough in certain audiences, but it's not consistent with how we discuss computational complexity. Which is a shame since they are talking about solving problems with computers.
If you consider their history of killing well loved products and foisting unwarranted products such as Google Plus onto customers, Google is for lack of a better word just plain stupid. Google is like a person with an IQ of 200 but would get run over by oncoming traffic because they have zero common sense.
I've not seen "accidental" complexity used to mean "domain" (or "environmental" or "inherent") complexity before. It usually means "the complexity you created for yourself and isn't fundamental to the problem you're solving"
Also, anything you do with enterprise (cloud) customers. People like to talk about scale a lot and data people tend to think about individual (distributed) systems that can go webscale. A single system with many users is still a single system. In enterprise you have two additional types of scale:
1) scale of application variety (10k different apps with different needs and history)
2) scale of human capability (ingenuity), this scale starts from sub-zero and can go pretty high (but not guaranteed)
Rich Hickey is famous for talking about easy vs. simple/complex and essential vs. incidental complexity.
“Simple Made Easy”: https://youtu.be/SxdOUGdseq4?si=H-1tyfL881NawCPA
Im a HW engineer and don't really understand "complexity" as far as this article describes it. I didn't read it in depth but it doesn't really give any good examples with specifics. Can someone give a detailed example of what the author is really talking about?