After having worked in different organization with different levels of maturity, and different backgrounds I mature some opinion on why software is difficult.
But the answer is difficult and messy, and nobody like it.
People don't scale
One solid technical leader can lead around 10 productive developers, I have never seen a tech leader being effective with more than 10 developers to lead.
But they are routinely asked to lead much bigger groups.
Tech leaders, resort to not overseeing technical work, but to provide guidance.
Guidance scales better, but it is impossible to get technical product out of thin guidance.
Eventually, there will be choices that no amount of guidance will make simpler.
Moreover, guidance does not work well with computers. Computers don't need guidance, they want instructions. Ideally correct instructions.
Hermetic API are difficult
We recognize that people and groups don't scale - we stretch them anyway looking for efficiencies - but the main tools to manage big tech organizations are API.
If teams communicate only through API, the problems become tractable. Not simple, tractable.
However, hermetic API are almost impossible to design and implementation details are leaked and relied upon.
We made progress in putting non-functional requirements into SLA, but it is not enough.
Moreover, hermetic API are difficult to change, in a growing organization this makes little sense.
Software is a focus intensive industry
The industry like to churn over engineers.
It makes sense from a financial and power balance perspective. No one is important enough to require an unproportional pay rise or benefits.
It makes no sense from a technical perspective.
Software systems are encoded in source code first, in the brain of the engineers second and in documentations lastly.
Reading code is hard, and the more complex and important is the software, the hardest it is to understand its codebase.
Churning engineers means losing the capacity to modify the software without huge investments in onboarding new developers.
Documentation is hardly useful, especially when modifying the software and not when using it.
A vision and bias to action is hard to find and develop in big orgs
People with bias for actions and with a vision are hard to find. Even harder to find in big orgs.
Operator that pick tasks from a queue and complete them are what big orgs optimize for, but not what is needed while doing innovation.
If operators pick tasks from a queue, who put those tasks in the queue? Is there anybody working to make the group better?
Who is working ON the group instead of working IN the group?
Lack of tooling
We don't know how to manage the complexity of modern software.
Tools for understanding the complexity behind software are not existing.
We are still unable to come up with an accepted index for complexity. Cyclomatic complexity is one of those index, nobody uses it. We definitely need something better.
We don't have it.
The way software is developed is not qualitatively different from what was 40 years ago.
But the complexity of our software definitely is.
The curse of dimensionality, it applies to software as well
In data analysis, the volume to explore growth exponentially with the dimension of the data.
After a few dimensions, most problems become intractable.
The same applies to software.
At the beginning, features and data structure are neatly designed to be orthogonal of each other. This does not last long.
New requirements force to intertwine features and their implementation. Making it harder to design, test and implement anything new.
Testing is broken
Testing is the only way we realistically use today in the industry to have some guarantee that the last code change did not break anything important.
Combined with the curse of dimensionality mentioned above, we end up with huge matrixes of tests that are hard to run locally, slow and expensive.
None of the problem highlighted here comes from inexperienced developers joining the industry or from management that don't understand software, unable to provide the necessary support.
But those are real problem that make developing software much harder.
Solution to try
I don't think that everything is lost and that this is the best we can do.
Tools could be improved on a qualitatively level. We don't need a new way to run tests on a cloud environment. We need tools to tame software complexity.
Companies could give to engineers more power, higher pay, more interest in the company and reduce churn.
Companies could incentivize working ON the group instead of working IN the groups.
Software could be developed to optimize focus and not throughput. Smaller groups of people with few parallel working stream, mostly working on the same screen, mostly being in the same meeting. Working as a team, not as a collection of individuals.