Mostly because the rest of 5e is built around an assumption of relative balance.

Adventures in modern D&D tend to consist of a series of more-or-less balanced encounters, usually combat, that will tax but usually not kill the player characters. If you tune it to be too easy, that makes for a boring session, or one where the DM runs out of content because the set piece encounter didn’t last as long as it should have. If it’s too difficult, you might have PCs die in a way that doesn’t match expectations. If most of the time combat encounters are supposed to be balanced, and a player has invested in their character’s backstory, and there’s clearly an arc they’re supposed to follow to the end, it sucks to have them be eaten by feral dogs.

“The DM can fix it” is always true, but a cop-out. If players avoid a set-piece encounter in 5e, it feels like they’re avoiding the whole dang adventure. And while XP doesn’t have to come from combat, that’s the bulk of it, and the most clearly supported by the rules.

And other systems just don’t have the same problem. Narrative games, like Blades in the Dark, have characters face consequences but not die unless it would be narratively satisfying. Other games just aren’t built on the assumption of balanced encounters, so it doesn’t throw a wrench into things if players get an unfair advantage, or bypass an encounter altogether, or just plain run away. And something like PF2e, which is in the modern D&D model, does have a functional balancing system.

A functional balancing system also doesn’t really have the problem of constant, perfect balance. D&D’s CR system will let you design encounters that are Easy, Medium, Hard, or Deadly, and PF2e’s Threat levels include Trivial, Low, Moderate, Severe, and Extreme. It’s just that one works better than the other.

Obviously all of this is “fixable” by the DM, but still, that puts a lot of work on the DM just to make the game work as intended.

Balanced encounters doesn’t mean every encounter is just as difficult, it means the GM knows how difficult the encounter is going to be. Any system with good encounter building has recommendations for the level of challenge.

I think it’s more an issue of being able to know how difficult the encounter actually will be. That way it’s a conscious choice to throw an encounter at your party that will likely kill them.

In my experience, having stuck with CR religiously, you find yourself with tense, anyone-could-die sessions - against a pack of wolves from a random encounter. And then you have easy, bring it home encounters - against the arc’s big bad. It takes control of the fight’s narrative stakes out of the DM’s hands, and makes it more-or-less random.

I want to stress that it was fun to play this way, but eventually myself and my players longed for more dramatic final encounters, and so I had to homebrew creatures.

I can see where you’re coming from, but consider this: When it comes to games that have an emphasis on combat (and dnd certainly qualifies), it can be useful for GMs to think of their encounter XP budget as their narrative budget. In other words, the more dramatic a fight is, the more difficult it ought to be. Also, inside that encounter, the monsters that eat up most of your encounter’s XP budget deserve the most narrative spotlight.

Imagine a GM has been running for a group for some time. Now, a climactic fight is upon them. By all rights, this should be a narratively momentous occasion for the table. Only it isn’t. The system’s encounter design fails and the whole fight gets completely trivialized. But such is dnd and every once in a while, your planned encounters just end up falling on their faces.

And we accept this. Because that’s just how it’s always been, right?

Only imagine this happening in another system. What if the gang was playing a forged in the dark system instead of dnd. The situation in the fiction is dramatic and challenging. The GM rightfully calls for many risky and desperate rolls and judiciously applies standard and limited effect. The party should be in for a world of hurt. Except now their dice pools all of a sudden are tripled because the system somehow breaks. In fiction-first games like FitD, this would be unforgivable. The table would be correct to just ignore the core resolution mechanics here because they fail to represent the fiction at play.

But that’s exactly what happens with dnd on a regular basis. And it doesn’t have to be that way. Encounter balancing in pf2e and dnd 4e (with asterisks) works rather well.

But why should we care to have balance? Oh, tons of reasons. Just the example I’ve raised here is that balanced combat allows GMs to set out challenges for the party that match the fiction well. This is notably different from having every encounter be equally challenging. Just that they roughly match what the GM had envisioned. And there are lots and lots more reasons.

This isn’t to say that balanced combat is superior to swingy, unpredictable combat. Both can be lots of fun. The key is to understand the type of game the table wants to play and lean towards that direction.

Not every dnd does it wrong. I’m pretty sure 4th Ed had it right. Pathfinder and 13th age are also kinda just editions of dnd, and they both have very tight encounter math!

Use a balanced system. Control the pacing of combat better as a GM that way. It also makes sure the difficulty fits the narrative.

The dice will still create the fun erratic moments.

Not a fan of the simply broken CR of dnd and tired of people defending it.