I'm sure there's a way Mojang could update while keeping as much of the code as consistent as possible to minimize breaking mods.
There is. Some majors softwares like CAD programs, image editors, 3d modelers have an API or in-built IDE that allows you to create addons. The internal functions can change but there are permanent "hooks" you can "connect" your addons to. That's what Forge tries to do, to provide a library of functions that handle the game core.
The thing is, Minecraft started as a one-man work, I believe its architecture wasn't conceived to be expanded or built upon. If you want to create an addon, you need to write a piece of software that override the core functions. That's not like "oh there's a function by which you can access all public variables that handle the character animation and modify it at your wish", no! it's hardcoded, so you need to override existing functions. Minecraft has been rewritten lately to make more features moddable and accessible. While it's not done, mods will be broken.