Why must stakeholder management always be so complex on large technology projects?

NO. It can be very simple and straightforward.

All large cross-functional project have a plethora of stakeholders each with their own agenda.

The typical agile methodology to address decision making is a responsibility assignment matrix (RACI or RAPID). My suggestion - don’t make a RACI or RAPID.

Try this instead - think of your stakeholders in these categories - DOERS, DECIDERS, INFORM.

DOERS - People responsible for a deliverable, tasks, or output as part of the project. Output can be UX design, architectural solution, a line of code, marketing copy, project schedule, PRD, any thing materially necessary to the complete the Project.
DECIDERS - People who are accountable for the project completing on time, to spec, with high-quality. Often times, this category of stakeholders tends to overlap with the DOERS. More on this later.
INFORM - This is everyone else. The engineering manager who wants to attend the meeting just to stay in the know; the PM who wants to be kept looped in on status reports; the VP whose team is impacted by the work you are doing but doesn’t necessarily have a stake or say. This is the hardest category to maintain and manage given that many people within this category believe they deserve a place on the DECIDERS table.

DECIDERS fall into 2 categories:

DAY DECIDERS - These are leaders directly accountable for delivering the various milestones of a project. They also tend to be Doers as well - Product Managers, Program Managers, Engineering Leads, Product Marketing Leads. They make the day to day decisions and keep the training running.
STRATEGIC DECIDERS - These are often Senior Leaders accountable for the project as it pertains to the overall strategic fit with company wide vision and goals. Strategic decisions like push the program out due to a delay, or needing more resources to prevent slippage, or review difficult decisions that cannot be made by daily deciders due to the potential trade-offs and impact, that is their job.

Some pitfalls to watch for with this simplified approach

Be careful of who you make the DECIDERS.
Senior Leadership must empower the DAY DECIDERS.
Keep the INFORM stakeholders looped in on a regular basis.

A responsibility assignment matrix is a powerful tool. Yet, it’s, RACI or RAPID or RASIC or altRACI, numerous incarnations within software engineering create unnecessary overhead work. In my opinion, its a great tool to analyze the complex nature of an organization and it’s poorly designed matrix structure. It’s feel-good-stuff work that all Project Managers must avoid.