Why Do Most Projects Fail?

At its heart, every project is about change.  It can be a small, simple change or a large change that will fundamentally change the way the organization functions.   The bigger the change, the bigger the risk of failure.

Why, do you ask?  Because change is inherently scary to most people.  What is known is safe. It may not be pleasant or it might even be downright awful, but it's still known.  You know how big that tiger is and how likely he is to want to eat you on any given day (or you think you do).  You don't know what lurks out in the unknown.  This trait probably kept humanity safe through most of its evolution.  After all, you don't have time to think about it and prepare logical arguments when the tiger (or even scarier unknown monster is about to eat you).  All you have time to do is decide to fight or run.   For most of history, change was not good (famine, plague, war, tigers, etc.), so this behavior is completely natural.

The fact that most projects don't involve monsters about to eat people doesn't matter.  The level at which humans react doesn't understand the difference between a real tiger/monster or a potential job change or loss.  To our subconscious it's all the same thing - survival.

What isn't logical is that most companies ignore the affect of change on people.  They think a better methodology will solve the problem.  They try to make projects better by imposing more process, more accountability and less humanity as a byproduct.  If you look at Communications Planning and Human Resources Planning in the PMBOK (Project Management Body of Knowledge), it talks about plans, resource management, etc. etc.   The PMBOK does mention the importance of good communications and resource management, but that's about all.  The rest of both knowledge areas are about planning and controlling.  Nothing about people, nor about the fact that you can't treat people like you do a machine.  Nothing about actually communicating, just how to plan it and document it.

None of which will keep a project from failing.  I have never seen a project saved because of a great communication plan.   A great communication plan will help you identify who you need to talk to, what information they need and how often.  What it won't tell you is how to talk to them - how to recognize and address their fears and concerns about the impending, how to get them excited and how to keep them motivated and involved.  That's what makes a project successful.  That's what communication needs to happen on a project.  And it isn't part of any methodology.
Agile Project Management at least recognizes this need and how important it is, but it still doesn't tell you how.  The PMBOK just recognizes communications as important - that's all. Then it proceeds to ignore how to communicate and manage people as people and proceeds to plan them the same way you would raw materials.

People react emotionally to change, not logically, and the emotion is usually fear.  Using logic to talk to someone's fears doesn't work.  You have to talk on the level of emotions, not facts.  Until you've addressed the emotions, the facts don't matter and won't be heard.  That's why most projects fail - they don't address people as people, who are emotional first and rational later (remember the tiger? ) and neither do the methodologies.
The key to good project management is being a very good listener and communicator, on both an emotional and rational level.  Especially on an emotional level - that's where change happens,