first question first - the answer is, there can be a lot of reasons:
-different companies make hiring decisions differently. at some places, one person makes the call alone. at others, it is a consensus decision.
-similiarly, different firms have different policies on who exactly may extend an employment offer, and when, and how.
-the assistant boss may have thought you were great but got vetoed by the big boss.
-your references may not have checked out to the satisfaction of the hiring manager(s)
-any number of other things could have happened "behind the scenes" - hiring freeze, cost reduction efforts, a need to hire more diverse employees to meet affirmative action quotas, an internal transfer candidate surfaced, etc.
now for the "longevity" question - my personal preference is to hire the person with the personality and aptitude I need and to teach them the "technical" aspects of the job. however, not everyone operates the same way (see the first point, above.) some managers emphasize education - some require a certain level of experience - still others look at how much time a person spends with a company to be sure they aren't hiring a "job hopper."
when you interview, you are selling yourself, and part of being a good salesperson is to determine what the other party's needs and requirements are so you can demonstrate how you will meet those needs.
good luck.