I think the problem is the tour itself, it has got so much attention and anyone who gets yellow or win a stage is a hero in his country. Also, in history there were always road captains/bosses in the peloton, who calmed things down when it got wild, but now, there is less respect from young riders (not always a bad thing). So many riders taking unnecessary risk to win a stage/get yellow. It would definitely help to include prologue every year, so the yellow jersey would be out of reach for most after stage 1.
There's also a few guys who can go for GC that have good form in recent classics (Pog, Roglic and Ala) so the other GC teams who are not so used to little northern roads are pretty stressed because they know there's potential to lose a heap of time on such stages.