One way Merckx Index's idea is somewhat impractical: it's not always possible to construct a breakaway stage, a climbers' stage, etc etc. We're lucky this year that the Olympics were held in a city with the potential for an excellent course, but many don't have the topography for such a varied stage race. Take Paris, for instance, the likely 2024 host: where would you have your climbers' stage?
Olympic Games usually have venues outside the host city. For example, Manaus is about 2,800 km from Rio yet it was one of the 2016 Olympic venues. The Massif Central, by comparison, is just a few hundred kilometres from Paris.
Generally having venues outside far outside host city applies only to football and in the cases where host city is in inland, sailing. I think in Rio everything was within 50 km, with the sports which need more natural place to host (for example mountain biking, equestrian eventing and shotgun events in shooting)