The beanpoles are taking over more because of developments in parcours design and team tactics rather than developments in rider physiology. Vicente Trueba was probably the first of the historical line of featherweight climbers, standing barely 1m50 and 55kg, but racing in the 1930s the surfaces of roads and the less controlled were such that he couldn't compete on the flat. There have been lots of flyweights over the years, but only the truly great ones of them - Charly Gaul, Federico Bahamontes, Lucien van Impe, José Manuel Fuente, Lucho Herrera et al - became the kind of legendary champions whose exploits will never be forgotten. When Lucho Herrera was racing against Hinault, there was often over 150km, close to 200km of time trialling, either individual or team, which benefited a more all-round rider who could put out the TT power, requiring a more powerful build than the frail, pint-sized grimpeurs. Also, especially at the Giro, adverse weather conditions would often mean that those climbers were more susceptible to illness and, when crashes ensued on the poorer quality roads than today, more likely to be injured (this tendency towards the smaller, bony riders to get injured more often continues to this day with people like Igor Antón). Only the truly legendary climbers could put together the kind of palmarès to go down in history forever, and that's why those guys named above are the ones we remember.
We probably have the 90s and early 2000s to thank for the current crop of scrawny climbers at the top as well as dieting (and yes, doping) fads that have meant the tendency now is for power riders to diet down to compete in the mountains rather than for climbers to bulk up for the time trials. The ability of a dominant time trialist to exercise total control of Grand Tour racing as Indurain did, with the gaps he could build up far in excess of those he would lose in the climbs, led to a reduction in distance raced contre-le-montre, and the simultaneous development of a controlling template for the stage racers (developed by Banesto and perfected by US Postal) and the sprint train technique (perfected by Saeco) meant we were looking at more riders being protected in the pack for longer, and the pack holding together for longer, producing fewer time gaps created in the flat stages and the emergence of more 'pure' sprinters in the modern sense making it harder for late attacks to stay away, meant that being strong enough to cope with the jostling in the bunch in tough rouleur stages became less important for stage racers.