As others have said, height hasn’t really been going down, but weight and BMI have.
www.cyclistshub.com
Due to allometric scaling taller guys will generally be better at TT because they have a bigger power increase than frontal area increase. Aka body weight and muscle mass increase exponentially per linear increase in frontal area, but drag only increases linearly to frontal area (despite increasing exponentially to speed). So a 10% weight increase might mean 8% more power and ~6% more frontal area, a benefit for TT but not necessarily climbing. The weight of the bike is a bigger percentage of total weight being moved for really tiny guys though which probably makes a small difference when 2 climbers of different size are at a similar level.
TT rules have benefited shorter riders because of the way they have to set up their bikes, so that is actually a rule worth changing to even the playing field, which they did this year, aka tall people now reap the benefits that were artificially taken from them.
Remco looks freakishly aero, but small guys like him have to be in order to approach the same power to CdA of the top big guys. There will probably be a top big guy built with godlike proportions like Remco some day and you’d expect him to be even better than we’ve ever seen.
Then there’s all the stuff like lung capacity and cardiac output, heat dissipation, which aren’t necessarily only important in relative terms..

Tour, Giro, and Vuelta: Grand Tours Stats Compared (2023)
Check these unique cycling Grand Tours statistics. You find out how the distance, speed, rider's weight, height, BMI, and much more developed over time.

Due to allometric scaling taller guys will generally be better at TT because they have a bigger power increase than frontal area increase. Aka body weight and muscle mass increase exponentially per linear increase in frontal area, but drag only increases linearly to frontal area (despite increasing exponentially to speed). So a 10% weight increase might mean 8% more power and ~6% more frontal area, a benefit for TT but not necessarily climbing. The weight of the bike is a bigger percentage of total weight being moved for really tiny guys though which probably makes a small difference when 2 climbers of different size are at a similar level.
TT rules have benefited shorter riders because of the way they have to set up their bikes, so that is actually a rule worth changing to even the playing field, which they did this year, aka tall people now reap the benefits that were artificially taken from them.
Remco looks freakishly aero, but small guys like him have to be in order to approach the same power to CdA of the top big guys. There will probably be a top big guy built with godlike proportions like Remco some day and you’d expect him to be even better than we’ve ever seen.
Then there’s all the stuff like lung capacity and cardiac output, heat dissipation, which aren’t necessarily only important in relative terms..