I was thinking about this today, and wondering how someone who is actually racing and training on the road is gaining muscle bulk. Yes, Wiggo had a track focus, but he sure as hell was still riding and racing on protour teams on the road, pretty much the rest of the year.
And despite it being a track event, other than the start, upper body muscle doesn't do much for a pursuiter. You're sitting down after a lap or less, and weight in the corners - muscle or otherwise - is not an advantage, the opposite is truer. The first lap is 1/16 of the race, and you can lose say 1 second on a bad vs a good start at that level, which is ~1/240th of the event duration.
My guess is there was far less upper body muscle to lose than some people think.
They may be track riders, but they're still doing 20,000 km a year training.