Well Tony stands 1.86m so not that small

However, Tony is a power rider. The problem for those guys is, that they have the power to do well in any terrain, but recovery is a issue. So a guy like Tony has to develop into a GT classement rider, has to ad the endurance to his power to simplicify things. In 2009 he was on a very good way, and last year was a setback, it happens. However he learnt from his mistkes and should be back on track.
Wheras those tall super light weight riders, they are not living on their power, but on their low weight/power ratio. However, it's a very thin line for them and once they have hit this low weight/power peak point there's not much room to go for them.
They can improve in endurance a little, but their leg power is always the limitation. If they wanted to become more powerful legs they would have to change their nutrition, hence gaining weight, hence their weight/power ratio won't go up. Another thing is, that having such a little weight is very bad for your bodys substance, hence those super light tall guys usually don't last that long. They've a shorter peak span than other riders.