And again you are forgetting that is it likely all of the top riders from the 1980s until at least 2005 were doping, and many of them were using the same doctors and the same methods. What I am suggesting is that all of the top riders who were doping, probably had fairly similar enhancements in BOTH performance and recovery. If you take that away from ALL of them, then they all go a bit slower and recover a bit more poorly, which makes them go a bit slower in the last week of a GT. The TdF is 100yrs old. We know from history that it is humanly possible to complete such a feat of endurance without the use of EPO..... you just go slower.
In actual fact, some people probably do respond better to PEDs than others and some riders probably did have a bigger PED budget. So if you take the PEDs away, then maybe some riders have a bigger performance drop than others. However, the performance drop (which is analogous to performance gain) will not be so great that a 7 time consecutive TdF winner, suddenly becomes a rank and file rider that cannot ever rise beyond water bottle fetcher. If you take away the PEDs from him and nobody else, then sure, but not if you keep the playing field level.