We cannot fault a rider for the quality of their competition, you can only beat the riders next to you at the start.
In any case, Cav was (and maybe will be again) dominant enough and versatile enough that it should not matter. As highly as people rate the Tour Cav has more wins than anyone except Merckx. Points jerseys in all three grand tours, maybe most notably the hard fought 2013 Giro. A World Championship and one silver on the road, and if we want to consider the track, three more golds and another silver.
And for those who want to credit all his wins to Renshaw and the HTC train, consider his MSR win. One of the most impressive sprinting performances we have seen.