Sometimes he wheelsucks because he simply can't do anything else, like in the Giro. This, I don't think anybody would hold against him. Other times, when he simply will not take the initiative in the situation, such as on Ventoux in the Dauphiné last year, is all about sucking wheels, and that's where his bad reputation comes from.
He's far from the worst wheelsucker in the péloton - José Serpa Pérez and Levi Leipheimer are two I can name off hand who are worse - but because he has the reputation for sucking wheels, people notice it more when it's Cadel. And other teams call him out for it more because he usually doesn't have a teammate up there, so it's more obvious.