Indeed Ag2r were already pushing it on the front, although they possibly started pushing a little harder once they knew Thomas wasn't there. Since they were riding already however, I don't think they had a reason to wait. The fact they continued to push even after Thomas made it back also suggests they weren't just riding against Thomas. I also heard the suggestion that they were riding for the white jersey of Latour.
Furthermore, I don't think Sky were whining that bad. Thomas didn't go up to Bardet to scold him, I think "thanks for that" is a reasonable way to respond to the situation.
I don't think it matters if you're going downhill in the peloton or uphill with only the favourites though, what matters more is if you were already riding on the front, although there are of course no written rules prohibiting you from attacking regardless.