Armstrong's first proper demise happened in 2005 when L'Equipe ran with the "le mensonge Armstrong" headline about his positive retroactive EPO finding relating to samples taken during the 99 TdF. 
		
		
	
	
It was simply too late to do anything about it in terms of actual punishment. 
Also, it could be argued his downfall started in the 2004 & 2005 Tours when the US Postal team (& then Discovery) started to become a little bit embarrassing, like Hincapie winning a mountain stage & other domestiques also super performing to the point of taking the p*ss.  
It should be noted the mood after the 2003 Tour was super positive (pun intended), i.e. 'people' (audiences & press alike) absolutely loved the Armstrong-Ullrich duel in that Tour which is even today still considered one of the best ever (even with clinical matters taken into account). It was the greed though in the 2004 & 2005 Tours which pretty much f*cked him & his entire team in terms of public relations. Like the general awkwardness expressed by everyone when he sprinted past Klöden just for another power flex, or riding over the 'EPO Lance' road marking as he mowed down Basso on the Alpe ITT. 
And this is one of the reasons why I mentioned the fact Jumbo need to be careful, i.e. for example Bahrain's problems with the authorities pretty much started after Padun's one man demolition job of the Dauphiné's weekend mountains stages a few years ago. All that was required for sh*t to hit the fan was for a few complaints from French teams, some disquiet among the press, accusations of greed etc. & suddenly Bahrain were in the crosshairs of the cops.