You can't really improve the prestige of those jerseys by wanting it.It shows me, that the Tour should make the secondary jerseys even more of a big deal.
Imagine what a race we would get, if the Tour started with 4-5 teams having the green jersey as a stated goal, and 4-5 teams with polka dots as the stated goal, because those two jerseys were almost as important as yellow.
If anything I'd like to see the KoM ditched for a sort of inter giro jersey