I'm no pro, but I've ridden up Ventoux from Bedoin five times over the years. I rate it the hardest climb in France that I've done.
I rank it a level above Tourmalet, Izoard, Madeleine, Telegraph/Galibier. I'd rank Alpe d'Huez a level down again, along with Pailheres, Hautacam, Aubisque...