I actually think the profile of the Col de la Loze looks possibly a tiny bit harder than the Col de Portet, but it's really difficult to say. The Italian side of the Agnello, the north side of the Galibier (if you see it as one climb) or the Ventoux are also very comparable but it's fair to say the Col de la Loze is among the most difficult.
The crucial thing in this sort of discussion is though, imo the profiles say a lot more than a guy who says something on TV. Juan Antonio Fletcher usually calls ten climbs the hardest climb of the giro every year, the guys who say this sort of stuff might not have ridden every other contender for hardest TdF climb, and even if they did their condition might have been very different. Just because someone says something doesn't mean it's true.