One thing to keep in mind is that riding a climb, and racing a climb, are two completely different things. If you're just riding a climb, you can rest on the flatter bits and save your maximum effort for where it's necessary. If you're racing, you may have to maintain maximum effort all the way up just to hang on to competitors' wheels.
One of the hardest climbs I have ever done was a 10 km climb at a steady 4% - little more than a false flat. Why was it so hard? Because I was trying to hang on to a bunch that had considerably stronger riders than me in it. Secondly, weather and miles in the legs can turn otherwise modest climbs into killers.
However, what makes a climb tough in my experience is a) length, b) sustained steepness, c) altitude, and possibly d) road quality.
The four toughest climbs within a day's drive of where I live are: a 30 km climb averaging 5%, but with 5 km of 10% at the top; a 6 km climb at 11%, a 12 km climb at 8% with the first 5 km at a steady 11-13 and finishing with a km of 13% (reputedly, I pulled out after the first 5, and a 3 km cilmb that has 600 m of 20%, a short dip, and then another 2 km at around 14%.