In summary (and most of what is said above is correct) ...
Order of Preference:
1. Bike fit, bike fit, bike fit.
2. Saddle. Yup, sore a$$ = not much fun.
3. Great tyres. GP4000s, Schwalbe Ultremo ZX etc = less punctures, more grip, more comfort, more fast. All good.
4. Frame that is designed for what you actually want. For example, if you want to ride crits, or short rides on smooth roads you will choose a very different frame from what you would choose for long days in the saddle, hills or rough roads. Geometry, frame material, build characteristics vary greatly. Compare, for example, the truly hideously awful ride of a Venge with the sublime feel of a [insert your favorite frame here - for me Colnago C59]
5. Wheels can make a massive difference. I suppose hubs are technically "components" but great hubs, rims, spokes and a quality build will make a massive difference.
6. Components. You mentioned 105 vs Ultegra. If you have a limited budget go 105 everytime. If you don't, then DA Di2 11-speed is where it's at
