It's about a lot more than that. Again, several WT pros have been on the start line for Unbound, and they're 0'fer.
The problem is that you are going to see people scout through the results of these races, and see names that they recognise, and whose performance levels they know, and extrapolate from that.
For example, looking at Unbound this year, obviously Swenson won it, but then you see Petr Vakoč in 2nd - 8 years as a pro with 5 at the WT level, but whose level never rebounded after his 2018 injuries.
You have Lachlan Morton 3rd, somebody who has been around the pro scene for a decade and whose best year was actually with Jelly Belly, largely topping out as a domestique at the top level.
You have Laurens ten Dam 4th, a 42-year-old who has been retired as a road pro for four years and whose results had been tending downward for a few years prior to that.
Ian Boswell is 5th, also road retired for four years, but a decade younger, somebody who also topped out at WT domestique kind of level.
Peter Stetina is in 7th, likewise retired from road in 2019, was a very good mountain domestique back in 2012 and was very consistent throughout his career at a mid-tier WT pro kind of level.
In 10th you have Jasper Ockeloen, a former Rabobank CT development rider who never lived up to expectations and topped out at the Continental Pro level, retiring from road cycling five years ago.
In 12th you have Alexey Vermeulen, still semi-active on the road but who doesn't ride a full calendar, and hasn't had a pro team since 2018. He rode a couple of years at the top level but didn't stick.
In 14th you have Jan Bakelants, a recent retiree at 37 who was once a very good WT pro but whose results had been diminishing for a few years prior to his retirement.
In 19th you have Larry Warbasse, your example of a WT pro who has been on the startlist and not won - he had flown to the US from Italy after finishing the Giro five days prior, so definitely you can't really say he prepared for the race optimally, shall we say.
That's 9 riders in the top 20 whose abilities fans of road cycling will be aware of or familiar with, and who will impact - and potentially prejudice - the esteem they hold the discipline in. After all, you're looking at a lot of riders who retired several years ago, or riders who made it to a good level but were never elite, so seeing them up among the stars of the burgeoning gravel scene will give some people a sense that it's kind of a career graveyard or a competition where journeyman pros and XCO/road also-rans can be the big fish in a small pond.
But at the same time, that might not be because it's actually the case, but because the events suit different people and because the format is still in its infancy, it's still developing its own infrastructure and its own specialists. After all, I compared it before to the Ski Classics, which have a different format to the World Cup cross-country; lots of mostly flat and rolling terrain, if there are climbs they tend to be longer and more sustained, more double poling, and much longer distances on average than you see in the World Cup; but if you look at winners of events like Vasaloppet, the pinnacle of this calendar, you see plenty of ex-World Cup skiers and veterans filling the front of the field, because the less dependency on fast twitch fibres, the longer an older athlete can stay competitive, and the peak years of endurance athletes tend to be later than the peak years of athletes in more explosive disciplines as a result, so with gravel relying a lot more on endurance than comparable CX and road formats it is perhaps less surprising to see older athletes succeeding in it.
Likewise sportscar racing, where you can find ex-F1 and Indycar drivers being competitive well into their 40s at Le Mans, as experience in car management and discipline becomes more essential than raw pace; it's the combination of speed and safe speed that is more important, and so many drivers who are elite F1 drivers but who have an aggressive driving style full of corrections and minuscule adjustments will not adapt as well as drivers who were less elite in F1 but have smoother steering and style that places less stress on the car; it's two different styles of racing, and being good at one will give you a leg up to being good at the other, but is not the be all and end all. I see gravel and road cycling as being similar in this respect.
I suspect the problem at the moment is that there is very little by way of recognition to attract young pro riders to gravel as a format of choice; until there are sufficient places where professional riders - not hobbyists and independents - select gravel as their preferred format and can make enough money in it for it to be a viable alternative to XCO and road, then you will always see the likes of Keegan Swenson as outliers; if the discipline is largely drawing its better known names from other disciplines, then those comparisons of the performance levels of riders - regardless of how fair or unfair they may be - based on their performances in those other disciplines will always be a hindrance.