It is exactly what we are talking about. When the potential gain is smaller for an already aerodynamic rider from a road bike to a TT bike, compared to a lesser aerodynamic rider, then this also means that starting from your own statement ("Being better on flat TT doesn’t necessarily translate to being a better rouleur") that it in fact does mean the loss (as opposed to gain) on a road bike (compared to a TT bike) is bigger for the lesser aerodynamic rider, in this case, Pogacar. It's the exact same argument, just looking at it from the opposite end. And we all agree Evenepoel is the more aerodynamic rider.
To make it easier to understand, when you have two cars of the same weight, same size, equal suspension etc, but one has a more powerful engine and the other has better aerodynamics. Now, assuming you let the same people rework the body to make them more aerodynamic, which car do you believe can make the bigger improvements? The one that already is aerodynamic, or the one that is not so aerodynamic but more powerful?