Well, in this case the danger would mainly be from gusts of wind, which are completely down to the weather, and not at all down to rider behavior. So things like deciding to use road bikes or no disc wheels allowed are actually fine safety measures by me.That is not the same logic, you've taken it to an absurd end for the purposes of avoiding all nuance. This is with regards to an extremely specific situation, not 'every descent'.
What do you do in extreme weather conditions which fall on that boundary? Creating a binary of if it's not too dangerous to cancel it is safe enough to race or cancel is not a serious suggestion, it's just the opinion of a cranky viewer.
But weather you're going all out or not has very little bearing on weather you get swept up by a gust of wind, which is why the safety argument for neutralization makes no sense to me.