They form hydrophobic bonds, which are basically the kind of interactions that cause oil droplets to coalesce in water. In an aqueous environment like the blood, a non-polar portion of any protein molecule can bind other non-polar molecules and uncharged atoms like Xenon. The energy for bonding is provided by the repulsive force between the non-polar molecules or atoms and water.
As an analogy, charged or electrostatic interactions are like those between a man and a woman--opposites attract. Non-polar or hydrophobic interactions are like those between two Americans of either gender in a third world country. Their dissimilarities from people of a very different cultural background act as a bond between them. Notice that the interaction is dependent on context, i.e., a surrounding environment that is very different from the interacting entities. On an alien planet, even Scott and VeloCity would bond closely. Here in the U.S., not so much.
Xenon binding to Hb would presumably be non-competitive, that is, it would affect oxygen binding not by competing for the same binding site as oxygen, but by binding to a distance part of Hb and thereby changing the conformation of the molecule. If this conformational change extends to the portion of the molecule binding oxygen, it can reduce this binding. This is non-competitive interaction.
As a fun analogy, imagine a rider taking out another rider at the sprint by forcing him into the wall. That is competitive inhibition. Then imagine a rider falling down in the front of the peloton, resulting in a massive pileup, with a race favorite caught in the back. That is non-competitive inhibition.
Non-competitive inhibition is very common in biochemistry, but usually it involves binding to specific sites. In the case of Xenon, though the binding is presumably quite nonspecific--any non-polar region.