Xander spends months choosing to not talk about his fears that have always existed.

Then right before marriage -Xander lives through the physical manifestation of his greatest fears. it makes hella sense that is the straw that breaks the camel's back.

I understand why he left Anya at the altar. In that moment leaving the situation was the kindest thing he could do according to him.

He was triggered and deeply upset and so he left the building, kinda like he left his house every holiday and slept outside to avoid his family's fighting.