I have seen many Sannyasis (especially Swaminarayan) that don't look at women unless it's by accident. What is the cause behind this aversive behavior?
For sannyāsis, not looking at women is only the starting point, it is not the destination. They are actually trying to slay their indriyas to the point that they are unaffected by the opposite sex.
Śaṅkarācārya says in Vivekachūdāmani:
The deer, the elephant, the moth, the fish and the honey-bee---these five meet death because of their bondage to one of the five senses. What then is the condition of a person who is attached to all five?
Sense-objects are even more virulent in their tragic effects than a king cobra. Poison is fatal to one who swallows it, but the sense-objects kill him who merely looks at them. with his eyes.
One who is liberated from the terrible bonds of desires for sense-objects, so very difficult to renounce, is alone fit for liberation and none else, even if well-versed in all the six schools of philosophy.
In the legend of Viśvāmitra from Vālmīki Rāmāyaṇa, we see how Viśvāmitra is temporarily distracted by the apsara Menaka and loses track of his tapasya.
In the story of Dharma-vyādha we see that the sage, Kauśika, has probably mastered his senses over the opposite sex (as he has a lengthy conversation with the lady of the house) but he has not slayed his ego (ahankāra) and anger yet as he considers himself superior to a householder.
In the following story of two monks (probably of Buddhist origin), we see how the younger one holds on to the thought of a woman even after the senior monk has long forgotten about the incident.
A senior monk and a junior monk were traveling together. At one point, they came to a river with a strong current. As the monks were preparing to cross the river, they saw a very young and beautiful woman also attempting to cross. The young woman asked if they could help her cross to the other side.
The two monks glanced at one another because they had taken vows not to touch a woman.
Then, without a word, the older monk picked up the woman, carried her across the river, placed her gently on the other side, and carried on his journey.
The younger monk couldn’t believe what had just happened. After rejoining his companion, he was speechless, and an hour passed without a word between them.
Two more hours passed, then three, finally the younger monk could contain himself any longer, and blurted out “As monks, we are not permitted a woman, how could you then carry that woman on your shoulders?”
The older monk looked at him and replied, “Brother, I set her down on the other side of the river, why are you still carrying her?”
So for a sannyāsi, it is not just about 'not seeing' or 'not touching' a person of opposite sex, the ultimate goal is to reach the state where you remain unaffected even when facing the opposite sex.
The reason is actually very simple.For a Sanyasi or a Yogi maintaining celibacy(Brahmacharya) is one among the most important tasks.Its part of their sanyasa related vows as well.
So, they avoid all kinds of "Naari Sansargas"(associations with women) including even talking or looking at them.
Because,even glancing at a women can be considered as contrary to celibacy,as the Daksha Smriti says:
Thinking of a woman or her picture, praising a woman or her picture, sporting with a woman or her picture, glancing at a woman or her picture, secretly talking to a woman, thinking of a sinful action towards a woman actuated by sensuality, determining upon the sinful action, and bodily action resulting in the discharge of semen are the eight characteristics of copulation; and Brahmacharya is quite contrary to all these eight indications.
So, a sanyasi or even only a brahmachari(like students) carefully avoid women as much as possible.
The Manu Smriti even says:
2.215. One should not sit in a lonely place with one’s mother, sister, or daughter; for the senses are powerful, and master even a learned man.
Quoting from the "Ashtanga Yoga-the Significant Means of Shiva Tatwa" chapter of the Linga Purana :
To control the activities or goings on in the mind is Yoga. The Ashtanga or Eight-Limbed Yoga comprises of Yama- Niyama- Aasana- Pranaayama- Pratyahara-Dharana-Dhyana-Samadhi. Yama is denoted by good behaviour comprising Ahimsa (Non-Violence), Satya (Truth), Asteya (stealing), Brahmacharya (avoidance of sex) and Aparigraha or Excessive acquisition. Basic Niyamas is in fact an extension of Yama: To refrain from hurting or torturing co-Beings physically or mentally is Ahimsa; to convey whatever is seen, heard, and believed is Satya unless it does not hurt any body is known as Satya; to desist stealing of other’s property by deed, thought or otherwise is called Asteya; tonegate from relationship with another female or in respect of a female wih another male, excepting the wedded woman or man as the case may be and this should be observed by vision, thought or deed is called Brahmacharya or celibacy;
So, a Yogi or Sanyasi ,who follows the eight-fold path of Yoga ,avoids all forms of women associations as a part of his spiritual development and also to avoid chances of breaking his sanyasa or brahmacharya vows.
It is not just male Sannyasis not looking at women. Women Sannyasis must also avoid men. The reason for avoidance is given in the passage posted below.
Man dangerous to woman in the spiritual path
Just as a woman is a danger to a man desiring progress on the spiritual path, so is a man to woman. It is the man who is uxorious, and engrossed too much with women, that gets a woman’s body in the next birth. Having become a woman, and being unaware that it is My binding power of Maya, as dictated by her Karma, that approached her as husband offering her wealth, children and home, she succumbs to these attractions to Maya without knowing that they mark her spiritual doom, even as a male deer meets with its death at the hands of a hunter who attracts it by his musical call imitating the mating cry of the female.
Srimad Bhagavata Purana III.31.41-42