Do the Vedas talk directly about avataras?
Are there any direct references in main Vedic texts?
Note: kindly limit the references to samhita, brahmanas, aranyaka and 10 upanishads.
Yes Vedas do talk about avataras. Shathapatha Brahmana tells us that Prajapati assumed the Kurma Avatara. In Shukla Yajur Veda Samhita we find mantras which says Supreme Brahman took birth as Brahma. It also points towards Vaman Avatara.
सुभू: स्वयम्भू: प्रथमोन्तर्महत्यर्णवे ।
दधे ह गर्भमृत्वियं यतो जात: प्रजापतिः।।23.63।।
Then originator of whole universe "Swayambhu Paramatma" (Supreme Soul) in the middle of a great lake ,laid down embryo timely ,from which Brahma (Prajapati) was born.Yajurveda Samhita 23.63
Here we can see that ,the supreme soul himself incarnated as Brahma. Through womb.
5 And as to its being called 'kūrma' (tortoise);Prajāpati, having assumed that form, created living beings. Now what he created, he made; and inasmuch as he made (kar), he is (called) 'kūrma;' and 'kūrma' being (the same as) 'kaśyapa' (a tortoise), therefore all creatures are said to be descended from Kaśyapa.
This mantra indirectly indicates towards Lord Vishnu's "Narasimha Avatara". i.e. Man-Lion form .As it says Lord Vishnu roam around the earth in the form of fierce Lion etc.
प्र तद्विष्णु: स्तवते वीर्येण मृगो न भीमः कुचरो गिरिष्ठाः।
यस्योरुषु त्रिषु विक्रमणेष्वधिक्षियान्ति भुवनानि विश्वा ।।5.20।।
Pra TadVishnuhu Stavate Viryena Mrugo Na Bhima Kucharo Girishtha
YasyoRushu Trishu VikramNeShawaDhiKshiYanti Buvanani Vishwa
Roaming around the earth in fiearc avatars like Lion etc. Residing on mountains , all-pervading Vishnu is highly adorable through his Manhood (Bravery). All the lokas resides under the three giant strides of him. Vishnu deva is praised here as thus.Yajurveda Samhita 5.20
Above mantra is also giving us indirect reference about Lord Vishnu's "Vaman Avatara" as it also talks about three strides of him.
As we know the concept of avatars is more clear in puranas. In Vedas we can see primary source of them. Vedas being abstract than puranas do not describe them in much details. The word avatar does not directly appear in the Vedic literature. But from the definition we can see that avatara also means"to make one's appearance" ,this concept is visible in vedas.
The concept of taking Avatara is not alien to Vedas. For example in Adhyatma Prakarana Sukta of AtharvaVeda Samhita avatara of Mahadeva is described:
व्रात्य आसीदीयमान एव स प्रजापतिं समैरयत् ।।
स प्रजापति: सुवर्णमात्मन्नपश्यत्तत्प्राजनयत् ।।
तदेकमभवत्तल्ललाममभवत्तन्महदभवत्तज्जेष्ठमभवत्तद्ब्रह्माभवत्तत्तपोऽभवत्तत्सत्यमभवत्तेन प्राजायत ।।
सोऽवर्धत स महानभवत्स महादेवोऽभवत् ।।
स देवानामीशां पर्यैत्स ईशानोऽभवत् ।।
स एकव्रात्योऽभवत्स धनुरादत्त तदेवेन्द्रधनुः ।।
नीलमस्योदरं लोहितं पृष्ठम् ।।
नीलेनैवाप्रियं भ्रातृव्यं प्रोर्णोति लोहितेन द्विषन्तं विध्यतीति ब्रह्मवादिनो वदन्ति ।। [Atharva Veda 15:1]
There was Vratya. He roused Prajapati to action. Prajapati beheld gold in himself and engendered it. That became unique, that became distinguished, that became great, that became excellent, that became Brahman, that became Tapas, that became Truth: through that he (Vratya) was born. He grew, he became great, he became Mahadeva. He gained the lordship of the Gods. He became Ishana. He became Eka Vratya. He held a bow, even that Bow of Indra. His belly is dark-blue (Nila), his back is red (Lohitam). With dark-blue he envelops a detested rival, with red he pierces the man who hates him: so the knowers of Brahman say.
So, the theory seen above is:
Vratya (Supreme Being) ➡Takes birth through Prajapati➡becomes Mahadeva/Ishana/Eka Vratya