First of all, the caste (of the would be Sannyasi) does matter. A person who's not performing the duties of his caste (Varna) is declared unfit to renounce.. As also persons who do not have the Vedic fire..So, by this, the lowest caste persons are clearly ruled out..
- One should never allow renunciation to those who have just settled as house-holders, those who have committed great sins, those who have
lost caste due to non-performance of the principal purificatory rites
(vratyas) and the accursed.--- From the Narada Parivrajaka Upanishad
But during the Sannyasa rituals the aspirant sacrifices the symbols of his caste- the top-knot or the Shikha and the sacred thread or the Yajnopavita.
Now, when a man has come to know the highest truth, the eternal
Brahman, let him then take a single staff and discard the topknot
together with the sacred string --From the same Upanishad.
Now, after taking Sannyasa the ascetic does not seem to have a caste, because if he had, he would have been bound by the mandatory duties that are his Varna Dharmas. But, he does not seem to have anything of that sort.
"Fools, tied to customs of class and order, get their rewards
according to their deeds. Men who abandon the duties of class and the
like become sated by their own bliss.
- "Classes and orders contain subdivisions and have a beginning and an end. They are nothing but a great torment. Free from attachment to
the body and to sons and the like, let a man live in supreme and
Maitreya Upanishad's chapter 4 (linked to the Sama Veda)
Few more verses from the same Upanishad's 1st chapter:
I-16. How can the danger (of duality) approach me, resorting as I do
to the inner bliss of the Self, who despise the female goblin of
desires, who view the phenomenal world as in illusion and who am
unattached to it ? I-17. Those ignorant people who stick to castes and
orders of life obtain the (worthless) fruit of their respective
actions. Those who discard the ways of caste, etc., and are happy with
the bliss of the Self become merged in Brahman (lit. Purushas). I-18.
The body consisting of various limbs and observing the (rules of)
castes and orders has a beginning and an end and is only a great
trouble. Free of attachment to one’s children, etc., and the body, one
should live in the endless supreme happiness.
So, as per these verses, Sannyasi discard the rules of caste.. and they do so by renouncing.
Also, for the Sannyasi, the caste is like any other bondages, and which they have renounced.
- Bondage consists in imagining a plunge into the flux of existence with its possessive claims on fields, gardens, houses, children,
wives, brothers, mothers and fathers.
- Bondage is the conceit of egoistic agency in regard to actions, etc.
- Bondage is the imagination prompted by the desire for the eight powers, anima, etc.
- Bondage is the imagination prompted by the yearning for adoring gods, men, etc.
- Bondage is the imagination (leading to) the practice of Yoga with its eight limbs, Yama, etc.
24. Bondage is the planning of action and duties bound up with castes and stations of life.
From Niralamba Upanishad (linked to Shukla Yajurveda)
But a minor conflict arises when we see the following verse from another minor Upanishad:
"A mendicant thus should neither accept nor even look at such things
as gold. One may respond that there is no harm in merely looking at
them.  On the contrary, it is truly harmful, because if a
mendicant looks at gold with longing he becomes a murderer of a
Brahmin; if a mendicant touches gold with longing he becomes an
Paramhamsa Upanishad's chapter 4
Now, how can one become an outcaste, who do not have any caste at all?
But, besides that, it's safe to conclude that the Sannyasis don't retain any caste.