I am only aware of scriptures that deal with the origin of the 4 Hindu castes. How about outcastes and Mlecchas?
First of all there is no 5th caste.
Manu Smriti 10.4. Brahmana, the Kshatriya, and the Vaisya castes (varna) are the twice-born ones, but the fourth, the Sudra, has one birth only; there is no fifth (caste)
So, the outcastes will be born out the mixtures of the other 4. These kinds mixtures are proscribed in the scriptures.
10.10. Children of a Brahmana by (women of) the three (lower) castes, of a Kshatriya by (wives of) the two (lower) castes, and of a Vaisya by (a wife of) the one caste (below him) are all six called base-born (apasada).
10.11. From a Kshatriya by the daughter of a Brahmana is born (a son called) according to his caste (gati) a Suta; from a Vaisya by females of the royal and the Brahmana (castes) spring a Magadha and a Vaideha.
10.12. From a Sudra are born an Ayogava, a Kshattri, and a Kandala, the lowest of men, by Vaisya, Kshatriya, and Brahmana) females, (sons who owe their origin to) a confusion of the castes.
These Chandala, Suta etc are considered in the scriptures as outcastes. Again, out of the unions of these outcastes will be born numerous other kinds of outacstes.
Mleccha basically indicates a foreigner or one who is alien to the Vedic culture. It is quite possible that such races originally beloned to one of the 4 castes but they degraded themselves to the status of the Vratyas by not following the Samskaras and prescribed rituals.
10.43. But in consequence of the omission of the sacred rites, and of their not consulting Brahmanas, the following tribes of Kshatriyas have gradually sunk in this world to the condition of Sudras;
10.44. (Viz.) the Paundrakas, the Kodas, the Dravidas, the Kambogas, the Yavanas, the Sakas, the Paradas, the Pahlavas, the Kinas, the Kiratas, and the Daradas.
10.45. All those tribes in this world, which are excluded from (the community of) those born from the mouth, the arms, the thighs, and the feet (of Brahman), are called Dasyus, whether they speak the language of the Mlekkhas (barbarians) or that of the Aryans.
So, the Chinese race is mentioned here as originally to be a Kshatriya race before they degraded themselves to the status of Vratyas.
Therefore outcastes are born out of proscribed unions of the 4 castes. And, Mlechhas are created probably due to failure of performing the rituals by the 4 castes.