First of all, the sons who are born out of proscribed marriages will not belong to any of the 4 castes (varna-s). As there is no 5th caste, they all will belong to the unknown caste or the out-cast. But they have different names so that they can be differentiated.
10.4. Brahmana, the Kshatriya, and the Vaisya castes (varna) are the twice-born ones, but the fourth, the Sudra, has one birth only; there
is no fifth (caste).
Secondly, this topic is already dealt with in great detail in the Manu Smriti. See the following collection of verses:
10.8. From a Brahmana a with the daughter of a Vaisya is born (a son) called an Ambashtha, with the daughter of a sudra a Nishada, who is
also called Parasava.
10.9. From a Kshatriya and the daughter of a Sudra springs a being, called Ugra, resembling both a Kshatriya and a Sudra, ferocious in his
manners, and delighting in cruelty
10.10. Children of a Brahmana by (women of) the three (lower) castes, of a Kshatriya by (wives of) the two (lower) castes, and of a Vaisya
by (a wife of) the one caste (below him) are all six called base-born
(apasada).
10.11. From a Kshatriya by the daughter of a Brahmana is born (a son called) according to his caste (gati) a Suta; from a Vaisya by females
of the royal and the Brahmana (castes) spring a Magadha and a Vaideha.
10.12. From a Sudra are born an Ayogava, a Kshattri, and a Kandala, the lowest of men, by Vaisya, Kshatriya, and Brahmana) females, (sons
who owe their origin to) a confusion of the castes.
10.13. As an Ambashtha and an Ugra, (begotten) in the direct order on (women) one degree lower (than their husbands) are declared (to be),
even so are a Kshattri and a Vaidehaka, though they were born in the
inverse order of the castes (from mothers one degree higher than the
fathers).
10.14. Those sons of the twice-born, begotten on wives of the next lower castes, who have been enumerated in due order, they call by the
name Anantaras (belonging to the next lower caste), on account of the
blemish (inherent) in their mothers.
10.15. A Brahmana begets on the daughter of an Ugra an Avrita, on the daughter of an Ambashtha an Abhira, but on a female of the Ayogava
(caste) a Dhigvana.
10.16. From a Sudra spring in the inverse order (by females of the higher castes) three base-born (sons, apasada), an Ayogava, a
Kshattri, and a Kandala, the lowest of men;
10.17. From a Vaisya are born in the inverse order of the castes a Magadha and a Vaideha, but from a Kshatriya a Suta only; these are
three other base-born ones (apasada).
10.18. The son of a Nishada by a Sudra female becomes a Pukkasa by caste (jati), but the son of a Sudra by a Nishada female is declared
to be a Kukkutaka.
10.19. Moreover, the son of by Kshattri by an Ugra female is called a Svapaka; but one begotten by a Vaidehaka on an Ambashtha female is
named a Vena.
Some occupations are mentioned in the following verses:
10.32. A Dasyu begets on an Ayogava (woman) a Sairandhra, who is skilled in adorning and attending (his master), who, (though) not a
slave, lives like a slave, (or) subsists by snaring (animals).
10.33. A Vaideha produces (with the same) a sweet-voiced Maitreyaka, who, ringing a bell at the appearance of dawn, continually. praises
(great) men.
10.34. A Nishada begets (on the same) a Margava (or) Dasa, who subsists by working as a boatman, (and) whom the inhabitants of
Aryavarta call a Kaivarta.
10.35. Those three base-born ones are severally begot on Ayogava women, who wear the clothes of the dead, are wicked, and eat
reprehensible food.
In general, for all practical purposes, they can be treated as a Sudra. So, they can betake the jobs which are prescribed for the Sudras in the scriptures.
10.41. Six sons, begotten (by Aryans) on women of equal and the next lower castes (Anantara), have the duties of twice-born men; but all
those born in consequence of a violation (of the law) are, as regards
their duties, equal to Sudras.
All verses are from the Manu Smriti.