What are the order & duties of the children born from intermixing/hybridisation of 4 classes?

Question

There are 4 classes in descending order:

1. BrAhmaNa
2. Kshatriya
3. Vaishya
4. Shudra

If a man of class X mates with a woman of class X, then definitely the child will inherit the class X.

How the intermixing of 2 classes (X & Y) works with respect to the child's class & duties?

Note: caste-system is not appropriate for this Qn, as we are discussing about "classes" and not the "castes". See this.

Update: Some people may get offended due to wrong tagging in our site, so I will make a clarification here. There is a big difference between "caste-system" (tag available) and "class-system" (tag rejected). See this answer for a detailed reference.
In short, if we consider "Bill Gates" and "Bill Clinton" as head of their family trees, then "Gates" & "Clinton" are castes, however "businessman" and "politician" are classes. Some may find former as racist, however this Qn intends to discuss about the latter.

Answer 1

First of all, the sons who are born out of proscribed marriages will not belong to any of the 4 castes (varna-s). As there is no 5th caste, they all will belong to the unknown caste or the out-cast. But they have different names so that they can be differentiated.

10.4. Brahmana, the Kshatriya, and the Vaisya castes (varna) are the twice-born ones, but the fourth, the Sudra, has one birth only; there is no fifth (caste).

Secondly, this topic is already dealt with in great detail in the Manu Smriti. See the following collection of verses:

10.8. From a Brahmana a with the daughter of a Vaisya is born (a son) called an Ambashtha, with the daughter of a sudra a Nishada, who is also called Parasava.

10.9. From a Kshatriya and the daughter of a Sudra springs a being, called Ugra, resembling both a Kshatriya and a Sudra, ferocious in his manners, and delighting in cruelty

10.10. Children of a Brahmana by (women of) the three (lower) castes, of a Kshatriya by (wives of) the two (lower) castes, and of a Vaisya by (a wife of) the one caste (below him) are all six called base-born (apasada).

10.11. From a Kshatriya by the daughter of a Brahmana is born (a son called) according to his caste (gati) a Suta; from a Vaisya by females of the royal and the Brahmana (castes) spring a Magadha and a Vaideha.

10.12. From a Sudra are born an Ayogava, a Kshattri, and a Kandala, the lowest of men, by Vaisya, Kshatriya, and Brahmana) females, (sons who owe their origin to) a confusion of the castes.

10.13. As an Ambashtha and an Ugra, (begotten) in the direct order on (women) one degree lower (than their husbands) are declared (to be), even so are a Kshattri and a Vaidehaka, though they were born in the inverse order of the castes (from mothers one degree higher than the fathers).

10.14. Those sons of the twice-born, begotten on wives of the next lower castes, who have been enumerated in due order, they call by the name Anantaras (belonging to the next lower caste), on account of the blemish (inherent) in their mothers.

10.15. A Brahmana begets on the daughter of an Ugra an Avrita, on the daughter of an Ambashtha an Abhira, but on a female of the Ayogava (caste) a Dhigvana.

10.16. From a Sudra spring in the inverse order (by females of the higher castes) three base-born (sons, apasada), an Ayogava, a Kshattri, and a Kandala, the lowest of men;

10.17. From a Vaisya are born in the inverse order of the castes a Magadha and a Vaideha, but from a Kshatriya a Suta only; these are three other base-born ones (apasada).

10.18. The son of a Nishada by a Sudra female becomes a Pukkasa by caste (jati), but the son of a Sudra by a Nishada female is declared to be a Kukkutaka.

10.19. Moreover, the son of by Kshattri by an Ugra female is called a Svapaka; but one begotten by a Vaidehaka on an Ambashtha female is named a Vena.

Some occupations are mentioned in the following verses:

10.32. A Dasyu begets on an Ayogava (woman) a Sairandhra, who is skilled in adorning and attending (his master), who, (though) not a slave, lives like a slave, (or) subsists by snaring (animals).

10.33. A Vaideha produces (with the same) a sweet-voiced Maitreyaka, who, ringing a bell at the appearance of dawn, continually. praises (great) men.

10.34. A Nishada begets (on the same) a Margava (or) Dasa, who subsists by working as a boatman, (and) whom the inhabitants of Aryavarta call a Kaivarta.

10.35. Those three base-born ones are severally begot on Ayogava women, who wear the clothes of the dead, are wicked, and eat reprehensible food.

In general, for all practical purposes, they can be treated as a Sudra. So, they can betake the jobs which are prescribed for the Sudras in the scriptures.

10.41. Six sons, begotten (by Aryans) on women of equal and the next lower castes (Anantara), have the duties of twice-born men; but all those born in consequence of a violation (of the law) are, as regards their duties, equal to Sudras.

All verses are from the Manu Smriti.

Answer 2

Note: Actually it's better, if I make a chart of all the intermixed children names & duties. However, I am posting this answer to make a reference elsewhere. Later depending on time & interest, I will be creating a chart and posting here.

The names in "bold" are the different castes.

---

From AnusAsana Parva:

Yudhishthira said, 'Through inducements offered by wealth, or through mere lust, or through ignorance of the true order of birth (of both males and females), or through folly, intermixture happens of the several order What, O grandsire, are the duties of persons that are born in the mixed classes and what are the acts laid down for them?

---

Bhishma said, 'In the beginning, the Lord of all creatures created the four orders and laid down their respective acts or duties, for the sake of sacrifice.
The Brahmana may take four wives, one from each of the four orders. In two of them (viz., the wife taken from his own order and that taken from the one next below), he takes birth himself (the children begotten upon them being regarded as invested with the same status as his own). Those sons, however, that are begotten by him on the two spouses that belong to the next two orders (viz., Vaisya and Sudra), are inferior, their status being determined not by that of their father but by that of their mothers. The son that is begotten by a Brahmana upon a Sudra wife is called Parasara, implying one born of a corpse, for the Sudra woman's body is as inauspicious as a corpse. He should serve the persons of his (father's) race. Indeed, it is not proper for him to give up the duty of service that has been laid down for him. Adopting all means in his power, he should uphold the burden of his family. Even if he happens to be elder in age, he should still dutifully serve the other children of his father who may be younger to him in years, and bestow upon them whatever he may succeed in earning.

A Kshatriya may take three wives. In two of them (viz., the one taken from his own order and the other that is taken from the order immediately below), he takes birth himself (so that those children are invested with the status of his own order). His third wife being of the Sudra order is regarded as very inferior. The son that he begets upon her comes to be called as an Ugra.

The Vaisya may take two spouses. In both of them (viz., the one taken from his own order, and the other from the lowest of the four pure orders), he takes birth himself (so that those children become invested with the status of his own order).

The Sudra can take only one wife, viz., she that is taken from his own order. The son begotten by him upon her becomes a Sudra.

---

A son that takes birth under circumstances other than those mentioned above, comes to be looked upon as a very inferior one If a person of a lower order begets a son upon a woman of a superior order, such a son is regarded as outside the pale of the four pure orders. Indeed, such a son becomes on object of censure with the four principal orders.

---

If a Kshatriya begets a son upon a Brahmana woman, such a son, without being included in any of the four pure orders, comes to be regarded as a Suta The duties of a Suta are all connected with the reciting of eulogies and encomiums of kings and other great men.

The son begotten by a Vaisya upon a woman of the Brahmana order comes to be regarded as a Vaidehaka. The duties assigned to him are the charge of bars and bolts for protecting the privacy of women of respectable households. Such sons have no cleansing rites laid down for them.

If a Sudra unites with a woman belonging to the foremost of the four orders, the son that is begotten is called a Chandala. Endued with a fierce disposition, he must live in the outskirts of cities and towns and the duty assigned to him is that of the public executioner. Such sons are always regarded as wretches of their race. These, O foremost of intelligent persons, are the offspring of intermixed orders.

The son begotten by a Vaisya upon a Kshatriya woman becomes a Vandi or Magadha. The duties assigned to him are eloquent recitations of praise.

The son begotten through transgression, by a Sudra upon a Kshatriya women, becomes a Nishada and the duties assigned to him have reference to the catching of fish.

If a Sudra happens to have intercourse with a Vaisya woman, the son begotten upon her comes to be called Ayogava. The duty assigned to such a person are those of a Takshan (carpenter). They that are Brahmanas should never accept gifts from such a person. They are not entitled to possess any kind of wealth.

---

Persons belonging to the mixed castes beget upon spouses taken from their own castes children invested with the status that is their own. When they beget children in women taken from castes that are inferior to theirs, such children become inferior to their fathers, for they become invested with the status that belongs to their mothers Thus as regards the four pure orders, persons beget children invested with their own status upon spouses taken from their own orders as also upon them that are taken from the orders immediately below their own. When, however, offspring are begotten upon other spouses, they come to be regarded as invested with a status that is, principally, outside the pale of the four pure orders. When such children beget sons in women taken from their own classes, those sons take the status of their sires. It is only when they take spouse from castes other than their own, that the children they beget become invested with inferior status. As an example of this it may be said that a Sudra begets upon a woman belonging to the most superior order a son that is outside the pale of the four orders (for such a son comes to be regarded as a Chandala who is much inferior). The son that is outside the pale of the four orders by uniting with women belonging to the four principal orders, begets offspring that are further degraded in point of status. From those outside the pale of the four orders and those again that are further outside that pale, children multiply in consequence of the union of persons with women of classes superior to their own. In this way, from persons of inferior status classes spring up, altogether fifteen in number, that are equally low or still lower in status. It is only from sexual union of women with persons who should not have such union with them that mixed classes spring up. Among the classes that are thus outside the pale of the four principal or pure orders, children are begotten upon women belonging to the class called Sairindhri by men of the class called Magadha. The occupation of such offspring is the adornment of the bodies of kinds and others. They are well-acquainted with the preparation of unguents, the making of wreaths, and the manufacture of articles used for the decoration of the person. Though free by the status that attaches to them by birth, they should yet lead a life of service. From the union of Magadhas of a certain class with women of the caste called Sairindhri, there springs up another caste called Ayogava. Their occupation consists in the making of nets (for catching fish and fowl and animals of the chase). Vaidehas, by uniting themselves with women of the Sairindhri caste, beget children called Maireyakas whose occupation consists in the manufacture of wines and spirits. From the Nishadas spring a caste called Madgura and another known by the name of Dasas whose occupation consists in plying boats. From the Chandala springs a race called Swapaka whose occupation consists in keeping guard over the dead. The women of the Magadhi caste, by union with these four castes of wicked dispositions produce four others who live by practising deceit. These are Mansa, Swadukara, Kshaudra, and Saugandha. From the Vaideha springs up a cruel and sinful caste that lives by practising deception. From the Nishadas again springs up the Madranabha caste whose members are seen to ride on cars drawn by asses. From the Chandalas springs up the caste called Pukkasa whose members are seen to eat the flesh of asses, horses and elephants. These cover themselves with the garments obtained by stripping human corpses. They are again seen to eat from broken earthenware These three castes of very low status are born of women of the Ayogava caste (by fathers taken from different castes). The caste called Kshudra springs from the Vaidehaka. The caste called Andhra which takes up its residence in the outskirts of towns and cities, also springs up (from the Vaidehakas). Then again the Charmakara, uniting himself with a woman of Nishada caste, begets the class called Karavara. From the Chandala again springs up the caste known by the name of Pandusaupaka whose occupation consists in making baskets and other things with cleft bamboos. From the union of the Nishada with a woman of the Vaidehi caste springs one who is called by the name of Ahindaka. The Chandala begets upon a Saupaka woman, a son that does not differ from the Chandala in status or occupation. A Nishada woman, by union with a Chandala, brings forth a son who lives in the outskirts of villages and towns. Indeed, the members of such a caste live in crematoria and are regarded by the very lowest orders as incapable of being numbered among them. Thus to these mixed castes spring up from improper and sinful union of fathers and mothers belonging to different castes. Whether they live in concealment or openly, they should be known by their occupations. The duties have been laid down in the scriptures for only the four principal orders. As regards the others the scriptures are entirely silent. Among all the orders, the members of those castes that have no duties assigned to them by the scriptures, need have no fears as to what they do (to earn their livelihood).

Overall it's recommended Not to let the marriage happen among unauthorised combination for the good of society.