As I had read in books on Indian History (which certainly do not discuss ideas in detail), the caste system as had been originally devised was not rigid; in other words, a person 'born into' a caste could change his work and thereby, shift his caste. I am curious to know if the scriptures give us an idea about the transition of the nature of the caste system. On a side note, I'd also like to know if untouchability prevailed during the ancient times.
Earliest mention of 4 classes of humans was made in Purusha Sukta from Rig Veda X.90.12.
The Brahman was his mouth, of both his arms was the Rājanya made. His thighs became the Vaiśya, from his feet the Śūdra was produced.
We have to remember that this division was based on guna and karma , but not based on birth based.
This was reiterated by Sri Krishna in Gita, which was a part of Mahabharata.
चातुर्वर्ण्यं मया सृष्टं गुणकर्मविभागशः।
तस्य कर्तारमपि मां विद्ध्यकर्तारमव्ययम्।।4.13।।
The four castes have been created by Me through a classification of the gunas and duties. Even though I am the agent of that (act of classification), still know Me to be a non-agent and changeless.
In Ramayana, there appears to be no stigma in couple belonging to different varnas getting married.
The ascetic boy, who was hit by Dasaratha's arrow, says his parents are from different varnas.
संस्तभ्य शोकं धैर्येण स्थिरचित्तो भवाम्यहम्।
ब्रह्महत्याकृतं पापं हृदयादपनीयताम्।।2.63.52।।
'I am now calm. I bear my pain with patience. Remove the fear from your mind that you have committed a sin by slaying a brahmin.
न द्विजातिरहं राजन्मा भूत्ते मनसो व्यथा।
शूद्रायामस्मि वैश्येन जातो जनपदाधिप।।2.63.53।।
'O king Dasaratha, lord of the land, I am not a brahmin. Feel no sense of guilt in your mind. I was born of a sudra mother and a vaisya father.
So during Ramayana era, which preceded Mahabharata era, there was no stigma in such marriages.
Hence, the division of human proceeded from fluid system to rigid system.
Is there any writing in the scriptures that tell us how the caste system went from fluid to rigid?
No, the caste system never went from fluid to rigid. In fact, only in modern times is it going from rigid to fluid.
I will cite from ancient scriptures like the dharmashastras of Manu, Apastamba, and Itihasas like the Mahabharata to show that the consensus in ancient times among scholars was that the caste system was birth based and that caste is fixed from birth.
Manusmriti 10.5: Among all castes, those only who are born of consorts wedded in the natural order, as virgins of equal status, are to be regarded as the same (as their father).
Āpastamba (2.13.1).—‘Sons begotten by a man who approaches in the proper season a woman of equal caste, who has not belonged to another man, and who has been married legally, have a right to follow the occupations of their castes.’
Viṣṇu (16.1).—‘On women equal in caste to their husbands, sons are begotten who are equal in caste to their fathers.’
Yājñavalkya (1.90).—‘From women of the same caste as their husbands are born sons of the same caste.’
Baudhāyana (1.17.2).—‘Sons of equal caste spring from women of equal caste.’
Caste promotion only happens in subsequent lives, if they have correctly performed their duties:
Āpastamba (2, 11.10-11).—‘In successive births, men of the lower castes are born in the next higher one, if they have fulfilled their duties. In successive births, men of the higher castes are born in the next lower one, if they have neglected their duties.’
Vidura in the Mahabharata war was a Shudra who was very well respected and revered by Brahmanas due to his Brahminical qualities. Many Brahmanas came to him for knowledge. However, even though he could teach some knowledge, he could not instruct them into upasanas and mantras. He says:
As I was born a Shudra, I cannot teach any further. - Mahabharata, Udyoga Parva, 41-5
Vedanta Desikan, a medieval Sri Vaishnava scholar, says this about caste elevation and exceptional cases like Vishvamitra, etc:
Those who abide by the shastras should not believe in the elevation from lower castes stated in the deceptive shastras of heretics. The case of such as Vishvamitra, who became a Brahmin after being a Kshatriya, was due to certain special causes as the charu eaten by his mother without knowing that it was intended for producing a Brahmin child - it should not be considered that it is applicable to others as well, for it is against the texts in the shastras. The incidents and conduct of the Alvars who were possessed of powers even superior to those of Vidura and others should not be taken as precedents for our conduct. If we examine even their conduct carefully, we shall find that they did not transgress the rules of their respective castes. - Rahasyatraya Sara page 303
So, ancient Hindus followed a rigid, birth based caste system.
It is only in modern times that many Hindus believe that the caste system is based on self-identification. Vivekananda was the main person to promote this trend, and reformist Hindus have now adopted this belief. Orthodox followers of Vedanta do not believe that caste is based on anything other than birth.
For verses from scriptures like the Mahabharata that appear to talk about caste elevation in the same life and guna-based caste system, take a look at the correct interpretation in this answer.
Also take a look at my answer here that shows that the Vedas declare that caste is based on birth.
Varna System based on the birth is rigid only in Satyuga when everyone follow their Dharma, Varna and Ashram strictly. As the Yugas deteriorates, varna system based on birth starts collapsing. For example Ravana of Treta Yuga born from a Brahmin father and Rakshasha mother was an example of outcast. Similarly, are the mixed offsprings Yadu, Puru of a Pratiloma/Anuloma marriage etc., of Yayati(a Kshatriya) and Devyani(a Brahmin)/Sharmistha(outsider) and their further offsprings.
When Arjuna asks Krishna the main reason of Mahabharata/Ramayana and deterioration of Dharma in the society by the end of Dvapra-Kaliyuga, Krishna answers that not abiding by the Smriti's birth based varna system and laws is the prime reason that corrupted souls get a chance to take birth in higher castes like Kansa, Duryodhana, Ravana etc., Infact, this is the main reason Satyuga deteriorates into Kaliyuga slowly.
Bhagavad Gita 1.38 – 1.39 Their thoughts are overpowered by greed and they see no wrong in annihilating their relatives or wreaking treachery upon friends. Yet, O Janardan (Krishna), why should we, who can clearly see the crime in killing our kindred, not turn away from this sin?
Bhagavad Gita 1.40 When a dynasty is destroyed, its traditions get vanquished, and the rest of the family becomes involved in irreligion.
Bhagavad Gita 1.41 With the preponderance of vice, O Krishna, the women of the family become immoral; and from the immorality of women, O descendent of Vrishni, unwanted progeny are born.
Bhagavad Gita 1.42 An increase in unwanted children results in hellish life both for the family and for those who destroy the family. Deprived of the sacrificial offerings, the ancestors of such corrupt families also fall.
Bhagavad Gita 1.43 Through the evil deeds of those who destroy the family tradition and thus give rise to unwanted progeny, a variety of social and family welfare activities are ruined.
This was also explained in Mahabharata 13: Anusasana Parva SECTION XLVIII
Thus to these mixed castes spring up from improper and sinful union of fathers and mothers belonging to different castes. Whether they live in concealment or openly, they should be known by their occupations. The duties have been laid down in the scriptures for only the four principal orders. As regards the others the scriptures are entirely silent. Among all the orders, the members of those castes that have no duties assigned to them by the scriptures, need have no fears as to what they do (to earn their livelihood). Persons unaccustomed to the performance or for whom sacrifices have not been laid down, and who are deprived of the company and the instructions of the righteous whether numbered among the four principal orders or out of their pale, by uniting themselves with women of other castes, led not by considerations of righteousness but by uncontrolled lust, cause numerous mixed castes to come into existence whose occupations and abodes depend on the circumstances connected with the irregular unions to which they owe their origin.
Whether a man happens to be possessed of learning or not, lust and wrath are natural attributes of humanity in this world. Women, therefore, may always be seen to drag men into the wrong path. This natural disposition of women is such that man's contact with her is productive of misery to him. Hence, men possessed of wisdom do not suffer themselves to be excessively attached to women.'
The nature of 4 Varnas based on Gunas/Karmas was explained in Geeta Chapter 18
Bhagavad Gita 18.41 The duties of the Brahmins, Kshatriyas, Vaishyas, and Shudras—are distributed according to their qualities, in accordance with their guṇas (and not by birth).
Bhagavad Gita 18.42 Tranquility, restraint, austerity, purity, patience, integrity, knowledge, wisdom, and belief in a hereafter—these are the intrinsic qualities of work for Brahmins.
Bhagavad Gita 18.43 Valor, strength, fortitude, skill in weaponry, resolve never to retreat from battle, large-heartedness in charity, and leadership abilities, these are the natural qualities of work for Kshatriyas.
Bhagavad Gita 18.44 Agriculture, dairy farming, and commerce are the natural works for those with the qualities of Vaishyas. Serving through work is the natural duty for those with the qualities of Shudras.
Bhagavad Gita 18.45 By fulfilling their duties, born of their innate qualities, human beings can attain perfection. Now hear from me how one can become perfect by discharging one’s prescribed duties.
Bhagavad Gita 18.46 By performing one’s natural occupation, one worships the Creator from whom all living entities have come into being, and by whom the whole universe is pervaded. By such performance of work, a person easily attains perfection.
But eventually, the knower of Brahman alone is a true Brahmin. And a birth in a Brahmin clan nor the sacred thread nor Sandhyavandan alone guarantee the knowledge of Brahman especially in Kaliyuga and in Yugas except Satyuga, when the Dharma has deteriorated and a mixed varna society is functioning. Satyakama Jabala, Visvamitra attained Brahminhood despite being born in mixed caste, Kshatriya respectively. When the entire creation is working on Karmas, than how can one believe that mere birth in a Brahmin clan guarantees superiority and real Brahminhood? Hence real brahminhood has to be earned through real Karmas not inherited.
' Janmanaa jayate shudraha sanskarad dvij ucchyate veda pathhatbhavet vipraha brahma janati brahmana ' (Atri Smriti, 141)
By birth everyone is a shudra. After following rituals, one becomes a dvij. After knowing the scriptures, one becomes a vipra and only after attaining the knowledge of brahman(God) does one becomes a brahmin.
Srimad Bhagavatam The Degraded Dynasties and Corrupt Nature of the Rulers of Kali-yuga
(1) S'rî S'uka said: 'And then, oh King, under the strong influence of the time [of Kali-yuga] religiousness, truthfulness, cleanliness, tolerance and mercy as also duration of life, physical strength and memory will diminish day after day [see also 1.16]. (2) Among the people of the age of Kali all one attains by good birth, behavior and qualities will be wealth while material power will be the only factor determining what is just and reasonable. (3) Relations between men and women will be based upon sensual pleasure, business will be ruled by deceit, masculinity and femininity are there for sexual attraction and a sacred thread will suffice to be considered learned Brahmin. (4) External marks only determine someone's spiritual status and constitute the basis for mutual exchanges, because of a lack of funds one is less credible and scholarship consists of word jugglery.
The Varna System is more Guna based than Birth based in the Kaliyuga because of the dominant Mleccha societies and weakened Vedic social structure, intermixing and lack of Dharma because of the effects of Kali.
' Braahmanosya mukhmaasita, baahu raajanya krita. Uru tadasya yadvaishyaha padabhyaam shudro ajaayata. ' " From his(God) mouth emerged Brahmins(priests), from shoulders emerged Kshatriyas(warriors or governors), from thighs emerged Vaishyas(traders, businessmen, agriculturist) and from his feet emerged Shudras(serving class) " (Purushsukta -Rig Veda)
These parts of Brahman translates to Nirguna Purusha and Triguna Prakriti eventually, where Brahmin = Nirguna Purusha(head/Wisdom & Moksha)
Kshatriya = Satva Guna (hands/Protection & Dharma)
Vaishya = Rajas Guna (stomach/Reproduction & Artha)
Shudra = Tamas Guna (feet/Physical work & Kama)
One involved in the temporary material sense-gratification is a Shudra(low wisdom) anyways and one in the monkhood and renunciation is a Brahmin(high wisdom). The Satyuga to Kaliyuga takes the majority population on earth from Brahminhood to Shudra instincts and vice-versa in current ascension from Kaliyuga to Satyuga in a Yuga motion of 24000 years as explained by Sri Yukteshvara Giri. Even Adi Shankaracharya had to learn that the same Aatma/soul is present even in a Chandala and a Brahmin.
Thus the talk went on, and gradually drifted to the topic of Shankaracharya. The disciple was a great adherent of Shankara, almost to the point of fanaticism. He used to look upon Shankara's Advaita philosophy as the crest of all philosophies and could not bear any criticism of him. Swamiji was aware of this, and, as was his wont, wanted to break this one - sidedness of the disciple.
Swamiji: Shankara's intellect was sharp like the razor. He was a good arguer and a scholar, no doubt of that, but he had no great liberality; his heart too seems to have been like that. Besides, he used to take great pride in his Brahmanism -- much like a southern Brahmin of the priest class, you may say. How he has defended in his commentary on the Vedanta - sutras that the non - brahmin castes will not attain to a supreme knowledge of Brahman! And what specious arguments! Referring to Vidura he has said that he became a knower of Brahman by reason of his Brahmin body in the previous incarnation. Well, if nowadays any Shudra attains to a knowledge of Brahman, shall we have to side with your Shankara and maintain that because he had been a Brahmin in his previous birth, therefore he has attained to this knowledge? Goodness! What is the use of dragging in Brahminism with so much ado? The Vedas have entitled any one belonging to the three upper castes to study the Vedas and the realisation of Brahman, haven't they? So Shankara had no need whatsoever of displaying this curious bit of pedantry on this subject, contrary to the Vedas. And such was his heart that he burnt to death lots of Buddhist monks -- by defeating them in argument! And the Buddhists, too, were foolish enough to burn themselves to death, simply because they were worsted in argument! What can you call such an action on Shankara's part except fanaticism? But look at Buddha's heart! Ever ready to give his own life to save the life of even a kid -- what to speak of "[(Sanskrit)]-- for the welfare of the many, for the happiness of the many"! See, what a large - heartedness -- what a compassion!
Disciple: Can't we call that attitude of the Buddha, too, another kind of fanaticism, sir? He went to the length of sacrificing his own body for the sake of a beast!
Swamiji: But consider how much good to the world and its beings came out of that 'fanaticism' of his -- how many monasteries and schools and colleges, how many public hospitals and veterinary refuges were established, how developed architecture became -- think of that. What was there in this country before Buddha's advent? Only a number of religious principles recorded on bundles of palm leaves -- and those too known only to a few. It was Lord Buddha who brought them down to the practical field and showed how to apply them in the everyday life of the people. In a sense, he was the living embodiment of true Vedanta.
Verse 3.8.10: yo vā etadakśaraṃ gārgyaviditvāsmiṃlloke juhoti yajate tapastapyate bahūni varṣasahasrāṇi, antavadevāsya tadbhavati; yo vā etadakśaraṃ gārgyaviditvāsmāllokātpraiti sa kṛpaṇaḥ; atha ya etadakśaraṃ gārgi viditvāsmāllokātpraiti sa brāhmaṇaḥ ॥ 10 ॥
- He, O Gārgī, who in this world, without knowing this Immutable, offers oblations in the fire, performs sacrifices and undergoes austerities even for many thousand years, finds all such acts but perishable; he, O Gārgī, who departs from this world without knowing this Immutable, is miserable. But he, O Gārgī, who departs from this world after knowing this Immutable, is a knower of Brahman.