Root of caste system is human nature and misconception. Rig Veda 9.63 says:
इन्द्रं वर्धन्तो अप्तुर: कृण्वन्तो विश्वमार्यम् । अपघ्नन्तो अराव्णः ॥
This is a prayer to make ALL HUMANS Aryas. Veda makes a distinction between Shudra & Arya meaning bad people and noble people. Of course, birth is as per karma but this is a very clear message from the Veda itself that Varna can be changed! Now if we see Upanishads or Brahmanas we almost always get the same picture. Look at Chandogya Upanishad talks about the initiation story of another Upanishad's author:
Gautama asked him, ‘O Somya, what is your lineage?’ Satyakāma said: ‘Sir, I do not know what my lineage is. When I asked my mother, she said to me: “I was very busy serving many people when I was young, and I had you. As this was the situation, I know nothing about your lineage. My name is Jabālā, and your name is Satyakāma.” So, sir, I am Satyakāma Jābāla’. Gautama said to him: ‘No non-brāhmin could speak like this.
So the son of a highly impure shudra woman and unknown man was declared Brahmin. Now look at the story of Vedic author Vatsa found in Tandya Brahmana 14.6.6, Jaiminiya Brahmana 3.234:
The two sons of Kanva, Medhatithi and Trioka, contended about the sacred lore (brahman). They said : ' Come, let us cross the flaming fire'. They crossed the flaming fire. Tris'oka crossed over it (unharmed), but of the other it scorched the eye-lashes. Ho (Trisoka) said to him (to Medhatithi) : ' I have vanquished thee '. ' No ', said he, 'thou art the son of an Asura-mother ; even the deities have not wished to touch thee'. Then, they (said): 'Come, let us cross the water'. They crossed the flowing Rathaspa. Tris'oka crossed over it (unharmed), but of the other the rims of his cart were moistened by the water of the Rathaspa (translation uncertain, text corrupted !). He (TriSoka) said to him (to Medhatithi): 'I have vanquished thee'.
let us walk according to the rite through fire (to decide) which of us two is the better brahmin
So there was clear misconception and human nature at work here. But Agni preferred Shudra's son. Shudra's son was a superior Brahmin compared to birth-Brahmin. Another example from the Brahmanas is in Kausitaki Brāhmana 12.3 and Aitareya Brāhmana 8.19
"You are the son of a female slave. We will not eat with you" (dāsya vai tvam putro si na vayam tvaya saha bhaksayisyama iti). Kavasa became angry and ran away. He recited a hymn of praise to Sarasvati, who then followed after him: sarasvatim etena suktena tustava tam heyam iyaya. Realizing what had happened, the surprised seers hurried to him to express their reverence: "0 seer, homage be to you. Do not harm us. You indeed are the best of us...
Again we find human nature and misconception in this story. But Saraswati considers a Shudra's son superior to all birth-Brahmins combined at being a Brahmin. The birth-Brahmins also later accept they are inferior to him. He authored a part of Rig Veda. There are 108 canonical Upanishads as per Muktika Upanishad. One of them is Vajrasuchika Upanishad which says
It is said that a Brahmana is so because of his caste. This is not acceptable because there are diverse communities in the world... ...Among these many have attained the highest rank, despite of their lower birth and given proof of their wisdom. Therefore a Brahmana is not so because of his community.
Self-explanatory but people still try to twist and discard Vedic scripture because of human nature or misconception from semi-sacred Smritis. People who consider smriti main scripture are called Smartas and they are biggest caste lover. But most Hindus and all other Hindu scripture does not accept this absolutist view.