Why do Vedas, which are authorless and eternal, contains something like caste system which divides people on the basis of birth and even if they divide people on the basis of guna or karma why there is restriction on them for reading the vedas, performing sacred rituals etc. Isn't something which is eternal and authorless and whose goal was to tell the humans to realize the truth and become free(attain moksha) should be open to all humans?
Why do Vedas, which are authorless and eternal
They are not authorless and eternal. The Vedas are authored by the rishis based on their spiritual experiences. This is said by the rishis themselves in the Vedas. For example:
RV 1.61.4: asmā idu stomaṃ saṃ hinomi rathaṃ na taṣṭeva — “For him, I design this hymn, just as a carpenter designs a chariot”
RV 1.94.1: imaṃ stomamarhate jātavedase rathamiva saṃ mahemā manīṣayā — “This stotra we make for the most sacred Jātaveda with deep meditation, just like building a chariot”
Rig Veda Book 4 Hymn 26 and in Brihad-aranyaka Upanishad (1.4.10): Seeing this, the sage Vamadeva repeated at every moment: "I was Manu. I was the sun-god."
contains something like caste system which divides people on the basis of birth and even if they divide people on the basis of guna or karma why there is restriction on them for reading the vedas, performing sacred rituals etc.
Because based on past merits and sins, they have the right to perform some duties whereas some do not have that right:
Chandogya Upanishad V.10.7 - Those whose conduct here has been good will quickly attain a good birth (literally womb), the birth of a brahmin, the birth of a Ksatriya or the birth of a Vaisya. But those whose conduct here has been evil, will quickly attain an evil birth, the birth of a dog, the birth of a hog or the birth of a chandala.
A person who kills a holy man (Brahmin), or steals his property and gold, will be born as a Chandala in a future life. His punishment will be that he is obligated to work in the graveyard, which is the duty of chandalas. Contrarily, someone who does good deeds will be born as a Brahmana, who is required to teach the Vedas.
So based on past merit and sins, one is rewarded pure and impure work. This is just like how in the modern jail system, criminals have to do community service work by picking up trash.
whose goal was to tell the humans to realize the truth and become free(attain moksha) should be open to all humans?
As a matter of fact, moksha is open to all castes, but the path to moksha is different.
Bhagavad Gita 9.32 - O son of Pṛthā, those who surrender to me, even though they are of sinful birth, women, vaishyas, or shudras, they will still attain the supreme goal (moksha).
Bhagavad Gita 18.66 - After abandoning all the other means of liberation, surrender only to me. I will release you from all sins.
So moksha is open to all.
There is no caste system in the Vedas. The varna system of the Vedas is based on conduct. This is clear from the definition of chandalas in Chandogya Upanishad.
The Smriti definition of a Chandala is that such a person is born of a Sudra father and a Brahmana mother. This fanciful Smriti idea is clearly wrong and is against the Vedas. This is clear from the Chandogya Upanishad definition of Chandala.
Those whose conduct here has been good will quickly attain a good birth (literally womb), the birth of a brahmin, the birth of a Ksatriya or the birth of a Vaisya. But those whose conduct here has been evil, will quickly attain an evil birth, the birth of a dog, the birth of a hog or the birth of a chandala.
Chandogya Upanishad V.10.7
As you can see Chandogya Upanishad is specifying conduct as the reason for different births. It has nothing to do with the caste of the father or the mother.
What does the Chandogya Upanishad verse mean?
It has nothing to do with the caste system invented by the Smriti writers. It means that depending on the amount of good conduct a person will be born to parents of high Sattva Guna (Brahmin), to parents of middling Sattva Guna (Ksatriya) or parents of high Rajasa Guna mixed with some Sattva Guna (Vaisya). The vast majority of persons with neither good nor bad conduct will be born of parents who do not have the gunas of Brahmanas or Ksatriyas or Vaisyas but whose conduct is not as bad as that of animals. Those whose conduct has been exceptionally bad without any redeeming feature will have animal parents or human Chandala parents. This suggests that Chandala is defined as a person whose conduct is like an animal.
Another example that shows that the Varna system is not the caste system is given below.
One day a young boy came to the ashrama of Sage Haridrumata Gautama and said, “Revered Sir, I desire to live under you as a Brahmacharin. Please accept me as your student.”
The sage asked, “Dear boy, of what gotra or lineage are you?”
The boy replied, “Sir, I do not know of what gotra I am. I asked it of my mother. She said: ‘I also do not know of what gotra you are. I used to serve many people and I got you in my youth. So I am not sure of what lineage you are. However, I am Jabala by name and you are Satyakama’. So, Sir, I reveal myself to you as Satyakama Jabala.”
On hearing it, the Rishi Haridrumata Gautama smiled and said, “No one who is not a Brahmin can speak thus. Dear boy, bring the sacrificial fuel. I shall initiate you as a Brahmacharin, for you have not deviated from truth.” Thus was Satyakama Jabala initiated into the life of a Brahmacharin.
Chandogya Upanishad 4.4-4.8
The boy does not even know who his father is. It is from his conduct that the sage deduces that he is a Brahmin.
Why were Shudras not allowed to learn about the Vedas?
This happened when the Varna system of the Vedas where the varna is dependent on conduct morphed into the caste system where an entire endogamous group was identified with one Varna. Votaries of this caste system claim that they can identify the varna of a 7 year child simply from his last name! This odd interpretation of the Vedas changed the conduct based Varna system to the jati system where Varna is determined from the last name. Once Hindus began to follow this unreasonable system only communities which were arbitrarily classified as Brahmins, Kshatriyas and Vaishyas were allowed to learn the Vedas.
Criticism of the caste system in Hindu scripture
There has been criticism of the caste system in Hindu scripture.
(1) Yudhisthira said, " In human society, O mighty and highly intelligent serpent, it is difficult to ascertain one's caste, because of promiscuous intercourse among the four orders. This is my opinion. Men belonging to all orders (promiscuously) begat offspring upon women of all the orders. And of men, speech, sexual intercourse, birth and death are common. And to this the Rishis have borne testimony by using as the beginning of a sacrifice such expressions as -- of what caste server may be, we celebrate the sacrifice. Therefore, those that are wise have asserted that character is the chief essential requisite ..... Whatsoever now conforms to the rules of pure and virtuous conduct, him have I, ere now, designated as a Brahmana.'
(Mahabharata Aranya Parva Section CLXXIX)
(2)The sage Bhrigu said, ‘There is really no distinction between the different orders. The whole world at first consisted of Brahmanas. Created (equal) by Brahman, men have, in consequence of their acts, become distributed into different orders. They that became fond of indulging in desire and enjoying pleasures, possessed of the attributes of severity and wrath, endued with courage, and unmindful of the duties of piety and worship, - these Brahmanas possessing the attributes of passion, - became Kshatriyas. Those Brahmanas again who, without attending to the duties laid down for them, became possessed of both the attributes of Goodness and Passion, and took to the professions of cattle-rearing and agriculture, became Vaisyas. Those Brahmanas again that became fond of untruth and injuring other creatures, possessed of cupidity, - engaged in all kinds of acts for a living, and fallen away from the purity of behaviour, and thus wedded to the attributes of Darkness, became Sudras. Separated by these occupations, Brahmanas, falling away from their order, became members of the other three orders. All the four orders, therefore, have always the right to the performance of all pious duties and of sacrifices. Even thus were the four orders at first created equal by Brahman who ordained for all of them (the observances disclosed in) the words of Brahma (in the Vedas). Through cupidity alone, many fell away, and became possessed by ignorance. The Brahmanas are always devoted to the scripture on Brahma; and mindful of vows and restraints, are capable of grasping the conception of Brahma. Their penances therefore never go for nothing. They amongst them are not Brahmanas that are incapable of understanding that every created thing is Supreme Brahma.
(Mahabharata Santi Parva Section CLXXXVIII )
(3) The famous Rishi Yajnavalkya continued,’ .. All orders of men are Brahmanas. All are sprung from Brahma. All men utter Brahma. Aided by an understanding that. I inculcated this science teaching of Prakriti and Purusha. Indeed this whole universe is Brahma. From the mouth of Brahma sprung the Brahmanas; from his arms, sprung the Kshatriyas; from his navel, the Vaisysa; and from his feet, the Sudras. All the orders, (having sprung in this way) should not be regarded as pilfering from one another. Impelled by ignorance, all men meet with death and attain, O King, to birth that is the cause of acts. Divested of knowledge, all orders of men, dragged in terrible ignorance, fall into varied orders of being due to the principles that flow from Prakriti. For this reason, all should, by every means, seek to acquire knowledge. I have told thee that every person is entitled to strive for its acquisition. One that is possessed of knowledge is a Brahmana. Others, (viz., Kshatriyas, and Vaishyas and Sudras) are possessed of knowledge. Hence this science of emancipation is always open to them all.
(Mahabharata Santi Parva Section CCCXIX)
Yudhisthira says that it is difficult to ascertain a man's caste given widespread intercaste marriages. This directly shoots down the claim that varna of a person can be determined by heredity. The sage Bhrigu says that people who are not capable of understanding that all created beings are Supreme Brahma can not be called Brahmanas. The famous Rishi Yajnavalkya says ,’ .. All orders of men are Brahmanas. All are sprung from Brahma. All men utter Brahma. ' He also recommends that all should acquire kowledge by every means, i.e., Vedas should be made available to all. All these men clearly would disagree with the fanciful positions of Smritis if they were asked today.
The correct reason why some people should be denied access to the Vedas
There is a more reasonable explanation of why certain people should not be allowed to learn the Vedas if we reject the caste system and go back to the conduct based Varna system.
Teachings only to good persons
These teachings are not to be imparted to persons who are wicked, haughty, barbarous, foul in conduct, hypocritical, given to sensuous indulgences, interested exclusively in their worldly concerns, who are without devotion to Me or who persecute My devotees.
Srimad Bhagavata Purana III.32.39-40
Only the people who are not seriously interested in the Vedas and whose conduct is foul should not be allowed to learn the Vedas.
Of course in today's world when the Vedas have been translated and are available to the entire world even such a position cannot be supported and in fact cannot be even enforced.
There is no caste system in Vedas, Although one would find multiple references to the Varna system, which was a backbone of the Vedic society. Students were assigned subjects best suited for their intellect by their teachers, who themselves were of high integral and moral character, has required knowledge of the shastras, these teachers would observe the activities of their student and recommend the best varna basis the student's personal acumen and preferences. It was a subjective approach, although towards the end of the Vedic age this was replaced by a system based on birth.
Vedas were transferred down the centuries by enlightened seers, who lived a certain pure lifestyle and their thoughts had the same purity and clarity intact. Such though process can't be expected from an average person living the mundane life of a householder, without the right training in language and thought process one won't be able to grasp the meaning of the shlokas and lead to confusion and misinterpretation. They are open to all, but how much one can comprehend is dependent on their maturity in these matters.
To support the accepted answer's Bhagavad Gita quotes and explanation on:
whose goal was to tell the humans to realize the truth and become free(attain moksha) should be open to all humans?
Apastamba Sutra says in 2.1.2:
sarva varṇānāṃ sva dharma anuṣṭhāne parama parimitaṃ sukham || 2 ||
2. People of all castes, if they fulfil their [assigned] duties, enjoy [in heaven] the highest, imperishable bliss.
By "highest, imperishable bliss" the verse means moksha. Therefore, moksha is open to all castes.