First of all, chapter 4 of the Manu Smriti doesn't say that only the Yajur Veda is meant for humans. All the Vedas are meant to be chanted among men, it's just a question of whom each Veda is sacred to:
- Let him never recite the Rig-veda or the Yagur-veda while the Saman (melodies) are heard; (let him stop all Veda-study for a day and a night) after finishing a Veda or after reciting an Aranyaka.
Let him never recite the Rig-veda or the Yagur-veda while the Saman (melodies) are heard; (let him stop all Veda-study for a day and a night) after finishing a Veda or after reciting an Aranyaka.
The Rig-veda is declared to be sacred to the gods, the Yagur-veda sacred to men, and the Sama-veda sacred to the manes; hence the sound of the latter is impure (as it were).
- The Rig-veda is declared to be sacred to the gods, the Yagur-veda sacred to men, and the Sama-veda sacred to the manes; hence the sound of the latter is impure (as it were).
That is to say, the Rig Veda is concerned with praise of the gods, the Yajur Veda is concerned with the details of Yagnas that benefit Man, and the Samaveda consists of songs some of which are used to venerate the spirits of ancestors.
Now far as whether the Sama Veda is impure, the word impure is just being used metaphorically, in the sense that customs regarding the Sama Veda are similar to customs regarding impure things. Here is what Medhatithi says in his commentary on the Manu Smriti:
It is not really impure, but when it is heard, one must not study, just as in the presence of some impure thing or person.
