The inbuilt assumption in Hinduism is that the system works. In other words, there is no difference between birth-based and guNa-based varNa. They are supposed to be the same. One's varNa is a reward or punishment for karmas done in previous births. The karmas done in previous births affect the guNas associated with the soul. The soul or AtmA is reborn into a varNa that is supposed to be consistent with its guNas. Thus, in this system, the Brahmanas should only be populated by souls that are steeped in sattva guNa; the Kshatriyas, similarly, only by those jIvas possessing AtmAs steeped in rajas; the Vaishyas, similarly, only by those jIvas possessing a mixture of rajas and tamas; and the Shudras, only by those jIvas dominated by tamas.
From the Bhagavad Gita, Chapter 14, Verse 14, Krishna says
http://www.bhagavad-gita.org/Gita/verse-14-14.html (all the verses I am quoting, translations, and commentaries are derived from this same website)
yadA sattvE pravRuddhe tu pralayam yAti dEha-bhRut tadottama-vidAm lokAn amalAn pratipadyatE
When the embodied being transitions death, predominated by the mode of goodness – at that time, the pure worlds with the highest knowledge are attained.
As Ramanujacharya explains it,
"Lord Krishna explains that when the jIva, or embodied being, is in sattva guNa, or mode of goodness, at the time of death, then numerous blessed regions, where the knowers of AtmA tattva, or knowledge of the immortal soul, reside, are reached. The word amalAn means pure, exempt from blemish, and devoid of ignorance. The understanding is that the fate of a jIva, who dies under the influence of sattva guNa, would be that the next birth would take place in the environment of pious and holy people that have realised AtmA tattva; and so, combined with an inner impulse to pursue this way of life, and with opportunity in abundance, one would continue in sattva guNa, purifying one's existence by performing spiritual activities, until one achieves perfection."
Similarly, in 14-15, Krishna says that
rajasi pralayam gatvA karma-sangiShu jAyatE tathA pralInas tamasi mUDha-yOniShu jAyatE
Succumbing to death in the mode of passion, one takes birth among those beings attached to fruitful activities; similarly, dying in the mode of ignorance, one takes birth from the womb of an animal.
As Shridhara Swami explains it:
"Continuing, Lord Krishna explains that the result of raja guNa, or mode of passion, being predominant at the time of death is that the jIva, or embodied being, is born into families that are involved in vigorous activity. Likewise, the result of tama guNa, or mode of ignorance, being predominant at the moment of death is that the jIva is born in the wombs of irrational species as animals."
Similarly, Ramanujacharya explains this verse thus:
"Now Lord Krishna explains that when a jIva, or embodied being, dies while in raja guNa, or the mode of passion, one takes birth among those who perform activities out of desire for rewards; and having been born as such, will endeavour mightily to accomplish works that would be rewarded by immense wealth and power, even up unto the heavenly spheres. Similarly, if one dies while under the influence of tama guNa, or the mode of ignorance, then such jIvas will be born from the wombs of animals such as pigs and dogs, where one would be completely incapacitated to perform any activity that would give any spiritual benefit."
Likewise, Acharya Keshava Kashmiri explains this verse thus:
"Lord Krishna states that if death comes upon a jIva, or embodied being, while predisposed to raja guNa, or the mode of passion, then one takes birth next from those who are obsessed to perform actions in anticipation of rewards. Also, a jIva dying while under the influence of tama guNa, or mode of ignorance, is forced to take birth as an animal in the next life."
And Shankaracharya explains the verse thus:
"If he meets death when rajas is predominant, he is born among men, who are attached to action. The “irrational” refers to cattle and the like."
("mUDha-yOnishu" means "womb of irrational being," i.e., animal.)
So one's varNa in a future birth (as a sattva-dominant, rajas-dominant, or tamas-dominant soul is determined by the guNa in which one dies. This guNa is the result of one's actions in millions of past births. As is seen in Chapter 18, those who are steeped in sattva guNa are the Brahmana varNa; those who are steeped in rajas guNa are the Kshatriya varNa, and those who are steeped in tamas guNa are the Shudra varNa.
For instance, 18-42 says about the Brahmana varNa:
shamO damas tapaha shaucam kshAntir Arjavam Eva ca jnAnam vijnAnam Astikyam brahma-karma svabhAva-jam
The actions of a Brahmana, arising from his own nature, are serenity, self-control, austerity, purity, tolerance, honesty, knowledge of the Vedas, wisdom, and firm faith.
Note that this arises from birth ("arising from his own nature"), not nurture. This is the link between guNa and varNa.
So why doesn't this system seem to correspond to reality?
Well, when there is a contradiction, one of the assumptions must be wrong. Maybe the system is wrong?