If we say varnas can be determined entirely neglecting the birth factor then that will clearly contradict the Purusha Suktam which says how the four castes came out of (or born of) four parts of the Purusha's body.
brāhmaṇo'sya mukhamāsīd bāhū rājanyaḥ kṛtaḥ, ūrū tadasya yad vaiśyaḥ padbhyāgï śūdro ajāyata.
The Brahmana (spiritual wisdom and splendour) was His mouth; the Kshatriya (administrative and military prowess) His arms became. His thighs were the Vaisya (commercial and business enterprise); of His feet the Sudra (productive and sustaining force) was born.
So, the varnas very much depend on how they were born to start with. Then how can we expect that varnas can be decided by entirely neglecting the birth factor?
For example, the Sudras are the ones who were born out of the Purusha's feet. So, the the birth factor is the first deciding factor for determining the varnas. Then how can we ignore it?
Then of course come the Samskara and the conduct factors. Because the Scripturesscriptures say that everyone is a Sudra by birth; & one becomes a Dvija only by Samskaras.
See, the following verse from Manu Smriti, that says birth, samskara (purificatory rituals) and conduct all collectively determine a Brahmin. The purport is all those three factors are important and we can't neglect anyone of them.
Manu Smriti 10.3. On account of his pre-eminence, on account of the superiority of his origin, on account of his observance of (particular) restrictive rules, and on account of his particular sanctification the Brahmana is the lord of (all) castes (varna).