Has any Hindu scholar who upheld birth based Varna system refuted Assalayana Sutta?
Yes, the medieval Sri Vaishnava scholar Yamunacharya, who lived a hundred years before the time of Ramanujacharya, has refuted these arguments in his work called Agama Pramanya, or "Proof of the validity of the Pancharatra Agamas."
In it, the opponent claims that the followers of the Pancharatra Agamas are not Brahmanas, and therefore the Pancharatra Agamas are not authoritative. The opponent makes the same arguments as stated in the Assalayana Sutra. Yamunacharya refutes these arguments starting from verse 120 of the Agama Pramanya.
Here is the excerpt:
120. ... In view of the facts that the eminent Brahminhood of these Bhagavatas [ancient word for Vaishnavas] who follow the doctrine of the Bhagavan is evident by all criteria of knowledge, their acceptance of Pancharatra rather confirms its [the Agamas'] validity. He says: By the same means of knowledge by which the brahminhood of one set of people is evident, the brahminhood of another set of people is evident.
121. Objection [of the opponent]: But when one sees the small sons of the twice-born who wear the customary hairtuft, sacred thread, palasa wood stick and munja grass girdle, one knows, the moment the eyes fall on them, that they are brahmins.
Refutation [given by Yamunacharya]: And in our case, when one sees learned people who day after day study the Vajasaneyaka and Ekayana sakhas [now extinct sakha of the Shukla Yajur Veda that the Pancharatra Agamas are based on], wear prominently their sacred threads, upper garments and hairtufts, impart teaching, sacrifice, receive priestly stipends---does one not instantly know that they are brahmins? If it be held that outcastes, low-caste people, etc. may also illegitimately sacrifice, teach, carry palasa sticks, etc. and that they behave as though they were legitimate brahmins, and that therefore neither costume nor conduct provides positive certainty that a man is a brahmin, then the same applies to other priests than Bhagavatas.
Or be it granted that there are cases where people illegitimately display the marks of brahminhood; still though there may be doubt about the legitimacy of these marks in others because of their resemblance to pretenders (just as when there is a doubt that one may be mistaken about real silver too because one has mistaken nacre for silver), then there can be certainty of their genuine brahmins in all cases when no misapprehension occurs, because otherwise doubt would conflict with perception and lead to infinite regress.
122. Or it if be held that the others [non-Vaishnava brahmins] are genuine brahmins because they recollect those gotras which are peculiar to brahmins, the same applies to the Bhagavatas; for the Bhagavatas have the tradition "We are descendants of Bharadvaja, of Kasyapa, of Gautama, of Upagava."
Nor is this recollection of tradition of gotras unfounded or merely contemporary, for the same can be argued for all tradition of gotras. If there were doubt about descent since error could conceivably occur, this would confuse the whole world about the authenticity of their brahminhood. After all, anyone may fear that he really is a chandala if he suspects his mother of having had a lover; and how, my excellent opponent, can you be quite sure yourself that your birth entitled you to Veda-study? Therefore if the brahminhood of Bhagavatas, which is completely established by the recollection of the various gotras which has been passed on in uninterrupted transmission, stands unchallenged, then there is no difference whatever in this between the brahminhood of Bhagavatas and of others.
123...There remains a criterion to determine brahminhood in either case, either perception or inference or circumstantial-implication.
124. Objection. But how can perception convey that they are brahmins? For when we are close to two individuals whom we have not seen before, one a brahmin, the other a kshatriya, of the same age and the same appearance, we do not immediately observe the distinction that one is a brahmin and the other a kshatriya in the same way as we instantly observe the differentness of a goat, an elephant, a buffalo, etc. Nor is it proper to maintain that the visual sense conveys the brahminhood of a nearby individual in dependence on our recollection of his father's brahminhood etc.; for that recollection itself is impossible without a previous immediate cognition, just as the recollection of the son of a sterile mother is impossible. Nor can we know from inference that a man is a brahmin, for we do not find a concomitant mark. And such qualities as tranquility, self-restraint, austerity, purity etc., cannot be taken as marks of brahminhood, because they are available only in the case of a good brahmin and because they are not exclusively confined to brahmins. Nor can circumstantial-implication furnish proof of brahminhood, because it is not lacking in season and the fact that the sentence-meaning of the statement, "In spring the brahmin must add fuel to the fire," is otherwise unestablished does not therefore by circumstantial-implication furnish proof that a man is a brahmin; for knowledge of that sentence meaning presupposes knowledge of the word-meaning of brahminhood etc.
125. Refutation [given by Yamunacharya.] All this does not make for a defect in my argument. there is no invariarble rule that perception becomes manifest only at the first contact between sense and object and not otherwise. Perception is that which illumines the unmanifest while there is continuity of the operation of the senses. Thus, there can be perception of brahminhood;..... "What are the specific characteristics to which the elders apply the term brahminhood, or to which characteristics is the term applied?" It has been said often that it applies to those who possess recollection of gotra, Vedic ancestry, etc.; .... it is established that the Bhagavatas are brahmins, because they possess gotra, etc.
So, what Yamunacharya is saying is that Brahmins are known from their conduct and appearance, and especially their recollection of gotras. While it is possible that a low-caste man can have the conduct and qualities of a brahmana, or deceitfully dress like a brahmana and unlawfully perform rituals, or lie about his gotra, it does not refute the fact that brahminhood can be known from perception, because if one doubts that perception on the grounds that that perception can be erroneous, then one must doubt all perception on the grounds that perception itself can be erroneous.
If it looks like a brahmin, talks like a brahmin, and walks like a brahmin, then the natural idea is that it is a brahmin, unless there is evidence to the contrary.