This question already has an answer here:
Is there an inherent relation between varna (caste) and skin color? By inherent I mean does scripture say that castes were created with a particular skin color? The scriptures say that kshatriyas were created strong, brahmanas pious, etc. But what about skin color and physical appearance?
The Apastamba Dharma Sutras say something regarding this:
184.108.40.206. A Brāhmaṇa atones for the offence which he has committed by serving one day and night [a man of] the black race [krishna varna], if he bathes for three years, eating at every fourth meal-time.
"Krishna varna" refers to the 4th caste, Shudras.
But this doesn't necessarily have to be talking about their physical appearance, because black in color could be referring to the inherent guna of shudras, which is tamas.
Georg Buhler, the translator, says:
The expression krishna varna, 'the black race,' is truly Vedic. In the Rig-veda it usually denotes the aboriginal races, and sometimes the demons.