These "colors" are to be understood metaphorically or allegorically as Gunas (qualities). As mentioned in this answer, white represents Satva, Red represents Rajas and Black represents Tamas.
Furthermore, Bibek Debroy who translated the BORI critical edition of Mahabharata says the same in his translation.
ब्राह्मणानां सितो वर्णः क्षत्रियाणां तु लोहितः |
वैश्यानां पीतको वर्णः शूद्राणा मसितस्तथा ||५||
The complexion of brahmanas was white, while that of kshatriyas was
red. The complexion of vaishyas was yellow, while that of shudras was
black.
In Footnotes, he says
It is possible that this is meant to be taken figuratively and not
literally. Brahmanas possessed the sattva quality, kshatriyas
possessed the rajas quality, vaishyas possessed a combination of
sattva and rajas and shudras possessed the tamas quality.
In succeding lines Bharadvaja Muni asks, how Varna (complexion) alone decides distinction?
भरद्वाज उवाच||
चातुर्वर्ण्यस्य वर्णेन यदि वर्णो विभज्यते | सर्वेषां खलु वर्णानां
दृश्यते वर्णसङ्करः ||६||
“Bharadvaja said, ‘If the distinction between the four varnas is only
on the basis of complexion and that is how the varnas are to be
differentiated, then it is evident and can be seen that among the
varnas, there has been a mixture of varnas.
कामः क्रोधो भयं लोभः शोकश्चिन्ता क्षुधा श्रमः | सर्वेषां नः प्रभवति
कस्माद्वर्णो विभज्यते ||७||
Desire, anger, fear, avarice, sorrow, anxiety, hunger and exhaustion
influence everyone. How can varnas be differentiated on the basis of
this?
To this, Brigu Muni explains how Dvaijas (Dvi = two ; Ja = related to birth : Twice born, translated as Brahmanas) changed their Varnas by changing their Gunas (turned to other Varnas by their Karma).
भृगुरुवाच||
न विशेषोऽस्ति वर्णानां सर्वं ब्राह्ममिदं जगत् |
ब्रह्मणा
पूर्वसृष्टं हि कर्मभिर्वर्णतां गतम् ||१०||
Bhrigu replied, ‘There is no special difference between the varnas.
Everything in this universe first consisted of brahmanas. Brahma
created all of them earlier and they attained varnas because of
their deeds.
कामभोगप्रियास्तीक्ष्णाः क्रोधनाः प्रियसाहसाः |
त्यक्तस्वधर्मा
रक्ताङ्गास्ते द्विजाः क्षत्रतां गताः ||११||
There were Dvijas (brahmanas) who loved desire and objects of pleasure.
They were fierce and angry and loved courage. They abandoned their own
dharma and having turned red in their limbs, became kshatriyas.
गोषु वृत्तिं समाधाय पीताः कृष्युपजीविनः |
स्वधर्मं नानुतिष्ठन्ति
ते द्विजा वैश्यतां गताः ||१२||
There were Dvijas (brahmanas) who earned a living from animal husbandry and
subsisted on agriculture. They did not follow their own dharma, turned
yellow and became vaishyas.
हिंसानृतप्रिया लुब्धाः सर्वकर्मोपजीविनः |
कृष्णाः
शौचपरिभ्रष्टास्ते द्विजाः शूद्रतां गताः ||१३||
There were Dvijas (brahmanas) who loved violence and falsehood. They were
avaricious and turned to all kinds of deeds to earn a living. They
were dislodged from purity, turned black and became shudras.
In this way, depending on their deeds, brahmanas became other
varnas.
More importantly, Shvetashvatara Upanishad of Krishna Yajurveda defines Prakriti as Red-white-Black which are interpreted as Rajas-Satva-Tamas.
अजामेकां लोहितशुक्लकृष्णां बह्वीः प्रजाः सृजमानां सरूपाः
। अजो
ह्येको जुषमाणोऽनुशेते जहात्येनां भुक्तभोगामजोऽन्यः ॥ ५॥
There is one unborn prakriti−red, white and black−which gives
birth to many creatures like itself. An unborn individual soul
becomes attached to it and enjoys it, while another unborn individual
soul leaves it after his enjoyment is completed.
Adishankara says
Prakriti (nature) creates many creatures like itself which are
governed by three Gunas - sattva, rajas and tamas - called red, white
and black in this Mantra.
Furthermore while explaining Brahmasutras, Adi Shankara says,
For, he says, we have the following mantra (Sve. Up. IV, 5), 'There is
one agâ 1, red, white, and black, producing manifold offspring of the
same nature. There is one agâ who loves her and lies by her; there is
another who leaves her after having enjoyed her.'--In this mantra the
words 'red,' 'white,' and 'black' denote the three constituent
elements of the pradhâna. Passion is called red on account of its
colouring, i.e. influencing property; Goodness is called white,
because it is of the nature of Light; Darkness is called black on
account of its covering and obscuring property. The state of equipoise
of the three constituent elements, i.e. the pradhâna, is denoted by
the attributes of its parts, and is therefore called red-white-black.
Vajrasuchika Upanishad of Sama Veda says Brahmana is the one who has attained self realization and it also says Varna doesn't depend on color, birth or body.
tarhi deho brāhmaṇa iti cet tan na | ācaṇḍālādi padantānām manuṣyāṇām
pañcabhautikatvena dehasaikarūpatvāt |
jarā-maraṇa-dharmādharmādi-samya darśanāt |brāhmaṇas śveta-varṇaḥ |
kṣatriyo rakta-varṇaḥ | vaiśya pīta-varṇaḥ | śūdraḥ kṛṣṇa varṇa iti
niyamābhāvāt | pitrādi dahane putrādīnām brahma hatyādi
doṣasambhavācca | tasmān na deho brāhmaṇa iti || 5 ||
Then is the body the Brahmin? No, it is not so, because the body which
is composed of the five elements, is the same in all classes of human
beings down to the chandalas (outcastes), etc. And it is also observed
that old age and death, virtue [dharma] and vice [adharma] are found
to be common to all human beings. There is also no absolute
distinction (in the complexion of the four classes) that the Brahmin
is of the white complexion, that the Kshatriya is of the red
complexion, that the Vaishya is of the tawny complexion, that the
Sudra is of the dark complexion. [If the body is the brahmin] the
sons and other kinsmen would becoming guilty of the murder of a
Brahmin and other (sins) on cremating the bodies of their fathers and
other kinsmen. Therefore the body is not the Brahmin.
So, these colors represent Gunas.
