The animal avataras like fish,tortoise wouldn't have caste - but how about the human avataras?
Well, Vidura who is considered to be a Shudra, is considered an incarnation of Yama:
And there was a Rishi of the name of Animandavya of great fame. And he was conversant with the interpretations of the Vedas, was illustrious, gifted with great energy, and of great reputation. And, accused of theft, though innocent, the old Rishi was impaled. He thereupon summoned Dharma (Yama) and told him these words, 'In my childhood I had pierced a little fly on a blade of grass, O Dharma! I recollect that one sin: but I cannot call to mind any other. I have, however, since practised penances a thousandfold. Hath not that one sin been conquered by this my asceticism? And because the killing of a Brahmana is more heinous than that of any other living thing, therefore, hast thou, O Dharma, been sinful. Thou shalt, therefore, be born on earth in the Sudra order.' And for that curse Dharma was born a Sudra in the form of the learned Vidura of pure body who was perfectly sinless. (Mahabharata, Adi Parva, Section LXIII)