It is said that the Brahmins originated from the face, Kshatriyas from Shoulders and hands, Vaishyas from thighs and Shudras from feet of Purusha.
Reference: This is definitely mentioned in Purushasukta but not sure about the Rigveda. I will update once I find out more on this.
brāhmaṇo'sya mukhamāsīd bāhū rājanyaḥ kṛtaḥ,
ūrū tadasya yad vaiśyaḥ padbhyāgï śūdro ajāyata.
The Brahmana (spiritual wisdom and splendour) was His mouth; the Kshatriya (administrative and military prowess) His arms became. His thighs were the Vaisya (commercial and business enterprise); of His feet the Sudra (productive and sustaining force) was born.
While it is fair to think it divides the society, the society need all four to survive or to thrive. The symbolism is pretty good primarily because it represents the roles the people play in society. Please remember: NOWHERE IT IS MENTIONED THAT THE VARNA SYSTEM IS HEREDITARY.
A human being would need all four organs to function is most efficient manner. It is not possible to give up on any of these without hindering it's proficiency isn't it?