Rakshas definitely don't fall under the four varna system . So, What is the story of creation of Rakshas ?
As per Valmiki Ramayana: UTTARA KANDA: Sarga 4, Lord Brahama created creatures to protect water and those creatures came to be known as Rakshasas and Yakshas:
The lord of creatures sprung from water, first created water. And the lotus born one generated creatures for protecting that element. And thereupon those creatures humbly presented themselves before the creator, saying, What shall we do ?We are sore tried by hunger and thirst. The lord of creatures, laughing, addressed them, saying, Ye men, do ye carefully guard this (water). Thereat some said Rakshama, and others Yakshama.
Thus accosted by those afflicted with hunger and thirst, the creator said, Those among you who have said Rakshama, shall be called Rakshasas and those among you who have said Yakshama shall be called Yakshas. Among the Rakshasas, who thus guarded the sacred waters, were two brothers named Hoti and Prahoti, mighty as they were, like Madhu and Kaitabha. Those brave brothers became the leaders of the Rakshasas.
- Praheti went in forest for tapsya. Heti got married to Kala's sister Bhaya. Heti's son was Vidhyutkesha.
- Sukesha (who was abandoned by his mother soon after his birth but later was blessed by Lord Shiva and Paravati) was son of Vidhyutkesha.
- Sukesha's sons were Malyavan, Sumali and Mali.
- Malyavan's children: Vajramushti, Virupaksha, Durmukh, Suptaghana, Yaghyakopa, Matta, Unamnta, Analaa
- Sumali's children: Prahasta, Akampana, Viakata, Kalikamukha, Dhumraksha, Danda, Suparshva, Sanghadi, Praghasa, Bhaskarna, Kumbhinashi, Kaikashi (Ravana's mother), Rakaa, Pushpotaktaa
- Mali's children: Anala, Anila, Hara, Sampati (later they became ministers of Vibhishana)
Malyavan, Sumali and Mali as rulers of Rakshasas and Lanka: Lanka was built by Vishvakarma. Malyavan, Sumali and Mali used to live in Lanka. Later they attacked Devas and Lord Vishnu came to fight with them. Lord Vishnu killed Mali. Seeing Mali slain, Sumali and Malyavan, burning in grief, led precipitately towards Lanka along with their forces. But Lord Vishnu didn't stop killing Rakshas.
Kubera became king of Yakshas and Lanka: Later, they, incapable of coping with Vishnu, leaving Lanka, went to the nether regions (Rasatala) with their wives for dwelling there. Then Kubera started living in Lanka.
Ravana became king of Rakshasas and Lanka: Later Sumali encouraged his daughter Kaikesi to marry Visrava. Coming to know that his daughter's sons, Dasanana and his two brothers, got powerful boons (Dasanana was incapable of being slain by Nagas, Yakshas, Daityas, Danavas and the Devas) form Lord Brahma, Sumali shed all his fear of Vishnu and came out from Rasatala. Later Sumali encouraged Ravana to take Lanka back from Kubera. Vishrva advised Kubera to leave Lanka and build another city name Alkapuri on Kailash mountain which Kubera accepted.
According to the Brahmand Puran there are different types of Rakshasas, to be precise, seven and their orign is indicated by their names such as Paulastyas, Agastyas, Kaushiks etc.
In his previous birth Rāvaṇa was Hiraṇyakaśipu. That Rākṣasa ruled as king for thirteen sets of four Yugas......Mahodara, Prahasta, Mahāpārśva and Khara were the sons of Puṣpotkaṭā. She had a daughter (named) Kumbhīnasī also Triśiras. Dūṣaṇa and Vidyujjihva the Rākṣasa are remembered as the children of Vākā. There was aḍaughter also named Anupālikā.
Thus these ten Rākṣasas, the descendants of Pulastya (CATEGORY 1), were of very cruel activities. They are of very terrible pursuits, ever since their birth. All of these were unassailable even to the Devas. All of them had acquired boons. They were heroic and they were accompanied by sons and grandsons.
Aiḍaviḍa (i.e. Kubera), the descendent of Pulastya, who was Savyapiṅgala (? Tawny-coloured in the left side) became the king of all Yakṣas, all those Rākṣasas who were the descendants of Pulastya, and of all those cruel Brahmarākṣasas belonging to the families of Agastya and Viśvāmitra who still carried on the study of the Vedas and who performed regularly penances and holy rites.
The other three groups of Rākṣasas were the performers of Yajñas. They were Yātudhānas, Brahmadhānas and Vārtās (CATEGORIES 2, 3 & 4). They were moving about during the day time and they were not nocturnal wanderers like other demons. Their four groups are remembered by the wise (learned men).
So it seems there were some Rakshas tribes that followed the Vedic injunctions and roamed around in the day time while others were night-dwellers and opposed to the Vedic teachings.
They were Niśācaras (moving about at night), They were Paulastyas (descendants of Pulastya), Nairṛtas, Āgastyas and Kauśikas (CATEGORIES 5, 6 & 7 Paulastyas were already covered). Thus it is remembered that there were seven classes (communities) of those Rākṣasas. I shall describe their form and features inherited by them naturally.
The next verses give a detailed description and it appears that irrespective of their origin the various groups had a lot of common features in their physical appearance as well as habits:
They are tawny-coloured and have round eyes. They have big bellies and huge bodies. They have eight-curved fanglike teeth and pikelike ears. Their hairs stand upright. Their mouths appear as though slit open upto the ears (on either side). Their smoke-coloured upright hairs are like the Muñja grass. They have large thick heads shedding bright lustre. Their thighs and forearms are short. Their faces are copper-coloured. Their tongues and lips hang down. Their brows appear dangling. Their noses are thick.
They have blue (black) limbs with red necks. Their eyes are majestic and they appear terrible. Their voice and tone are extremely terrible. Their calves are hideous and rolled up. They are stout and have prominently protruding noses. They are firm and have sturdy bodies like rocks. They arc cruel and terrible ever since their birth. Generally they are consistent in their activities.
They put on (ornaments like) earrings, armlets and coronets. They bedeck themselves in diverse kinds of ornaments. They wear many kinds of garlands and use all types of fragrant scents and unguents. They are remembered as habitual eaters of cooked rice, meat and even human flesh. Thus is remembered the similarity in forms and features of demons by learned men. They are unequalled in strength, intelligence and ability to fight using the deceptive power of Māyā (or black magic).