Rakshas definitely don't fall under the four varna system . So, What is the story of creation of Rakshas ?
As per Valmiki Ramayana: UTTARA KANDA: Sarga 4, Lord Brahama created creatures to protect water and those creatures came to be known as Rakshasas and Yakshas:
The lord of creatures sprung from water, first created water. And the lotus born one generated creatures for protecting that element. And thereupon those creatures humbly presented themselves before the creator, saying, What shall we do ?We are sore tried by hunger and thirst. The lord of creatures, laughing, addressed them, saying, Ye men, do ye carefully guard this (water). Thereat some said Rakshama, and others Yakshama.
Thus accosted by those afflicted with hunger and thirst, the creator said, Those among you who have said Rakshama, shall be called Rakshasas and those among you who have said Yakshama shall be called Yakshas. Among the Rakshasas, who thus guarded the sacred waters, were two brothers named Hoti and Prahoti, mighty as they were, like Madhu and Kaitabha. Those brave brothers became the leaders of the Rakshasas.
- Praheti went in forest for tapsya. Heti got married to Kala's sister Bhaya. Heti's son was Vidhyutkesha.
- Sukesha (who was abandoned by his mother soon after his birth but later was blessed by Lord Shiva and Paravati) was son of Vidhyutkesha.
- Sukesha's sons were Malyavan, Sumali and Mali.
- Malyavan's children: Vajramushti, Virupaksha, Durmukh, Suptaghana, Yaghyakopa, Matta, Unamnta, Analaa
- Sumali's children: Prahasta, Akampana, Viakata, Kalikamukha, Dhumraksha, Danda, Suparshva, Sanghadi, Praghasa, Bhaskarna, Kumbhinashi, Kaikashi (Ravana's mother), Rakaa, Pushpotaktaa
- Mali's children: Anala, Anila, Hara, Sampati (later they became ministers of Vibhishana)
Malyavan, Sumali and Mali as rulers of Rakshasas and Lanka: Lanka was built by Vishvakarma. Malyavan, Sumali and Mali used to live in Lanka. Later they attacked Devas and Lord Vishnu came to fight with them. Lord Vishnu killed Mali. Seeing Mali slain, Sumali and Malyavan, burning in grief, led precipitately towards Lanka along with their forces. But Lord Vishnu didn't stop killing Rakshas.
Kubera became king of Yakshas and Lanka: Later, they, incapable of coping with Vishnu, leaving Lanka, went to the nether regions (Rasatala) with their wives for dwelling there. Then Kubera started living in Lanka.
Ravana became king of Rakshasas and Lanka: Later Sumali encouraged his daughter Kaikesi to marry Visrava. Coming to know that his daughter's sons, Dasanana and his two brothers, got powerful boons (Dasanana was incapable of being slain by Nagas, Yakshas, Daityas, Danavas and the Devas) form Lord Brahma, Sumali shed all his fear of Vishnu and came out from Rasatala. Later Sumali encouraged Ravana to take Lanka back from Kubera. Vishrva advised Kubera to leave Lanka and build another city name Alkapuri on Kailash mountain which Kubera accepted.