Does Maya (Materialistic illusion) fall into one of the following three categories:
Or is maya categorized as nirguna or that which is beyond the three gunas?
Maya is the mother of the three Gunas as rightly said by Rohith under your question. So, she/it can not be categorized under one particular Guna.
I have selected few passages from the Devi Bhagavata Purana (abbreviated as DB here) to "describe" the Maya, who/which actually is regarded as incomprehensible:
21-23. O Dvaipâyana! Thus saying, Visnu stopped. I was greatly astonished and asked that Eternal Vâsudeva, the Deva of the Devas, the Lord of the World, “O Lord of Ramâ! What is the form of Mâyâ? How is She? What is the measure of Her strength? Where She resides? Whose substratum is She? Kindly tell these to me. O Preserver of the Universe! I am greatly desirous to see Mâyâ; Shew Her to me quickly. O Lord of Ramâ! I am very eager to know Mâyâ. Be graciously pleased to describe tome the glory of Mâyâ.” 24-36. Visnu said :-- Mâyâ resides everywhere throughout this whole Universe; Her nature consists of the three Gunas; She is the substratum of all; She is omniscient, and acknowledged by all; invisible, and of diverse forms.
. Nârada said :-- O Thou whose only wealth consists in asceticism! I am now describing to you all those good stories; hear attentively. O Muni! This Mâyâ and Her Power are incomprehensible even by those who are the foremost amongst the Yogins. This whole Universe, moving and non-moving, from Brahmâ to the blade of grass, is enchanted by that Unborn and Incomprehensible Mâyâ; therefore no one can escape from the hands of that Mâyâ.
DB Book 6, Chapter 28.
O Nârada! Know the power of Mâyâ incomprehensible as the things seen in dreams cannot be certainly known that all those are false. O Muni! Neither I, nor S’ambhu, nor Brahmâ can measure the power wielded by Mâyâ and Her three Gunas, very hard to fathom. How, then, can any ordinary mortal know them!
Now as to whether Maya is Nirguna or Saguna, then it is said that when the Highest Paradevi conjoins her Shakti Maya, she then becomes Saguna. From that i could not conclude anything though.
O King! When Brahmâ, Visnu, Mahes’vara and the solar and lunar Kings, the fourteen lords of Manvantaras, Manu and others are under the control of the mayic Gunas, what to speak of other ordinary mortals, men and the other Jîvas. The whole world is under the control of Mâyâ; the Devas, men and all other beings. None should doubt on this point. All the embodied beings labour under the directions of Mâyâ; never can they work independently. This Mâyâ is again always residing in the Highest Essence, the Samvit or the Universal Pure Consciousness. Thus Mâyâ is dependent on the Highest Goddess,
In the following passage the Highest Devi Bhuvaneswari says that herself:
I am Nirguna. And when I am united with my S'aktî, Mâyâ, 1 become Saguna, the Great Cause of this world. This Mâyâ is divided into two, Vidyâ and Avidyâ. Avidyâ Mâyâ hides Me; whereas Vidyâ Mâyâ does not. Avidyâ creates whereas Vidyâ Mâyâ liberates.
Mâyâ united with Chaitanya (Intelligence), i. e., Chidâbhâsa is the efficient cause of this Universe; whereas Mâyâ reduced to and united with five original elements is the material Cause of the Universe. Some call this Mâyâ tapas; some call Her inert, material; some call Her knowledge; some call Her Mâyâ, Pradhâna, Prakriti, Ajâ (unborn) and some others call Her S'aktî. The S'aiva authors call Her Vimars'a and the other Vedântists call Her Avidyâ; in short, this Mâyâ is in the heads of all the Pundits. This Mâyâ is called various in the Nigamas.
DB Book 7, Chapter 32
Maya can't be categorised as Satva, Raja or Tama. Actually Maya is the source of these three Gunas or Gunas are work of Maya. Following verses from Vivekachudamani describes the nature of Maya:
Avidya (Nescience) or Maya, called also the Undifferentiated, is the power of the Lord. She is without beginning, is made up of the three Gunas and is superior to the effects (as their cause). She is to be inferred by one of clear intellect only from the effects She produces. It is She who brings forth this whole universe.
She is neither existent nor non-existent nor partaking of both characters; neither same nor different nor both; neither composed of parts nor an indivisible whole nor both. She is most wonderful and cannot be described in words.
Maya can be destroyed by the realisation of the pure Brahman, the one without a second, just as the mistaken idea of a snake is removed by the discrimination of the rope. She has her Gunas as Rajas, Tamas and Sattva, named after their respective functions.