Well, as far as I know, in Tantra tradition, Agama texts are those in which Devi questions and Shiva answers.
Examples of such texts are numerous like the Mahanirvana Tantra, Kularnava Tantra, Vijyana Bhairava Tantra etc.
And Nigama are those where Shiva questions and Devi answers. Texts of this nature are not many in number .
One example is the Kulachudamani Tantra.
Kulachudamani Tantra is a nigama, meaning that instead of Devi asking
questions answered by Lord Shiva (agama), he asks questions answered
by Devi, the goddess. In this tantra the cult goddess is
Mahishamardini, a Devi with some similarities to Durga.
Another example is the Rudrayamala:
The text takes the form of Shiva asking questions and Shakti
answering, making this nigama rather than agama form. Another example
of this style is found in the undoubtedly old Kulachudamani Tantra.
According to Astrojyoti.com (Vedic Scriptures of India):
The Vedic literature is also called by several other names –
- Nigama: Traditional wisdom transmitted from generation to generation
- Amnaaya: The root texts or primordial texts of (Hindu) tradition
- Trayi: The Vedic texts comprising of Versified mantras, prose mantras,
and melodies.
From Swami Sivananda's writings on Hindu scriptures:
V. Agamas (“Manuals of Divine Worship”)
The Agamas do not derive their authority from the Vedas, but are not
antagonistic to them.
They follow a four-fold method of worship: 1) jñana (“knowledge”); 2)
yoga (“concentration”); 3) kriya (“esoteric ritual”); 4) charya
(“exoteric worship”).
The most important books on the Agamas are: Ishvara-Samhita
Ahirbudhnya-Samhita Sanatkumara-Samhita Narada-Pancharatra
Spanda-Pradipika
The Agamas are divided into three categories:
The Vaishnava Agamas or Pancharatra Agamas (worship of Vishnu);
The Shaiva Agamas (worship of Shiva);
The Shakta Agamas or Tantras (worship of the Divine Mother or Shakti).
A. The Vaishnava Agamas
There are 215 Vaishnava Agamas, the most important ones being:
- Isvara Samhita
- Ahirbudhnya Samhita
- Paushkara Samhita
- Parama Samhita
- Sattvata Samhita
- Brihad-Brahma Samhita
- Jñanamritasara Samhita
The Vaishnava Agamas are divided into four classes:
a. Pancharatra, considered as the most authoritative. They consist of seven groups:
- Brahma
- Shaiva
- Kaumara
- Vasishtha
- Kapila
- Gautamiya
- Naradiya
b. Vaikhanasa
c. Pratishthasara
d. Vijñana-lalita
B. The Shaiva Agamas
There are 28 Shaiva Agamas, of which the chief is the Kamika Agama.
There are two principal divisions in Shaivism, both based on these 28
Agamas as well as the Vedas:
- Kashmir Shaivism, a.k.a. the pratyabhijna system, a non-dualistic philosophy; and
- Southern Shaivism, a.k.a. shaiva siddhanta, a dualistic philosophy.
Each Agama has upa-agamas (“Subsidiary Agamas”). Of these, only fragmentary texts of twenty are extant.
C. The Shakta Agamas
There are 27 Shakti Agamas, usually in the form of dialogues between
Lord Shiva and his consort Parvati. The most important ones are:
- Mahanirvana Tantra
- Kularnava Tantra
- Kulasara Tantra
- Prapanchasara Tantra
- Tantraraja Rudra-Yamala Tantra
- Brahma-Yamala Tantra
- Vishnu-Yamala Tantra
- Todala Tantra
And, according to Hindupedia.com:
Agamas and Tantras are a vast collection of knowledge and form a
major portion of spiritual literature and practices. Like the Veda,
they have come down through Guru-Sishya parampara, in oral
traditions. Agamas form the base for many of the popular as well as
specialist aspects of Hinduism. The word Agama means 'that which
has come to (us)'. Tantra means 'that which protects with detail'.
Sruti, the eternal word, is said to be of two forms – Nigama (Veda)
and Agama. Two kinds of texts, Agama and Tantra are in general
grouped under the same class of literature. There are three main
classes of Agamic/Tantric texts Vaishnava Agamas, Saiva Agamas and
Sakta Tantras, though not limited to these. The Vaishnava and Saiva
texts are generally called Agamas, while the word Tantra in general
applies to Sakta texts. However, technically Tantra is a part of
Agama and owing to the centrality of Tantra the two words are used
often interchangeably.
But, in Mahanirvana Tantra (Chapter 9) Sadashiva says:
All Mantras in the Nigamas, Agamas, Tantras, Sanghitas and Vedas, have
been spoken by Me. Their employment, however, varies according to the
Ages (11).
Now, since Agama, Nigama, Vedas etc. are mentioned separately, they must be different from each other.
In the first chapter of the Mahanirvana Tantra, Devi Adya (Mother Goddess) makes it clear to us, that according to the nature of Yugas, Sadashiva propounds and creates a variety of scriptures.
In Kali Yuga these scriptures are the Tantras or the Agamas & the Nigamas. They are the only mode of liberation.
So, as I have said earlier, Agamas are different from the Vedas. But it is different from Smritis and Puranas as well.
Shri Adya said:
O Bhagavan! Lord of all, Greatest among those who are versed in Dharmma, Thou in former ages in Thy mercy didst through Brahma reveal
the four Vedas which are the propagators of all dharmma and which
ordain the rules of life for all the varying castes of men and for the
different stages of their lives (18-19). In the First Age, men by the
practice of yaga and yajna prescribed by Thee were virtuous and
pleasing to Devas .....By the study of the Vedas, dhyana and tapas,
and the conquest of the senses, by acts of mercy and charity men were
of exceeding power and courage, strength and vigour, adherents of the
true Dharmma, wise and truthful.....
After the Krita Age had passed away Thou didst in the Treta Age
perceive Dharmma to be in disorder, and that men were no longer able
by Vedic rites to accomplish their desires. ....Having observed
this, Thou didst make known on earth the Scripture in the form of
Smriti, which explains the meaning of the Vedas, and thus delivered
from sin, which is cause of all pain, sorrow, and sickness,....
Then, in the Dvapara Age when men abandoned the good works prescribed in the Smritis, and were deprived of one half of Dharmma
and were afflicted by ills of mind and body, they were yet again
saved by Thee, through the instructions of the Sanghita and other
religious lore ...
Now the sinful Kali Age is upon them, when Dharmma is destroyed, an
Age full of evil customs and deceit. Men pursue evil ways. The Vedas
have lost their power, the Smritis are forgotten, and many of the
Puranas, which contain stories of the past, and show the many ways
(which lead to liberation)....By Thee also have been composed for
the good and liberation of men the Tantras, a mass of Agamas and
Nigamas, which bestow both enjoyment and liberation, containing
Mantras and Yantras and rules as to the sadhana of both Devis and
Devas.
The above passage makes it clear that in Krita Yuga, Vedas are the predominant Hindu shastra, in Treta, the Smritis, in Dwapara, it is the Samhitas and religious lore (I think Puranas are meant by this although I'm not quite sure) but in Kali Yuga all the above shastras have lost their powers. Hence Shiva created the Tantras or the mass of Agamas and Nigamas.
So, Agama is a distinct category of Hindu shastra, different from any other like the Vedas, Smritis and the Puranas.
It is repeatedly stated in the Tantra texts that, in this Yuga, the Veda mantras and other powerful mantras are powerless. They are often compared to snakes without any poison.
And that spiritual and material progress and liberation are only achieved by practicing the mantras given in the Tantra texts that is those given in the Agamas.
In Kularnava Tantra we find some definitions of Agama Texts.Which explains the term Agama and also why these texts are so called. Few among them are given below:
AchAra KathanAddwivya Gati PrApti NidAnataha | MahArtha Tattva KathAnAdAgamah Kathitha Priyeh||
Because it narrates the course of conduct, Achara, with a view to
arrive at the godly goal, divyaGAti, because it speaks of the great truth(Mahartha Tattva)
, it is called Agama.
In PAsupata Sutra Agama is defined as-
The Shastra that came(Agata) from the mouth of Maheswara(Shiva) through Guru Parampara is
Agama.
Yet another definition of Agama texts is -
The Shastra which have the seven signs viz :
Srishti(creation),Pralaya(destruction),Devata Puja
rules(vidhis),Mantra Sadhana of all kinds,Purashacharana,Shatkarma
,the four-fold Dhyana Yoga is called Agama by the wise.(This is as per
Viswa SAra Tantra)