In which all scriptures (Vedas, Tantras, Agamas, etc.) is the science
of mantra and the types of mantras mentioned?
The science of Mantras, Purascharana and allied stuffs are extensively dealt with in the Tantras. Tantras or Agamas deal with the subject more exhaustively than any other scriptures do.
That's primarily because, discussing the science of Mantras, is among the signs of Tantras (Tantra Lakshanas).
For example, as per ViswasAra Tantram, there are seven signs of a Tantric scripture:
Creation, dissolution, prescribed methods for deity worship, the
practice of all Mantras and related science, Purashcharana, proper
ways of performing the Shatkarmas, description of seven kinds of
DhyAna Yoga - The wise calls a scripture, which is endowed with such
seven signs, an Agama.
Where are the types of mantras like vAcika, upAnSu and mAnsika
mentioned? In which scripture?
First of all, these three are not types of Mantras. They are just the three ways of doing Mantra Japa.
Secondly, as already shown in this answer, the Tantras, Puranas, Smritis all talk about those classifications. So, there are many such scriptures.
Are there any other classification of mantras other than the above
three? If so, what and where is it mentioned?
Yes there are. Like, in general, the Mantras where deities are male, are called Mantras. Whereas, where the deity is female, the Mantra is called a VidyA.
Apart from that, Mantras are divided into three basic group - 1. Masculine (Pum Mantra or Purusha Mantra) 2. Feminine (Stri Mantra) and 3. Neuter (Napumska Mantra).
For example, the ViswasAra Tantram states:
MantrAh pumdaivata jneyA vidyA stridaivatA smritA |
humphatantAh syudvithantAh syuh striyo matAh ||
namohantAh syurituktA manavastridhA|
The Mantra of which deity is male is called a Mantra whereas it's
called a Vidya if the deity is female. Masculine Mantras are those which
end with Hum and Phat. Feminine Mantras are those that end with SwAhA.
and Mantras that end with Namaha are called Neuter Mantras.
So, Mantras that end with Namaha are neuter in gender, those which end with SwAhA are feminine and so on.
Similarly, Lord Shiva says:
MantrAh purushadevAh syurvidyAh stridevatA priye |
humparantAh prAne charati dakshine ||
vidyAh stri devatAh priye |
VAme prAne pravudhyante namohantAh
NAdidvayagate prAne sarve vodham prajAnti cha |
O beloved, the Mantra dedicated to a male deity is called Mantra and
the one dedicated to a female deity is called a VidyA. Hum and Phat
are found at the ends of Masculine Mantras. When PrAna VAyu is
predominately flowing in the PingalA (right nostril or NAdi) then the
masculine Mantras are energised. At the ends of feminine Mantras SwAha
is present. When the PrAna VAyu is predominantly flowing the IdA NAdi
(the left channel) at those times the feminine Mantras are energised.
At the end of a neuter Mantra, Namaha is present. All Mantras are
energised, when the the PrAna VAyu is eqaully flowing both the
KulArnava Tantram 16.41
Apart, from that there are Agneya and Sowmya Mantras.
ShAntike manavah sowmyA bhuyishthenduamritAksharAh |
syurviyatprAyAshchAgneyAh krurakarmasu ||
The use of Sowmya Mantras are precise in ShAnti Karmas. In these
Mantras, the letter Sa and Va are predominately present and they will
end with SwAhA. In Agneya Mantras, on the other hand, Ham and similar
Beejas are often found and they are used in Krura Karmas (i.e in
KulArnava Tantram 16.42
Now, these classifications, mentioned above, are also found in Puranas. For example in the NArada PurAna.
Quoting from the book "Essence of NArada PurAna":
The basic forms of Mantras are of ‘Stree’ (Feminine), Purusha
(Masculine) and Napumsak (Neuter). The Stree Mantras end up with
‘tha’or ‘swaha’; Purusha Mantras end up with ‘Hum’ and ‘Phut’; those
Mantras end up with ‘Namah’ are used in Napumsak Mantras. The
various Mantras are utilised in the context of six ‘Karmas’ deeds)
viz. ‘Shanti’ (Peaceful purposes), ‘Vashya’ (subjugation), ‘Stambhana’
(paralizing), ‘Dwesha’ (creating hatred), ‘Ucchhanana (uprooting) and
Where-ever Pranavaanta (ending with the word OM) and ‘maatras or
‘consonants like ‘repha’or ‘raakaar’ as also the word ‘Swaahaa’are
involved, these expressions are Agni-Sambandhi or related to ‘Agni
Karyas’ such as oblations to Fire and these are for‘Sowmya’or peaceful
(benevolent) purposes; the other Karmas (deeds) are invariably
malefic. When the Soma sambandhi Mantras are used, these
‘Agnishomatmatmak’ or Fire- oblation oriented and are uttered to
awaken Devas. Such Mantras are used along with control of breathing
through ‘Ida’ and Pingala Nadis’ (viz. left and right nostrils).
Moreover, depending on a particular disciple's compatibility with a particular Mantra, Mantras are of many kinds like - Siddha-Siddha, SiddhasAdhya, Siddhasusiddha, Siddha-Ari. Similarly, there are SAdhya-SAdhya, SAdhyasiddha and etc.
These classifications are based on several tests prescribed in the Tantras, which judge, whether a particular Mantra is suitable to be given to a particular disciple or not. So, these are not really general categories of Mantras as they vary with persons.
The names of few such tests, which the Guru has to perform before initiation, are based on several Chakras like - KulAkula Chakra, Rini-Dhwani Chakra, RAshi Chakra, Nakshtra Chakra, Akadama Chakra, Akathaha Chakras etc. These tests are conducted based upon the first letter of the disciple's name and it's relation to the first letter of the concerned Mantra.
One deeply interested in knowing the science of Mantra should thus ideally study the Tantras. Whatever I have stated here is not even 1% of that.
Related reading: What is a Mantra?