Is it true that chanting mantra without moving lips and tongue gives more effect? Is it called Aajapa mantra?
Yes, that's true. Usually three forms of chanting (japa) are mentioned in scriptures.
From Tantra SAra:
NijakarnAgocharohayam Sajapo MAnasah Smrito |
UpAnsunijakarnasya Gocharaha Parikirtitaha ||
MantromucchArayedvAchA Sajapo VAchika Smrito |
When the chants are completely done in one's mind and they are not audible even to oneself, its called mAnasa. When they are like whispers and the chants are audible to one's own ears, then it's called upAnsu. Loud japa, which everyone around can hear, is called vAchika japa.
So we have:
Ucchai japa or the vAchika japa- This is the loud japa which you can hear as well as others around you can too hear.
UpAnshu Japa- In this japa there are only slight movements of tongue and lips involved. This japa can be heard only by the one who is chanting and not by others.
MAnasa Japa- Repetition of the mantra completely in mind by contemplating on it's consciousness and meaning is mAnasa japa. Here, even lips, tongue don't move. And, even one who is doing it can not hear the japa.
Among these three kinds, the mAnasa is considered the best followed by upAnshu followed by vAchika. So, loud japa (or vAchika) is considered the worst kind. Now, this has been mentioned in many many scriptures.
For example, Lord Shiva says:
Ucchair japohadhamah prokta upAnshur madhyamah smritah |
Uttamo mAnaso devi trividhah kathitah japah ||
O Goddess, the loud japa is considered as the worst kind, the whispering japa (upanshu) is the middling kind and the japa in the mind (mAnasa) is the best form of japa. These three are said to be the kinds of japa.
KulArnava Tantram 15.55
Similarly, in another Agama, Lord Shiva says:
Nigadena tu yajjaptam lakshyam chopAmshunA samam |
MAnasena maheshAni koti jApya phalam bhavet ||
O Great Goddess, One upAnsu japa is equal to one lakh loud japa and by doing one mAnasa japa one gets the fruits of one crore upAnsu japa.
NityAshodashikArnava Tantram 5.18
In ShivAgama, Lord Shiva gives a different equation:
TasmAcchatagunopansuhu Sahasro MAnasah Smritoho |
UpAnsu japa is 100 times more fruitful than vAchika and mAnasa 1000 times.
Same thing is stated in the Manu Smriti, in different words:
Manu Smriti 2.85. An offering, consisting of muttered prayers, is ten times more efficacious than a sacrifice performed according to the rules (of the Veda); a (prayer) which is inaudible (to others) surpasses it a hundred times, and the mental (recitation of sacred texts) a thousand times.
The Vashista Dharma Sutras (26.9) also highly praises silent japa as follows:
The sacrifice consisting of softly recited prayers is ten times better than a sacrifice involving the immolation of an animal; reciting prayers silently is a hundred times better; and mental prayer, tradition says, is a thousand times better. The four types of sacrifices with cooked food* together with the sacrifices ordained in the Veda are not worth a sixteenth part of a sacrifice consisting of softly recited prayers. 11 Only through softly recited prayers does a Brahmin attain eternal bliss, in this there is no doubt. Whether he does other things or not, he is said to be a Brahmin who is a friend to all. 12 People who softly recite prayers, offer sacrifices, meditate, live at sacred fords, or have bathed after the ‘Head’ vow*––their sins do not endure.
Ajapa Japa is not necessarily MAnasa japa. It is something which is happening spontaneously in everyone of us continuously. Even if we consciously don't want to do it we are bound by nature to do it. It's the great Hamsa (one inhalation+one exhalation; Ha and Sah), the manifestation of the combined form Shiva (Ha)- Shakti (Sah) in all beings.
In the Sandilya Upanishad it says in Chapter 1 (Thirty Minor Upanisads, p 164, translator K. Narayanasvami Aiyar):
...Japa is the practicing of the mantras into which one is duly initiated by one's spiritual instructor, and which is not against [the rules of] the Vedas. It is of two kinds--the spoken and the mental. The mental is associated with contemplation by the mind. The spoken is of two kinds--the loud and the low. The loud pronunciation gives the reward as stated [in the Vedas]: [while] the low one gives a reward thousand times [of that]. The mental [gives] a reward a crore [times of that]...
Swami Vivekananda says (Complete Works, V1, p 190; available here under the heading Volume 1, Raja Yoga, sub-heading Raja Yoga in Brief - http://www.advaitaashrama.org/cw/content.php):
The following are helps to success in Yoga and are called Niyama or regular habits and observances; Tapas, austerity; Svâdhyâya, study; Santosha, contentment; Shaucha, purity; Ishvara-pranidhâna, worshipping God. Fasting, or in other ways controlling the body, is called physical Tapas. Repeating the Vedas and other Mantras, by which the Sattva material in the body is purified, is called study, Svadhyaya. There are three sorts of repetitions of these Mantras. One is called the verbal, another semi-verbal, and the third mental. The verbal or audible is the lowest, and the inaudible is the highest of all. The repetition which is loud is the verbal; the next one is where only the lips move, but no sound is heard. The inaudible repetition of the Mantra, accompanied with the thinking of its meaning, is called the "mental repetition," and is the highest.
And in his translation of the Mahanirvana Tantra, Arthur Avalon (Sir John Woodruffe) writes in his Introduction:
Japa is defined as "vidhanena mantrochcharanam," or the repeated utterance or recitation of mantra according to certain rules. It is according to the Tantra-sara of three kinds: Vachika or verbal japa, in which the mantra is audibly recited, the fifty matrika being sounded nasally with vindu; Upangshu-japa, which is superior to the last kind, and in which the tongue and lips are moved, but no sound, or only a slight whisper, is heard; and, lastly, the highest form which is called manasa-japa, or mental utterance. In this there is neither sound nor movement of the external organs, but a repetition in the mind which is fixed on the meaning of mantra...
There are many references to the 3 ways that japa can be performed through many different scriptures - all of them in agreement. One should talk with one's own guru to learn what is best for one's self and under what circumstances and conditions.