scriptural sources for how to chant vedic mantras and other things involved.
We get some idea about science of Mantras or how to recite mantras properly from panini's book Paniniya Shiksha. These are the treatise on the topic of Shiksha , which is one among the six Vedangas, dealing with Sanskrit phonetics and phonology.
In some of the verses Panini is telling us about how to chant vedic Mantras properly , what should be avoided while doing so , how should be the voice etc. It is basically the instructions given to students about the Chanting of Vedic Mantras , but we can get a fair idea about the rules to be observed and faults and merits of good chanting / recitation of mantras from these verses.
Here are some verses which do talk about the science of mantras .
माधुर्यमक्षरव्यक्तिः पदच्छेदस्तु सुस्वरः ।
धैर्यं लयसमर्थं च षडेते
पाठकाः गुणाः ।। 33।।
Sweetness , clearness ,separation of words ,right ascent ,patience and
ability to observe time are six merits in a reciter .
शङ्कितं
भीतिमुद्घृष्टमव्यक्तमनुनासिकम् ।
काकस्वरं शिरसि गतं तथा
स्थानविवजिर्तम् ।। 34।।
34 Shyness , fear ,extreme loudness , indistinctness , under
nasalisation , repressed tone , under cerebralization , non observance
of place of articulation (in general) and (proper) ascent (and
harshness , creating under separation between words , uneven tone ,
hastiness ,want of due palatisation : these are fourteen faults in
vedic chants.
उपांशु दष्टं त्वरितं निरस्तं विलम्बितं गद्गदितं
प्रगीतम् ।
निष्पीडितं ग्रस्तपदाक्षरं च वदेन्न दीनं न तु
सानुनास्यम् ।। ३५ ।।
One should not recite the vedic passage in under-tone, between one's
teeth ,quickly haltingly ,slowly ,with a hoarse voice , in a single
song manner , with repressed voice , omitting (occasionally) words and
Syllables and in plaintive voice.
गीती शीघ्री शिरःकम्पी तथा
लिखितपाठकः ।
अनर्थज्ञोऽल्पकण्ठश्च षडेते पाठकाधमाः ।। 32 ।।
Those who recite the Veda in a sing song manner , (too) quickly , with
the nodding of the head , use a written text at the time of recitation
, do not know the meaning of passages read and have a low voice are
six kind of bad reciters.
The method of chanting vedic mantras were handed down from Guru to Shishya mainly through oral tradition , since direct listening to Guru while he is chanting and repeating after him and then memorizing is the way to learn exact pronunciation of words as well as understanding of , tone pitch of voice and other technical things are very difficult to teach without actually making sound as this science mainly is associated with sound.
Here are the above
Sanskrit Verses.