What is initiation of Mantra? Why is it required for it to be actually effective for the reciter? I've read that people without initiation of the mantra would recite it for the whole lifetime and still it won't be effective.
Why is it required for it to be actually effective for the reciter?
The reason is - "the scriptures say so". That Mantra Japa is fruitful only if the Mantra is Guru-given and otherwise it is not.
For example, Lord Shiva says:
PAramparyAgamAmnAyam mantra AchArAdikam priye |
Sarvam gurumukhAllabdham saphalam syAnna chAnyathA ||
Traditions, Agama, Amnaya, Mantras, conducts and other similar things, O Beloved one, are fruitful only if obtained from Guru's mouth and not otherwise.
KulArnava Tantram 11.46
I've read that people without initiation of the mantra would recite it for the whole lifetime and still it won't be effective.
Yes that's true. And, that's exactly what Lord Shiva says in the following verse:
UpAsanA shatenApi yAm vinA naiva sidhyati |
TAm dikshA mAsrayed yatnAt sriguror mantra siddhaye ||
Even hundreds of UpAsanAs fail to achieve success without which, that DikShA, should be obtained with due care from SriGuru for Mantra Siddhi.
KulArnava Tantram 14.88
That means if the Mantra we are chanting is not taken from the Guru, then how much effort we put into chanting it, that effort all goes to vain.
And, the following verse says that only a Mantra which is duly obtained from a Sampradaya Guru gives results.
DikshA purvam kuleshAni pAramparyakramAgatam |
NyAya lavdhashcham yo mantrah sa cha siddho na samshayaha ||
The Mantra, which is legally obtained through DikshA and handed down through a Tradition (ParamparA), O Goddess of the Kulas, is fit to be mastered without doubt.
KulArnava Tantram 15.15
And, moreover, taking Mantra from any sources other than the Guru (like books etc) and chanting them is also censured as a sin in the scriptures:
Pustake likhitAn mantrAn avalokya japettu yah |
Sa jivenneva chandAlo mritah syAno bhavishyati ||
One who does Japa by seeing Mantras from books becomes a Chandala (i.e an outcaste) in the current life and after death (in the next life) becomes a dog.
A SyAnkhyAyan Tantra verse quoted in Nityotsava's 3rd Chapter
Few other verses of a similar nature (that criticize Mantra Japa without initiation) are found in this answer.
Now, coming to the question in the title:
What is Mantra initiation?
In brief, it is the rite where the Guru gives the Mantra to the disciple, while they are present in front of each other. Along with, a few other allied rituals are also done and is overall quite an elaborate and lengthy procedure.
The simple transfer of the Mantra is called Upadesha where as the full process is called DikshA SamskAra.
In this answer I have given few verses which describe the process of this Mantra Upadesha.
Gurustu PrAngukhobhutva ShishyA Prachi Mukha Stithah |
TrivAram Dakshine Karne VAme chaiva TathA Sakrit ||
Viparitam Tato Geyam StreesudrAnancha VAmatah |
Guru should face the east and the disciple the west direction. Guru should then speak the mantra into the disciple's right ear 3 times and left ear 1 time. For Sudras and women the process will be reversed.
The above verses are from the Rudra YAmala Tantram. It is usually like that, Mantra initiation is performed. But the process may vary depending on the Mantra and also depending on the sects.
For example, for a particular Mantra I know, it is required that the Guru should speak it for 8 times ( and not just 3 times) for the initiation to be complete.
Now, when God speaks a Mantra and another being receives it and masters it by doing the necessary practice, he/she is called the Rishi of that particular Mantra. Every Mantra has such a Rishi.
So, the first transfer of any Mantra always takes place between two conscious beings (viz: God-Rishi). That's why whenever such a transfer shall take place in the future, it must always be between two conscious beings, to be considered as effective.
But when the source is a book this condition is not met.