Is chanting any mantra without initiation a sin? What do scriptures say about this?
In Agamas DikshA or initiation is given utmost importance.
Mantras should be taken from a bonafide Guru only. If one chants mantra by obtaining it through some other sources (like books etc) then that is regarded as an offence.
For example, the KriyAsAra Tantram says:
Kalpe drishtA tu yo mantram japed gurumanAsritaha |
SutanAsho bhavet tasya phalam kinchinna vidyate ||
The person who, without depending on a Guru ( Gurumanasitaha), chants mantras by taking them from a book, gets his progeny destroyed [by doing so] (sutanasho) and does not get any benefits [of the chant] either.
The KulArnava Tantram is even more strict on this:
Pushtake likhitAnmantrAn vilokya prajapanti ye |
BrahmahatyAsamam teshAm pAtakam parikirtitam ||
One, who chants mantras by seeing them from books [that is not obtained duely from a sampradaya Guru] commits, thereby, sin as grave as BrahmahatyA (Brahminicide).
KulArnava Tantram, UlAsaha 5, Verse 22.
So, its always better to go for Guru dikshA if you are interested in mantras.
(1) Diksha if taken from a real Guru, who is a master in that Vidya, can do wonders because of transmission of energy. A Diksha involves three: Disciple, Guru and Deity.
Today Diksha business is running where only two are involved, Disciple and incompetent Guru. This will not work. It is mere satisfaction that you have taken Diksha but it is of no use unless you find a real Guru, which is difficult these days.
(2) There is too much fear because of misinterpretation of scriptures. From the time these were written, many interpretations have taken place. Initiation (Diksha) is required because in Kaliyug, Lord Siva has locked most of the mantras (Keelit) to avoid misuse. Only a real Guru can unlock and give it to you. Regarding progeny destroyed if mantra chanted without initiation, I think it refers to the side effects of few mantras like: Baglamukhi Mantra and similar mantras.
(3) Mantras like Mahavidya mantra should not be attempted without a Guru, however, other Mantras need not necessarily require Diksha. Here the only difference you may feel is that after Diksha (from a real Guru), you will progress 5x or 10x times whereas alone you can progress at a Turtle speed. Also, during recitation, if you experience something, no one can guide you. Only a real Guru can guide.