What is the significance of mantras and why do people chant them?
The Sanskrit word Mantra is formed from the root word "Man" meaning mind and "tra" meaning tools/instruments or vehicle. Hence Mantra serves as a instrument to transform the mind(towards pure primordial consciousness).
Mantras are core to Tantric practices. The mantras, especially Bija(seed) mantras are supposed to be primordial sounds heard by ancient seers in deep meditation. For example, Dr. David Frawley says this about Shakti mantras:
Most Shakti mantras contain the vowel–I, the vibratory ee-sound, which is the main primal sound of Shakti. Most contain the consonant-R, which is the seed of fire, heaven, light, order and dharma and has a stimulating and energizing effect. Some contain the consonant-L, which is the seed of earth, water, joy and bliss and has a calming and stabilizing effect. Many begin with either the letter-H, which indicates prana, light and the Sun, or the letters S or Sh, which indicate the Moon, the mind and water. Some like Au or Aim consist of vowels only.1 (Emphasis mine)
Like the Shakti mantras, other mantras are used to produce different effects.
In short, mantras are primordial vibrations which are supposed to have existed eternally long before the world was manifested in the way it is now and hence are considered divine and produce an effect on our consciousness. This is their significance.
The below reference has more information about Mantras.
 Mantra Yoga and Primordial Sound, Dr. David Frawley (Pandit Vamadeva Shastri)
Excerpts from a talk by Sri Sri Ravi Shankar:
When someone abuses you verbally, how do you feel? If someone says that you are a donkey, how do you feel? What does it do to you? What does it create? Anger! It shakes you. It creates some negative vibrations; you feel angry. You feel some sensations in the stomach, in the head.
If a bad word can create so much physiological reactions in you, do you think a very sweet name, a chanting, which includes a lot of energy of the cosmos does not do anything to your body?
It is unscientific and stupid to think that a mantra does not do anything to your body. That is not the case! It does.
Mantras bring out all the positive energy inside you. That is why it is called mantra kavach; a mantra creates the sense of an armor around your body.
Sometimes you meet some people and you feel like talking to them. You get good vibes from them. Sometimes you meet people and you want to avoid them. Do you know why? The negative vibes around a person make them repulsive. Mantras turn around these negative, repulsive vibrations, into more positive and attractive vibrations. This is the advantage of mantra chanting.
Today, if you go to New York, you will find hundreds of places where they learn chanting. People go for one hour in the evenings to chanting classes. They chant ‘Om Namah Shivaya’,’ Om Namo Narayana’, ‘Sri Rama Jai Ram Jai Jai Ram’. Of course, they have a different pronunciation.
In China, they say, ‘Ladhe Ladhe’ instead of ‘Radhe Radhe’.
In Taiwan, 7,000 people to 8,000 people sing ‘Ladhe Govind’ instead of ‘Radhe Govind’.
People have seen that there is an effect. And there really is!
Now when should you do it? If you do chanting after pranayama and meditation, then it has a better result. If you do it at a superficial level, then it is not so effective.