Black is the symbol of unhappiness. That's why, we don't wear it in temples. Although, some murtis in temples are normally black, like that of Shiva. This is because Gods and Goddesses are free from happiness and unhappiness.
Above all, its just a symbol. (You can think of it as another superstition...)
In Hinduism, we have a lot of colors symbolizing things. Below are a few others:
Red indicates both sensuality and purity.
Saffron indicates sacred and holy things.
Green indicates festive season and happiness.
Yellow indicates knowledge and learning.
White is a mixture of seven different colors hence it symbolizes a little bit of the quality of each.
Blue is a symbol of nature; the sky, the oceans, the rivers and the lakes.
Like any library, Hinduism Stack Exchange shares great information, but does not offer personalized advice, and does not take the place of seeking such advice from any Acharya, Pundit, astrologer, Guru or other trustworthy Counselor.