Yes, chanting holy names generates good karma. This can be seen from the fact that people in past and even today use to name their children with some God names (eg, Krishna, Shankar, Shiv, Vishnu, Mahesh, etc), such that when they call them by their names, they remember God as well. People in some parts of India while meeting other people often say 'Ram-Ram' and the other person also says the same in return. Some people chants prayers while taking bath, before or after meals. These all are small examples where people try to remember God while doing various worldly activities.

Now, if we want proof of chanting holy names turning good then one such example is there in Bhagwata Purana, Canto 6. There is a story of a brahmin called Ajamil. Ajamil was born in a Brahmin family and a learned Brahmin but later after being in the company of a prostitute does all bad throughout his life. He took this prostitute into his home as a maidservant. Inevitably, he became so entangled that he abandoned his family, wife and children and went off with the prostitute. Due to his illicit connection with the prostitute, he lost all his good qualities. He became a thief, a liar, a drunkard, even a murderer. He completely forgot about his original training as a Brahmin, and his whole life was ruined. While on his death bed fearing of death, he called his favourite son's name 'Narayan' which is also a name for Lord Vishnu. When Yamdutas come and were trying to take him away to punish him to hell, immediately Vishnudutas also came and saved him from Yamdoots saying that since Ajamil has chanted the name 'Narayan' while on death bed, therefore Yamdoots cannot take him away. Ajamil was hence saved and spend his rest of the life in penance of Lord Vishnu. This is explained in following extract from [Wikipedia article][1] :

>It is explained that while he was dying, which is a very fearful time, he began to call out to his pet child, whose name was Narayana. Narayana is another name of God or Krishna. At that time, the Yamadutas, the messengers of death, were coming. They were tying up the subtle body of Ajamila and preparing to take him to be punished by Yamaraja, the lord of death. At the same time, because he happened to be speaking the holy name of the Lord Narayana, the beautiful Visnudutas, the messengers of Krishna, also arrived there. They checked the activities of the Yamadutas, refusing to allow them to take Ajamila for punishment. The Yamadutas were bewildered. "Why are these effulgent and beautiful personalities checking our action? It's our duty to take sinful men to Yamaraja for punishment; then they are awarded another material body for the next life so that they can get the result of their sinful activity." There was a discussion between the Yamadutas and the Visnudutas. The conclusion was that although Ajamila was sinful throughout his life and gave up his religious life, his wife and children and begot children through a prostitute, he nevertheless was purified from all these sins because at the last moment he chanted the holy name of Krishna, Narayana. If this single act of the utterance of a four-syllable word Narayana can make such a difference to life after death, what larger worlds of fullness and majesty he may not conquer by really leading a noble life of Dharma in the memory of the Lord? -- so thought Ajamila. And that very moment he renounced everything to which he was attached, went to Benaras and engaged himself in austerities and meditation and in due time reached the abode of the Lord.


  [1]: https://en.wikipedia.org/wiki/Ajamila