Yes, it's likely to be true.
Short answer: ChAndAla-s are not allowed within the city limit, let alone the temples.
Note: We are talking about god's idol here and Not god. Refer the analogy for better interpretation.
Ancient Indian culture was highly relying on vibes and aura, based on which Certain things are auspicious & inauspicious.
##ChAndAla & their social role
ChAndAla & their social role
Refer this post: What are the order & duties of the children born from intermixing/hybridisation of 4 classes?
If a Sudra unites with a [Brahmana] woman belonging to the foremost of the four orders, the son that is begotten is called a Chandala. Endued with a fierce disposition, he must live in the outskirts of cities and towns and the duty assigned to him is that of the public executioner. Such sons are always regarded as wretches of their race. These, O foremost of intelligent persons, are the offspring of intermixed orders.
From above, it's clear that ChAndAla-s are usually outcasted from soceity (if not outlawed) & mostly they are not welcomed within the town limit itself.
If temple's idol is revered by several society people then certainly when a ChAndAla touches it, it's more likely to bring those inauspicious vibes he/she carries along. The idol may loose its original auspicious aura in front of devotees [once they know].
##Analogy
Analogy
If one earns a degree from a reputed institute, then he/she may not want to give that certificate paper in a dirty/careless hands. The memento of that degree in form of certificate, means a lot for the achiever.
However, suppose if the certificate is lost or got dirty, even then the education earned by that person will remain intact.
A realised person (including the lord) doesn't differentiate among various subjects of society:
BG 5.18 - The learned ones look with equality on a Brahmana endowed with learning and humility, a cow, an elephant and even a dog as well as ChAndAla.