For the question in the title there can not be any answer because as far as I know, the Vedas don't talk about temples.
So, I am answering the following question:
As per Parashara smirti, Shudras were considered less pure to perform
some auspicious activities.
Is there any vedic statement which supports this claim?
Yes, we have some statement ...
Yes, in ancient times that used to be the practice when a Dvija used to spend 48 years in the Brahmacharya Ashrama (the student phase of life). And, in that phase, maintaining celibacy is to be fulfilled as it's a vow.
The Baudhayana Dharma Sutras state that:
1 Forty-eight years, according to ancient practice, is the period
of studentship for studying ...
Vedacharya David Frawley explains the above issue in his book Gods, Sages and Kings: Vedic Secrets of Ancient Civilization, as follows:
The symbolism of ships is as pervasive in the Vedas as that of the
sea, which it tends to reinforce. The saving action of Agni, the
sacred fire, is frequently compared to a ship that carries us across
the river or ...