For the question in the title there can not be any answer because as far as I know, the Vedas don't talk about temples.
So, I am answering the following question:
As per Parashara smirti, Shudras were considered less pure to perform
some auspicious activities.
Is there any vedic statement which supports this claim?
Yes, we have some statement ...