As per Mimamsa, whenever Shruti contradicts Smriti, Smriti should be discarded:
Pûrva Mîmâmsâ ('But where there is contradiction Smriti is not to be regarded,' I, 3, 3)
A commonly cited example of a contradiction is this:
(a) The Smriti rule that the whole of the Udumbara-Post should be covered is in conflict with the Shruti rule that one should sing a Saman, touching the Post [no touching of the Post is possible when the whole of it is covered with cloth];
I don't see how the vedic rule contradicts the smriti rule; if the post is covered in cloth, you can still touch the post, just like how if a person is wearing clothes, you are still touching the person even if you touch his clothes.
If not, then one can rightfully sexually harass women by touching them in inappropriate places because they're wearing clothes. In that case, one would be able to claim they aren't touching the women because they are wearing clothes. Hence, there is no sexual harassment.