Spiritual impurity like not worshipping God, not being nice to other
people, always being angry and etc.
If that is what you are referring to as "spiritual impurity" then yes there is such a concept.
People who are wicked, don't perform their religious duties are considered to be "ever impure".
Perpetual impurity attaches to one who is always sickly, one who does not perform religious rites, one who is ignorant, one who is
notoriously henpecked, one who is grossly inclined to vile practices,
one who is always dependent upon others, and one who abstains from
Vedic study and the vow of celibacy.
Atri Samhita/Smriti
Similar verses from Daksha Smriti's Chapter 6 are given below:
Perpetual impurity is spoken of for all of them, who,- without
bathing, offering oblations to the Fire and making gifts, partake of
[their] meals'. (8)
Perpetual impurity is for a diseased person, a miser, one laden with
debts, one who does not perform religious rites, an illiterate person,
and especially for a hen
-pecked person. (9)
Daily impurity is for one who is addicted to gambling, etc., and for a
dependant. The impurity of a person, who does not perform the
S'raddhas, ends with his ashes (i.e., death). (10)
Temporary impurity is not for them, but a lifelong one. Thus impurity
according to the differentiation of merits has been spoken of. (11)
So, there are certain persons whose acts are such that scriptures consider them to be "permanently impure" as far as spirituality and religious matters are concerned.