Fighting in self defense is not wrong. What Swami Vivekananda said had said in a context. He is talking about the highest ideal in spirituality.
When a saint achieves such a high state of realization that he sees all beings as one with himself he does not see any point in fighting them for any reasons.
Swami Vivekananda is talking on the following lines:
Manu Smriti 12.91. He who sacrifices to the Self (alone), equally
recognising the Self in all created beings and all created beings in
the Self, becomes (independent like) an autocrat and self-luminous.
12.125. He who thus recognizes the Self through the Self in all created beings, becomes equal (-minded) towards all, and enters the
highest state, Brahman.
But fighting when there is a threat on life is also allowed.
8.348. Twice-born men may take up arms when (they are) hindered (in the fulfillment of their duties, when destruction (threatens) the
twice-born castes (varna) in (evil) times,
So, Swami Vivekananda's words are to be understood in the correct context. He is talking about the highest ideal of spirituality.