Although making charity is highly praised and a meritorious act, but at the same time it is also true that by accepting gifts the recipient's hands get sullied.
In Kularnava Tantra 15.78, Lord Shiva says:
jihvA dagdhA parAnnena karau dagdhau pratigrahAt |
para-stribhih kArya-siddhih katham bhavet ||
The tongue is burnt by the food of another; hands burnt by accepting
from another; the mind burnt by (thoughts of) women of others ; how
then can there be success in the endeavour?
This was said in the 15th chapter of the text that deals with Purashcharana and allied things. In one earlier verse it is said that if while performing some Dharmic acts/rites one lives on food given by others then half of the merit (that results from the Dharmic act) goes to the person who maintains (the performer of the act) and half to the performer. So, it's not desired to be maintained by others while doing Purashcharana of Mantras.
That by receiving gifts one becomes impure can be proven from other scriptures like Manu Smriti too.
At a funeral ceremony, for example, an invited Brahmin is supposed to receive gifts. But after that he is not supposed to recite the Vedas. Why?
Because, he has made his hands impure upon receiving those gifts. And, for a Brahmin, it is said that his mouth is his hand. So, when the hands get impure the mouth also gets impure and with that impure mouth one must refrain from reciting the Vedas. That is what is meant.
Manu 4.116. Let him not study near a burial-ground, nor near a
village, nor in a cow-pen, nor dressed in a garment which he wore
during conjugal intercourse, nor after receiving a present at a
4.117. Be it an animal or a thing inanimate, whatever be the (gift) at a Sraddha, let him not, having just accepted it, recite the Veda;
for the hand of a Brahmana is his mouth.
A pure and true Brahmin is only fit to be made gifts unto and not any other Brahmins. Because such a Brahmin is like the fire into which sacrificial offerings are poured. So, he is least affected by the impurity that results from receiving gifts.
3.212. But if no (sacred) fire (is available), he shall place (the offerings) into the hand of a Brahmana; for Brahmanas who know the
sacred texts declare, ’What fire is, even such is a Brahmana.’
So, Swami Vivekananda isn't making a contradictory statement at all and moreover his statement isn't a generic one but meant only for Yogis. But if he was saying that, irrespective of whether one is a Yogi or not, receiving gifts is bad, even then it would not have being wrong. Because as shown above, by accepting gifts one indeed gets impure. To subdue this effect gifts are required to be made only to true Brahmins who are pure like fire.
Therefore, there's no denying the fact that by receiving gifts one becomes impure and it's a hindrance in spiritual progress.