(I should say that this is not a definitive answer, unfortunately, and this is more of an amalgam of whatever I could find from my research. This is probably the closest we'll get.)
The book by Halbfass quoted by Wikipedia has this to say:
Incidentally, we should not forget the extent to which "Neo-Vedanta" and "Neo-Hinduism" were used in India and by Indians. Hacker himself borrowed the term "Neo-Hinduism" from the Jesuit scholar and long- time resident in India, Robert Antoine, who may have adopted it from the Bengali Brajendra Nath Seal, whom we already mentioned as an early champion of comparative philosophy. It seems likely that Seal (1864- 1938) was the inventor of this term, which he used to characterize the literary work of Bankim Chandra Chatterjee. "Neo-Vedānta" (or Bengali abhinava vedanta) was used by Indians as well as Europeans in 19th century Bengal.
And an earlier work by Hacker agrees:
We may note here that the term "Neo-Hinduism" ("Neuhinduismus"), which Hacker borrowed from the Jesuit scholar Robert Antoine (who may have adopted it from Brajendranath Seal, 1864-1938), is much more visible and significant in his writings than "Neo-Vedanta," which appears only sporadically. "Neo-Vedanta" or "Neo-Vedantism" and the corresponding Bengali expression abhinava vedanta had been used earlier by Christian missionaries as well as Hindu traditionalists against the innovations of Rammohan Roy.
That's all that I can find on the origin of the term. I've looked through most of Seal's books that are available online, and am yet to find any mention of the word 'neo-vedanta'
Since there doesn't seem to be a certain answer, here are some of the earliest uses of these terms I was able to find. The very first seems to be from 1844, in the Calcutta Review, though the author is left uncredited:
There are two citations in this book from 1851, which was written by an Indian person:
[...]
By the early 20th century, the term had evidently come into popularity in the Western world as well, and we see a number of Western scholars using the label.
Rammohan's name has come up a few times in this answer, and Seal indeed had a work called Rammohan Roy: The Universal Man, but this was published in 1933, which is well after the earlier examples I linked.
So:
- it was not invented by Hacker
- it goes back to at least the mid-nineteenth century
- it became a more popular term for Western scholars in the early-twentieth century.
