Is Anatma (or no true self as discussed in Buddhist doctrines), an antithesis of Advaita Vedanta? And is the concept of atman "egoistic"?
Advaita Vedanta differentiates the True Self from the false self (egotism).
What is the false self according to Advaita?
Therefore give up the identification with this lump of flesh, the gross body, as well as with the ego or the subtle body, which are both imagined by the Buddhi. Realising thy own Self, which is Knowledge Absolute and not to be denied in the past, present or future, attain to Peace. Cease to identify thyself with the family, lineage, name, form and the order of life, which pertain to the body that is like a rotten corpse (to a man of realisation). Similarly, giving up ideas of agency and so forth, which are attributes of the subtle body, be the Essence of Bliss Absolute.
Vivekachudamani 296 - 297
Identification with the gross body, subtle body, family, lineage, name, form, order of life etc. is associated with the false self.
This is not so different in Buddhism, as found in the Bhara Sutta in which the false self is referred to, as the "carrier of the burden".
What is the True Self according to Advaita?
This world, though an object of our daily experience and serving all practical purposes, is, like the dream world, of the nature of non-existence, inasmuch as it is contradicted the next moment. The dream (experience) is unreal in waking, whereas the waking (experience) is absent in dream. Both, however, are non-existent in deep sleep which, again, is not experienced in either. Thus all the three states are unreal inasmuch as they are the creation of the three Gunas; but their witness (the reality behind them) is, beyond all Gunas, eternal, one, and is Consciousness itself.
Aparokshanubhuti 56 - 58
If the universe be true, let it then be perceived in the state of deep sleep also. As it is not at all perceived, it must be unreal and false, like dreams.
The waking world doesn't exist in the dream world. The dream world doesn't exist in the waking world. And both don't exist in deep sleep, which is neither experienced in the waking world nor the dream world. However, the True Self, the Atman, the Eternal Witness is in all of them and is Consciousness itself.
The waking world (the physical universe), the dream world and deep sleep are unreal. Only Atman, the True Self, the Consciousness, is real.
Furthermore, despite the illusion of there being many people, there's only one Consciousness.
The Supreme (Purusha) known as "I" (ego) is but one, whereas the gross bodies are many. So how can this body be Purusha? The immutable Atman, the substratum of the ego, is thus different from these two bodies, and is the Purusha, the Ishwara (the Lord of all), the Self of all; It is present in every form and yet transcends them all.
Aparokshanubhuti 31 - 32
So, while the false self is identification with limited impermanent identities like name, form, gross body, subtle body, family etc., the True Self (Atman), the real "I", is the Pure Consciousness that is the Witness of everything and is the same True Self in all. It is also the Supreme Purusha, Ishwara and Brahman.
More about this can be found in the Vivekachudamani and Aparokshanubhuti, both authored by Adi Shankaracharya.
How is this different from Buddhism?
The Buddha taught "all phenomena is not self" (Dhp 279). He did not say there's no self at all (Attakari Sutta). Just that if you break down phenomena into its constituent parts, you find that none of it is self.
He used the analogy of the lute or vina in the Vina Sutta. The lute produces beautiful music, but if it's broken down into its constituent parts, music cannot be found in them. Music is simply an emergent phenomena that arises when different parts of the lute work together.
Similarly, when you carefully examine form (body), feeling (sensation), perception, mental formations and consciousness, you cannot find the self anywhere in them. This is because the self is an emergent phenomena according to the Buddha, a mental idea (Tuvataka Sutta), that arises when form, feeling, perception, mental formations and consciousness work together, according to a mechanism known as dependent origination, that he taught.
And what about consciousness? Why isn't consciousness the self in Buddhism?
In the Mahatanhasankhaya Sutta, the Buddha taught that there are six types of consciousness that arises from moment to moment, dependent on the six types of sense media and their sense objects - eyes and forms, ears and sounds, nose and smells, tongue and taste, body and tactile sensations, and mind and thoughts. He called them eye-consciousness, ear-consciousness, mind-consciousness etc. Consciousness is therefore impermanent, and dependently originated.
The Buddha taught that "all conditioned things are impermanent" (Dhp 277), and consciousness is conditioned by the six sense media and their sense objects. He also called the six sense media and their sense objects "The All" (Sabba Sutta), meaning you cannot find any other reality outside of them. Also, please see Bahiya Sutta.
Of course, the physical world is also conditioned and impermanent. Buddhism is not concerned with whether the physical world is real or unreal. It simply holds that it is conditioned and impermanent.
Dreams are mental formations or mental fabrications in Buddhism, and thus also conditioned and impermanent.
What about Nirvana? Is it not the self in Buddhism?
Nirvana is simply peace and bliss experienced by the mind when it is free of defilements, in Buddhism. It's not a state of Super Consciousness.
OP: And is the concept of atman "egoistic"?
According to Advaita, Atman is not the same as the false self (associated with identities like name, body, family, lineage etc.). Atman is the eternal True Self, the Consciousness that is the Eternal Witness.
However, according to Buddhism, both the false self (associated with identities) and consciousness without association with identities, are both conditioned and impermanent.
OP: Is Anatma (or no true self as discussed in Buddhist doctrines), an antithesis of Advaita Vedanta?
Yes. They definitely contradict each other, as explained above.