Everyone here has given the answers based on mythology.
Mythological stories were a tool to mainly teach people to either avoid certain behaviour and / or to aspire and strive to improve one's self by adopting desirable traits. To the intelligent, it should thus be self-evident that they are not meant for literal interpretation.
Brahman: The GOD (a very simple interpretation).
Brahmin: A student of the vedas who leads a dharmic life; scholar.
Brahma: God of creation (the creator; part of the trinity).
The Puranas describe the creator Brahma as the manifestation that Brahman assumes for the purpose of any creation. Brahma, thus having the nature of Brahman, is considered as the veda incarnate and the embodiment of knowledge.
Worship in Hinduism is steeped with symbolism. Very briefly -
Whether it is the vedic form or the agamic form of worship, both heavily utilise symbolism to reinforce a concept of what the deity being worshipped represents and / or the focus of the worship. Symbolism is used to reinforce religious teaching to increase ones understanding about some concept or the deity being worshipped, thus increasing our spiritual wisdom.
For example, the vedic form of worship do not utilise idols or shrines and is offered in the form of yagnas, almost always with a ritual fire. Oblations are offered into the fire, along with vedic chantings. As we place the ahuti (offerings) into the flame, we symbolically offer all our "impurities" - our anger, greed, jealousy, grudges, pains, obstacles - and pray to God to make us as pure as the offerings we give to Agni. Performing a Yagna itself is a symbolism of the unity of the physical (e.g. the heat from the fire) and meta-physical (e.g. the Vedic mantras), to achieve desired physical, psychological and spiritual benefits.
Now, the agamic form of worship is based on idols and temples, along with other associated rituals. The agamic scriptures have many parts out of which the Kriya pada of each Agama describes in detail the rules for constructing a temple or sculpting a diety (among many other things). And these rules have either a sound reason and / or is symbolic to specify something. For example, the mudras (gestures) of an idol that has fingers pointing down symbolize that the God is of a charitable nature, where as the mudra where the fingers point upwards (as in blessing someone) signify a protector.
Since we are talking about temples (whether of Brahma or Vishnu or anyone else), we are essentially talking about the agamic form of worship in Hinduism.
As mentioned earlier, the agamic scriptures deal with the philosophy and spiritual knowledge of worship of a deity. Every idol or form that is used for such worship is different because each symbolically represent the different characterisation of the deity being worshipped, based on the philosophy that the scriptures define about said deity.
As the scriptures (and existence of temple) point out, it isn't as if Brahma isn't / wasn't worshipped at all. The Garbhadhana (conception ritual), Punswana (fertilization ritual), Chudakarma or Keshanta karma (ceremonious tonsuring), Vivaha (wedding ritual) are all also associated with Lord Brahma. There are also funeral rites where Lord Brahma is invoked. All these can be symbolically considered as the act of creation or new beginning. Thus, logically, you would turn to the Creator to please / appease him.
But then, you can also turn to Vishnu, the protector, or Shiva the destroyer to please or appease them, for these very things. Why not offer worship to any one or any two or all of them? Such perceptions lower the importance of Brahma in the eyes of the followers of either Vishnu or Shiva, and would have certainly contributed to his loss of followers.
Brahma is also represented symbolically as the veda itself - the reservoir of all knowledge. (In this context, a cosmic power of the ultimate reality - the knowledge that contains the knowledge of all the phenomena evolved from it. Or in simpler words TRUE KNOWLEDGE.)
Worship is based on faith. And to faithfully believe something means you do not question it. Yet, the quest for knowledge starts with doubts, questions, debates and discussions. Thus, a faith based worship of Brahma - the manifestation of true knowledge - is a philosophical contradiction, as faith is often an hindrance to anyone on the quest of knowledge.
Thus, the REAL reason that ancient scholars didn't "worship" Brahma is to resolve such philosophical contradiction that his role as the creator and his primacy as the source of all knowledge created.
The quest for knowledge is itself a desire and an act of veneration of "knowledge" (as a deity). Thus, studying and following the vedas can by itself be considered as a worship of Brahma, and what all Brahmins do. And the evolution of our scriptures is a testament to this - an indirect reference to the importance (and equality) of Lord Brahma within the trinity, by the ancient scholars.
(Our philosophy has stood the test of time because it is based on sound reasoning - not because of what some God did to another God or somebody blessed or cursed someone. Such literal interpretation can really inhibit our spiritual growth.)
(For the spiritually inclined -
This also reminds me of the the approach of ancient Islamic scholars on how to teach their religion. When the young muslims were introduced to their religion, they were taught that "Doubt" is haram (forbidden; something to be avoided; a sin even if done with good intentions).
The reasoning was that due to the limitation of a child's understanding, more emphasis was placed on faith (which can also be described as absence of doubt). But as they become older they were urged to explore more - as prophet Muhammed (pbuh) did in long meditations resulting in the revelation of the holy Quran to him by God.)
