There is a general opinion that Sripad Adi Sankaracharya established the shanmatha system and also insituted the worship form of panchayatana. Yet there seems to be lot of disagreements even among the well read on that point. Do we have any direct evidence from the Acharya's own works or his immediate successors' works to support the view that he was the one who established the 2 systems?
Panchayatana Puja consists of worshiping the five principle deities of Hinduism viz - Ganapaty, Shiva, Vishnu, Devi (Ambika) and Surya.
Such a worship is already ordained in the scriptures. So, it can not be that Adi Shankaracharya has established the system. During the time he lived probably it was required to be revived. That's why he simply reinstated an already existing Puja system.
For example in Devi Bhagavatam's 9th book there is an entire chapter titled "On the destruction of the fear of the Yama of those who are the worshippers of the Five Devatâs".
Quoting verses from that book's 36th chapter:
8-33. Nârâyana spoke :-- Hearing the questions put forward by Sâvitrî, Dharmarâja remembered S’rî Hari and began to speak on subject that sever the bonds of Karma :-- O Child! O One of good vows! In the four Vedas, in all the books on Dharma, (Smritis) in all the Samhitâs, all the Itihâsas, all the Purânas, in the Nârada Pañcharâtram, in the other Dharma S’âstras and in the Vedângas, it is definitely stated that the worship of the Pañcha Devatâs (the five Devatâs) S’iva, S’akti Visnu, Ganes’a, and Sûrya is the best, the highest, the destroyer of the old age, disease, death, evils and sorrows, the most auspicious and leading to the highest bliss. In fact, the worship of these Pañcha Devatâs is the source of acquiring all the Siddhis (the success) and saves one from going to the hells. From their worship springs the Bhaktic Tree and then and then only the Root of the Tree of all Karmic bonds is severed for ever and ever. This is the step to Mukti (final liberation) and is the indestructible state
And, further we have the following Matsya Purana verse (quoted in Nitya Karma Puja Prakash):
Adityam gananAtham cha devim rudram cha keshavam |
PanchaDaivatyamityuktam sarva karmasu pujayeth ||
Surya, Ganesha, Devi, Shiva and Vishnu - These five deities are to be worshiped in all auspicious occasions.
So, worshiping those five deities is a scriptural injunction.
Regarding the Shanmata system (the worship of six deities) then that is also something prescribed in the scriptures.
For example, the same Purana (in a different chapter) also prescribes the worship of six deities before any Puja. The five deities are already mentioned, the 6th one being Agni.
Quoting from Essence of Devi Bhagavata Purana:
Maharshi Narayana explained to Narada the mehodology of Worship to Ganga in brief: one should have a hearty bath in Ganga, don clean clothes, be seated on the banks of the River to perform the daily Sandhyavandana first and invoke the blessings the Six Devatas viz.Ganesha, Sun, Agni, Vishnu, Siva and Sivani. Do worship Ganesha to remove the obstacles, Surya Deva for good health, Agni Deva for purification, Vishnu for wealth and power, Siva for knowledge and Sivani for salvation. Then, imagine a mental image of Ganga Devi in fulfledged form with ‘Ashta Hasta’ ( Eight hands) fully decorated with a smiling face, ornaments, flower garlands, and armoury and above all an ‘Abhaya Mudra’ or a protective posture of two hands- one to provide security and another for giving away boons. The ‘Dhyanam’ or meditation is to be highly concentrated and serious without mind wavering on any other object or thought, excepting the Image of Ganga, in physical form to facilitate ‘dhyana’ or meditation
However, in Shanmata system, instead of Agni, Karthikeya is worshiped. But note the connection Agni ----> Mars/Angaraka ----> Karthikeya (over ruling deity for Mars).
I am going to start posting my own answer for this question in parts, since I am still reading up on this topic and information from other people I have access to is not easy to get, due to strongly entrenched traditions/ bias.
'He rests with his rays in him'--still Selfhood cannot be ascribed to the sun, on account of his externality (parâgrûpatva). Immortality, &c. also cannot be predicated of him, as Scripture speaks of his origin and his dissolution. For the (so-called) deathlessness of the gods only means their (comparatively) long existence. And their lordly power also is based on the highest Lord and does not naturally belong to them; as the mantra declares, 'From terror of it (Brahman) the wind blows, from terror the sun rises; from terror of it Agni and Indra, yea, Death runs as the fifth.'
I went and looked where this verse was explained and ended up here, the Kathopanisad Bhasya v2.3.3 of Sri Shankaracharya.
bhayādasyāgnistapati bhayāttapati sūryaḥ | bhayādindraśca vāyuśca mṛtyurdhāvati pañcamaḥ || 3 ||
- From fear of him, fire burns; from fear, the sun shines; from fear, Indra and Wind; and Death, the fifth, speeds.
So, it seems counter to logic to claim that Sri Shankaracharya founded, or integrated in to advaita, a matha which had Surya as a supreme deity when he clearly differentiates Surya to be one of the "servants trembling from fear of the master" and not an Ishvara or aspect of Brahman.
Here's a good summary link on the worship of Surya in vedic literature.
I had come across a compilation of exchanges between scholars, published under the name "Sankararum, Vainavamum" (Sri Shankara and Vaishnavism), around 1960s-1970s, which pretty much revolved around this exact question. The entire text is in Tamil, so I will basically post a few snippets with my translation (staying true to the original to the best of my capacity).
Pg 21 of the file, pg 34 of the text: Sri Umapati Jagadisha Sharma writes in a letter: "Sri Vidyaranya wrote that "There are none to respect the statements of Ganapatyas, Kapalikas have run and hidden somewhere, Saivam has turned "ashivam" (I can't put the exact meaning of the word in this context), Arhamatham got denounced, Shaktamatham became illfated, there is no one to nourish Vaishnavism. All this is only because of the merciless attack by Sankaracharya's suktis". The response to this from Brahmasri Varahur Kalayasundara sastri (the scholar who responded on behalf of advaitins) on pg 87 of the text is very much unconvincing.
Now, I couldn't trace out the source for Sri Vidyaranya's statement. But assuming this exists, it certainly comes across as odd that Shankaracharya, as an avatara of Lord Shiva, would actually establish the same 6 schools that which he purportedly decimated.
Pg 48 of the file, pg 51 in the text: Varahur Sri Kalayanasundara sastri makes a statement that they (advaitins, and by extension, Sri Shankara) do not/did not consider the six deities as parabrahman but only amshas of the nirvisesha parabrahman.
Pg 65 of the file, pg 85 of the text: Varahur Sastri also makes a claim that there were originally mathams which had Indra, Brahma and even Vayu as their supreme deity.
That does bring up the question as to why the shanmathas selected only the aforementioned 6 deities, and not any of the other devatas/deities as stated by Varahur sastri.
Sri Varahur Sastri mentions in places that he had published a book by the name of "Sankararum, Shanmathamum" (Sri Shankara and the Shanmathas) in 3 parts, where he claims to have proven that it was indeed Sri Adi Shankara who established the 6 mathams. Unfortunately, I am unable to trace that book as well. (If anyone can direct me to the books, I would be grateful).
Addition 3 (Jan 12, 2019)
Stumbled across another interesting page. This page gives a completely different angle on Panchayatana puja.
The tendency of rapprochement in orthodox religious sphere in pañcadevopāsanā i.e., the worship of five deities as advocated by the Smārtas.
So, basically the author starts out saying that the system would have been initiated to bring about "harmony" among different systems. That is believable.
After stating that the smartha system was initiated by both Shaivas and Vaishnavas, the author gives a jist of the evolution of the system.
The worship of Shiva with Sun, Shakti, Gaṇeśa and Viṣṇu was performed in the Miśra Pāśupata school. This is the same as Smārta Pañcadevopāsanā:
रविं शम्भुं तथा शक्तिं विघ्नेशं च जनार्दनम् | यजन्ति समभावेन मिश्रपाशुपतं हि तत् ||
The later Smārta treatises such as as Smṛtimuktāphala prescribe the daily worship of these five deities for a householder:
आदित्यमम्बिकां विष्णुं गणनाथं महेश्वरम् | पञ्चयज्ञपरो नित्यं गृहस्थः पञ्च पूजयेत् ||
It is sometimes stated that the system in this form was popularized by the Advaita teacher Shaṅkara but it is extremely doubtful.
The author believes that the first step in moving to a pentad (Panchopasana) system was the introduction of the Trinity or the Trimurthis as we know them.
The first stage in the development of the pentad cult was the evolution of trinity composed of Brahmā, Viṣṇu and Shiva.
Then follows the description of temples with different deities as part of the evolving triumvarate or quadrumvarate then to finally panchadevopasana.
There were different versions of Panchayatana, evidently.
The Kilait Cp. of Somavarman in the middle of eleventh century A.D. invokes five deities, Brahmā, Gaṇapati, Viṣṇu, Shakti and Shiva. But this pentad is not of the Smārta variety as Sūrya has been substituted here by Brahmā.
And then he ends with the conclusion thus:
It may, therefore, be concluded that the system of five deities as envisaged by the Smārtas came into vogue by eleventh century A.D. and that it indicates the rapprochment of the Vedic and āgamic tendencies. The views that Pañcadevopāsanā was introduced by Shaṅkarācārya is evidently incorrect.
The conclusion that many, including me, have arrived at is that Sri Shankara could not have instituted such systems when his siddhanta as found in his key works runs counter to the paths. But as it would be with all things in spirituality and philosophy, each one is attracted to what is agreeable to them at that point in time, determined and guided by their karmas and gunas.
I will close my answer here, since most of the other material I have on hand are not in English and translating all those is not going to be practical. Hopefully, if anything, this answer might have kindled the need for a search by others so that Sri Adi Shankara Bhagavatpada's true legacies are brought out.