Humans have been naturally endowed with the ability to communicate verbally and thereby replicate these sounds with all parts of their mouth. That is to say in order to produce these sounds all parts of the mouth ranging from the tongue, throat, nasal passage, head, palate, teeth and even lips must be used for correct pronunciation. The points of pronunciation of sounds have been explained by Panini in his Ashtadhyayi Sutras, a view of which can be found on the following website-
With reference to the articulation sutras of Panini (all of which can be found on the above website), I am citing only those which are necessary to pronounce ॐ नमः शिवाय
The Pranava Mantra OM (ॐ)is a combination of three letters viz अ (a), उ(u) and म(m) and thereafter a slight nasal vibration referred to as ardhamatra. These parts of the Pranava have been described as a beautiful swan in the Nādabindūpaniṣad, a yoga Upanishad of the Rigveda. The relevant verse is:
अकारो दक्षिणः पक्ष उकारस्तूत्तरः स्मृतः।
मकारं पुच्छमित्याहुरर्धमात्रा तु मस्तकम् ॥
Meaning: अ is the right wing of the swan whereas उ is its left wing. The म letter is the tail of the swan and the nasal vibration i.e. the ardhamātrā is it’s head.
Now coming to the relevant Panini Sutras which speak of the articulation of sounds we need to see the Sutras which talk of the place of articulation of ओ and म. As per the below Sutra:
ओदौतौ: कण्ठोष्ठम् । (ओ and औ are pronounced using a combination of the throat (कण्ठ) and lips (ओष्ठ))
It is pertinent to note here that ओ in Sanskrit is considered as a combination of two sounds i.e. अ and उ. The Sutras for pronunciation of अ and उ say that अ is pronounced through the throat and उ using a rounding movement of the lips. They can be found on the abovementioned website. ओ being a combination of अ and उ is naturally pronounced through the throat as well as rounding of the lips. This is in consonance with the aforementioned Sutra.
उपूपध्मानीयानाम् ओष्ठौ। (पवर्ग pronounced using the lips) and ञमङणनानां नासिका च। (the last letters of each Varga, in our case म uses the nasal passage as well)
म will be pronounced by shutting the lips and letting the sound flow through the nasal passage. Thereafter there will be a slight sounding of the ardhamātrā or prolonged nasal sound
Now we proceed further to the other parts of the mantra
लृतुलसानां दन्ता:। (all letters of तवर्ग shall be pronounced using the teeth) and ञमङणनानां नासिका च। (the last letters of each Varga, in this case न, uses the nasal passage as well)
न will accordingly be pronounced by hitting the tongue against the teeth and letting sound flow from the nasal passage. Since it is not only न् which is to be pronounced but न (न् + अ) the tongue will move down from the teeth and leaving the lips open, the अ sound will emanate from the throat.
We again encounter म which will be pronounced as stated above. However this time, unlike OM (ओम्) is not म् but an अ follows ( म = म् + अ) and thus after shutting to release the म sound, the lips would have to opened to get out the अ sound.
Next comes the visarga or two dots after म, which as per:
अकुहविसर्जनीयानां कण्ठ:। (अ, कवर्ग, ह, विसर्ग are from the throat)
Is to be said using a gush of air which comes from the throat.
Now a property of the visarga is such that gush of air is supposed to be the same as that of the previous vowel.
For example suppose we have to pronounce the word पूर्वेभिः, it would be pūrvēbhiHi.
Similarly in नमः, the gush of air would have to take on the form of अ. It would be pronounced as namaHa. Since while pronouncing a, the lips are open and sound comes from the throat, so also it would apply to नमः
इचुयशानां तालु। (इ, चवर्ग, श and य are pronounced using the palate of the mouth)
According शि (श् and इ) would resound when the tongue hits the palate. Further to produce शि, there would be a minor backward stretch on the lips and the would be open.
similar movements to pronounce य in शिवाय , i.e. use the palate for य् and lips open to let out अ form the throat
And finally as per:
व् will be pronounced where the teeth hit the lower lip, and thereafter to pronounce आ (a longer form of अ), the throat would have to be used, letting the lips open again, after they were bit by the upper teeth.
From the above we can see that chanting mantras for correct vibrations requires all parts of the mouth. Keeping the lips closed would muffle essential sounds and therefore the vibrations they produce in the body. Therefore the lips also must essentially be continuously used to chant any mantra, including ॐ नमः शिवाय.
Hope this is the answer you were looking for.