The story of Nandi punishing Vyasa is mentioned in this chapter (Kashi Khanda chapter 95) of Skandha Purana. What that chapter describes is, Vyasa sees Saunaka and other sages wearing Tripundras and chanting Rudra Suktas and chanting Shiva names and stating that:
एक एव हि विश्वेशो मुक्तिदो नान्य एव हि ।।
इति ब्रुवाणान्सततं परिनिश्चितमानसान् ।। ९ ।।
विलोक्य स मुनिर्व्यासस्तासर्वान्गिरिशात्मनः ।।
उत्क्षिप्य तर्जनीमुच्चैः प्रोवाचेदं वचः पुनः ।।
Visvesha is one alone. He and no one else is the bestower of salvation.” On seeing them all with their souls in communion with Girisha, Sage Vyasa lifted up his index finger and said thus again.
Now, Vyasa says:
परिनिर्मथ्य वाग्जालं सुनिश्चित्यासकृद्बहु ।।
इदमेकं परिज्ञातं सेव्यः सर्वेश्वरो हरिः ।। ११ ।।
वेदे रामायणे चैव पुराणेषु च भारते ।।
आदिमध्यावसानेषु हरिरेकोऽत्र नापरः ।। १२ ।।
सत्यं सत्यं त्रिसत्यं पुनः सत्यं न मृषा पुनः ।।
न वेदादपरं शास्त्रं न देवोच्युततः परः ।। १३ ।।
लक्ष्मीशः सर्वदो नान्यो लक्ष्मीशोप्यपवर्गदः ।।
एक एव हि लक्ष्मीशस्ततो ध्येयो न चापरः ।। १४ ।।
भुक्तेर्मुक्तेरिहान्यत्र नान्यो दाता जनार्दनात्।।
तस्माच्चतुर्भुजो नित्यं सेवनीयः सुखेप्सुभिः ।। १५ ।।
विहाय केशवादन्यं ये सेवंतेल्पमेधसः ।।
संसारचक्रे गहने ते विशंति पुनःपुनः ।। १६ ।।
“After churning the vast expanse of words and coming to firm conclusions again and again, this alone has been understood (by me that), Hari, the Lord of all, should be resorted to. In Veda, in Purāṇas, in Rāmāyaṇa and in Bhārata, Hari alone and no one else comes in the beginning, middle and end.
This is the truth; this is the truth; this is the truth, thrice asserted. It is not false to assert that there is no scripture other than Veda; there is no God other than Acyuta. The Lord of Lakṣmī, is the bestower of everything. No one else. Lakṣmīśa is the bestower of salvation. Hence Lakṣmīśa alone should be meditated upon. No one else. No one other than Janārdana is the bestower of enjoyment of pleasures here and of salvation hereafter. Hence the Four-armed Lord should always be resorted to by those who desire happiness. Those men of little intellect who serve another (deity) abandoning Keśava, enter the mysterious and painful cycle of worldly existence again and again.
The sages present there then say that Vyasa should go to visit Varanasi where Lord Vishweswara himself is present:
गच्छ वाराणसीं व्यास यत्र विश्वेश्वरः स्वयम् ।।
न तत्र युगधर्मोस्ति न च लग्ना वसुंधरा ।।२६।।
इति श्रुत्वा मुनिर्व्यासः किंचित्कुपितवद्धृदि ।।
जगाम तूर्णं सहितः स्वशिष्यैरयुतोन्मितैः ।। २७ ।।
O Vyāsa, go to Vārāṇasī where Viśveśvara himself is present. There are not seen changing features of the different Yugas there. Nor does it touch the earth. On hearing this, Sage Vyāsa who became angry in his mind, left the place quickly accompanied by his disciples numbering about ten thousand.
Vyasa after visiting several other temples at Varanasi reaches Vishweswara temple and recites the same thing, then Nandi (Shailadi) paralyses both hand and speech of Vyasa:
पुनरूर्ध्वभुजं कृत्वा दक्षिणं शिष्यमध्यगः ।।
पुनः पपाठ तानेव श्लोकान्गायन्निवोच्चकैः ।। ४४ ।।
परिनिर्मथ्य वाग्जालं सुनिश्चित्यासकृद्बहु ।।
इदमेकं परिज्ञातं सेव्यः सर्वेश्वरो हरिः ।। ४५ ।।
इत्यादि श्लोकसंघातं स्वप्रतिज्ञा प्रबोधकम्।।
यावत्पठति स व्यासः सव्यमुत्क्षिप्य वै भुजम् ।। ४६ ।।
तस्तंभ तावत्तद्बाहुं स शैलादिः स्वलीलया ।।
वाक्स्तंभश्चापि तस्यासीन्मुनेर्व्यासस्य सन्मुनेः ।। ४७ ।।
After concluding the dance thus, Vyāsa, the son of Satyavatī, stood in the middle of his disciples and lifted up his right hand. He recited the same verses singing them aloud: “After churning the vast expanse of words and coming to firm conclusions, this alone has been understood (by me) that Hari, the Lord of all, should be resorted to.” All the verses beginning with the above indicating his assertion, Vyāsa recited with the left hand raised. Thereupon Śailādi made the hand paralysed playfully. There was also stupefaction of the power of speech of the excellent sage Vyāsa.
Lord Vishnu then comes there and after telling glory of Lord Visheswara releases speech paralysis of Vyasa:
ततो गुप्तं समागम्य विष्णुर्व्यासमभाषत ।।
अपराद्धं महच्चात्र भवता व्यास निश्चितम् ।। ४८ ।।
तवैतदपराधेन भीतिर्मेपि महत्तरा ।।
एक एव हि विश्वेशो द्वितीयो नास्ति कश्चन ।। ४९ ।।
तत्प्रसादादहं चक्री लक्ष्मीशस्तत्प्रभावतः ।।
त्रैलोक्यरक्षासामर्थ्यं दत्तं तेनैव शंभुना ।।4.2.95.५०।।
तद्भक्त्या परमैश्वर्यं मया लब्धं वरात्ततः ।।
इदानीं स्तुहि तं शंभुं यदि मे शुभमिच्छसि ।।५१।।
अन्यदापि न वै कार्या भवता शेमुषीदृशी ।।
पाराशर्य इति श्रुत्वा संज्ञया व्याजहार ह ।।५२।।
भुजस्तंभः कृतस्तेन नंदिना दृष्टिमात्रतः ।।
वाक्स्तंभस्तद्भयाज्जातः स्पृश मे कंठकंदलीम्।। ५३ ।।
यथा स्तोतुं भवानीश प्रभवाभि भवांतकम ।।
संस्पृश्य विष्णुस्तत्कंठं गुप्तमेव जगाम ह ।।५४।।
ततः सत्यवतीसूनुस्तथा स्तंभितदोर्लतः।।
प्रारब्धवान्महेशानं परितुष्टोतुमुदारधीः ।। ।। ५५ ।।
Thereupon Vishnu came there secretly and spoke to Vyāsa: “O Vyāsa, a great offence has certainly been committed by you. Even I am greatly afraid due to this offence of yours. There is only one Lord of the universe, and he is Viśveśa. There is no one else. I am the discus-bearer, due to his favour. I am the Lord of Lakṣmī due to his power. It was by Śaṃbhu himself that the capacity to sustain the three worlds was granted to me. Due to the devotion to him the greatest Aiśvarya (prosperity, lordship) was obtained by me as a boon from him. If you desire my welfare, eulogize Śaṃbhu. Do not entertain a thought like this to eulogize me on any other occasion, O son of Parāśara.” On hearing this, Vyāsa expressed by means of gestures: “The paralysis of the arm has been brought about by that Nandin by his mere eyes. The stunning of the speech has been caused by my fear of him. Touch the root of my throat so that I shall be able to eulogize Bhavānīśa, the destroyer of worldly existence.” After touching his throat, Viṣṇu secretly went away. Thereupon the son of Satyavatī whose arm had been paralysed in that manner, began to eulogize Maheśāna liberal-mindedly.
Then,now Vyasa sings the famous Vyasa Ashtakam (eight verses) which goes as:
एको रुद्रो न द्वितीयो यतस्तद्ब्रह्मैवैकं नेह नानास्ति किंचित् ।।
यद्यप्यन्यः कोपि वा कुत्रचिद्वा व्याचष्टां तद्यस्य शक्तिर्मदग्रे ।। २६ ।।
Brahman alone is one. There are not many entities here. Hence Rudra is only one. There is none second to him. If there be anyone else anywhere, let him who has the capacity to do so, explain it to me.
यः क्षीराब्धेर्मंदराघातजातो ज्वालामाली कालकूटोति भीमः ।।
तं सोढुं वा को परोऽभून्महेशाद्यत्कीलाभिः कृष्णतामाप विष्णुः ।। ५७ ।।
Highly terrible poison Kālakūṭa with flaming sparks around was born of the Milk Ocean, due to the churning with Manḍara. No one other than Maheśa could endure it. On account of its flames, Viṣṇu turned back.
यद्वाणोभूच्छ्रीपतिर्यस्य यंता लोकेशो यत्स्यंदनं भूः समस्ता ।।
वाहा वेदा यस्य येनेषुपाताद्दग्धा ग्रामास्त्रैपुरास्तत्समः कः ।। ५८ ।।
Śrīpati (Viṣṇu) was his arrow. The entire earth was his chariot. Brahmā, the Lord of the world, was his charioteer. The horses were the Vedas. Who is on a par with Rudra by whom the group of three Puras was burned by the discharge of the arrow?
यं कदर्पो वीक्षमाणः समानं देवैरन्यैर्भस्मजातः स्वयं हि ।।
पौष्पैर्बाणैः सर्वविश्वैकजेता को वा स्तुत्यः कामजेतुस्ततोन्यः ।। ५९ ।।
Kandarpa who was the sole conqueror of the universe through his flowery arrows, viewed Rudra as of the same level as the other Devas. He himself was reduced to ashes. Who is worthy of eulogy other than that conqueror of Kāma?
यं वै वेदो वेद नो नैव विष्णुर्नोवा वेधा नो मनो नैव वाणी ।।
तं देवेशं मादृशः कोल्पमेधा याथात्म्याद्वै वेत्त्यहो विश्वनाथम् ।। 4.2.95.६० ।।
Which man of meagre intellect like me (can) understand Viśvanātha in his true self, because the Veda does not know him? Neither Viṣṇu nor Brahmā, neither mind nor speech know that Lord of Devas.
यस्मिन्सर्वं यस्तु सर्वत्र सर्वो यो वै कर्ता योऽविता योऽपहर्ता ।।
नो यस्यादिर्यः समस्तादिरेको नो यस्यांतो योंतकृत्तं नतोस्मि ।। ६१ ।।
I bow down to him wherein everything is stationed, who is everywhere, who is all, who is the maker, who is the protector, who is the destroyer, who has no beginning, who is the beginning (cause) of everything, who is the sole one, who has no end and who causes the end of everything.
यस्यैकाख्या वाजिमेधेन तुल्या यस्या न त्या चैकयाल्पेंद्रलक्ष्मीः ।।
यस्य स्तुत्या लभ्यते सत्यलोको यस्यार्चातो मोक्षलक्ष्मीरदूरा ।।६२।।
(He is that Lord) the utterance of whose one name is on a par with a horse-sacrifice; by a single bowing down to whom Lakṣmī (glory) was attained by the insignificant Indra; by whose eulogy Satyaloka is obtained and by whose adoration the glory of salvation is not far off.
नान्यं देवं वेद्म्यहं श्रीमहेशान्नान्यं देवं स्तौमि शंभोर्ऋतेऽहम् ।।
नान्यं देवं वा नमामि त्रिनेत्रात्सत्यं सत्यं सत्यमेतन्मृषा न ।। ६३ ।।
I do not know any other Lord than Śrīmaheśa; I do not eulogize another Lord except Śaṃbhu; I do not bow down to a Lord other than the Three-eyed One. This is the truth, the truth, the truth, not a lie.”
Nandi becomes happy on hearing this prayer and blesses Vyasa and also tells about the glory of the hymn Vyashashtaka:
इत्थं यावत्स्तौति शंभुं महर्षिस्तावन्नंदी शांभवाद्दृक्प्रसादात् ।।
तद्दोः स्तंभं त्यक्तवांश्चाबभाषे स्मायंस्मायं ब्राह्मणेभ्यो नमो वः ।। ६४ ।।
When the great sage eulogized Śaṃbhu thus, there was a favourable glance from Śaṃbhu whereby Nandin undid the paralysis of his arm and smilingly said, “Obeisance to all of you Brāhmaṇas.”
नंदिकेश्वर उवाच ।।
इदं स्तवं महापुण्यं व्यास ते परिकीर्तितम् ।।
यः पठिष्यति मेधावी तस्य तुष्यति शंकरः ।।६५।।
व्यासाष्टकमिदं प्रातः पठितव्यं प्रयत्नतः ।।
दुःस्वप्नपापशमनं शिवसान्निध्यकारकम् ।। ६६ ।।
मातृहा पितृहा वापि गोघ्नो बालघ्र एव वा ।।
सुरापी स्वर्णहृद्वापि निष्पापो स्याः स्तुतेर्जपात् ।। ६७ ।।
O Vyāsa, this prayer proclaimed by you is highly meritorious. Śaṅkara becomes pleased with the wise one who will read and recite this. This Vyāsāṣṭaka (‘Eulogy in Eight Verses by Vyāsa’) is to be read and recited in the morning with concentration. It makes evil dreams and sins subside. It gives the devotee (salvation in the form of) vicinity to Śiva. By repeated utterance of this prayer, all the sinners become rid of their sins, such as a slayer of mother, a slayer of father, a killer of cows, a killer of boys, an imbiber of liquor, a stealer of gold etc.