Your question is related to famous Yajnavalkya and Shakalya debate , which which is mentioned in Shapatha Brahmana (11.6.3.11) as well as Brihadaranyaka Upanishad (Chapter 3-Brahmana 9 ).
Some Background of the Story - At the Horse sacrifice of King Janaka of Videha thousands of sages from land of Kuru and Panchala assembled at his place to witness the sacrifice. Janaka by seeing the gathering of large no. of sages at his place and desirous to know who is the Brahman greatest of them all , offered a bounty of thousands of cows with thousands of gold pieces on their horns to the Brahmastha (ब्रह्मिष्ठ) (greatest Brahnana of all) . They started the debate. In between Sage Yajnavalkya who was also present there asked his disciple to carry the bounty of cows. By this act the Brahmanas got angry and started questioning Yajnavalkya challenging his act of taking away the bounty without proving he is Brahmastha.
Many sages asked him the questions and Yajnavalkya defeated them all by answering their questions one of which was Shakalya. He was talking on the behalf of the sages of Kuru-Panchala land. Shakalya asked 1000 questions to Yajnavalkya and Yajnavalkya answered all his questions , then Yajnavalkya asked one question to Shakalya , which he did not know the answer. Yajnavalkya cleared out before asking the question to Shakalya that his life is at stake and he will curse him if he fails to answer. Shakalya failed and by the curse of Yajnavalkya died at the place immediately.
The same story of Yajnavalkya-Shakalya debate is in detail discussed in many other texts like puranas as well providing us lots of other information regarding the incident. For instance Vayu Purana - Chapter 60 -Dilation on the division of vedas - Verse 35 to rest of the chapter is also narrating this story and the death of Shakalya. Brahmanda purana Part 1 - Chapter 34 is entirely related to this topic of controversy between Shakalya and Yajnavalkya.
From the verse no. 57-58 of above chapter of Vayu purana we also get to know that Yajnavalkya answered all the questions of Shakalya and asked him to answer his single question , and said that the stake for this debate is his life and he will curse him if he fails.
Shakalya failed to answer Yajnavalkya's question died at the same place. When Sahkalya died from the curse of Shree Yajnavalkya all those even who were present at the assembly incurred the sin of Brahmana-Slaughter (Brahma-Hatya). All the sages then got worried and went to Lord Brahma , who guided them about how to become free from this curse , which is mentioned in below shlokas of Vayu purana.
शाकल्ये तु मृते सर्वे ब्रह्मघ्नास्ते बभूविरे |
तदा चिन्ता परां
प्राप्य गतास्ते ब्रह्मणोsन्तिकम || Vayu 60 .67 ||
When Shakalya died , those present there incurred the sin of
Brahmana-Slaughter. They approached Brahma ,worried by this sin.
Having understood their difficulty Lord Brahma instructed them to go to a place called Balukeshwara , Malabar Hills near present Mumbai , and to take abath in a pond near there to get rid of this Brahma-Hatya sin. . This is the place where Lord Parashurama installed a linga of sand , below shloka is the Lord Brahma's Instructions to sages.
द्वादशार्कं नमस्कृत्यं तथा वै वालुकेश्वरम् |
एकादशा तथा
रुद्रान्वायुपुत्रं विशेषत: ||
कुण्डे चतुष्टये स्नात्वा ब्रह्महत्या
तरिष्यथ || Vayu 60.69 ||
After bowing before Dvadasarka (the twelve suns) , the deity Valuka
, the eleven Rudras and Vayu Putra (Hanuman) in particular and having
bathed in four kundas or reservoirs or holy pools , you will shake off
(lit. swim across) the (sin) of Brahmanas murder. And the sages
rushed to the place mentioned by Lord Brahma.
उत्तेर्श्वरम् नमस्कृत्यं वाडवानां प्रसादत : |
सर्वेपापविनिर्मुक्तो गतास्ते सुर्यमण्डलं || Vayu 60.71 ||
Having offered obeisance to uttersvara , out of grace of Brahmanas
(Vadavas) , all of them became free from sins and proceeded towards
the solar region.
After following the instructions of Lord Brahma all the sages were relieved from the sin of Brahma-Hatya. So all the sages present there incurred the sin of Brahma-Hatya including Yajnavalkya.
But wait there is another twist to this story of death of Shakalya.
According to Skanda Purana -Vol. 8-Book 3 -Brahma Kanda -Setu Mahatmya -Chapter 48- cessation of Great Sins Incurred by King Sankara - Verse no. 47-52. Sage Shakalya and his wife were killed by King of Pandya land named Shankara while hunting the game. And he did incurred the sin of Brahma-Hatya. Which is mentioned in below shlokas
Sri Suta said
47-52 On being told thus by sages , O Brahmanas , That son of Shakalya
named jangala deposited the bones of his parents in the holy spot of
Ramesvara in the manner mentioned by the sages . he performed Shraddha
and other rites as well.
This is conformed in one other source here which is in Hindi.
Also
Adi Shankaracharya in his commentary of The Bṛhadāraṇyaka Upaniṣad Verse 3.9.26: (second last paragraph) is mentioning that Rishi yajnavalkya cursed Shakalya that he will die at a unholy place at unholy time .
A previous anecdote is here referred to. In (Book XI of the Śatapatha
Brāhmaṇa entitled) the Aṣṭādhyāī[15] there occurs a dialogue between
Yājñavalkya and Śākalya with a similar ending. There Yājñavalkya gave
a curse: ‘“You shall die in an unholy place at an inauspicious time,
and even your bones shall not reach home.” He died exactly like that;
Conclusion - So there are two possibilities exists here Shakalya died by curse of Yajnavalkya immediately OR he died later on by Pandy king Shankara ,with curse involved.
The question is did Yajnavalkya committed Brahma-hatya by killing Shakalya ? lets analyze this with the help of all above with two points.
1) No- if we consider points mentioned by Adi Shankara and story of Skanda Purana he did not committed Brahma-hatya , he just cursed Shakalya and the bait of their debate was Shakalya's life. He did not physically killed Shakalya , so technically no direct Brahma-Hatya and Shakalya died later from the hands of Pandya king and the Pandya kind incurred the sin of Brahma-Hatya.
2) Yes- if we are looking from the angle of story mentioned in Vayu purana , all the sages present at the discussion occurred the sin of Killing a brahmana that means Yajnavalkya also was part of it. Later all the sages got freed from this sin by bathing at a holy spot. So he did committed Brahma-hatya or it happened by his act or curse
So the Answer is both yes and no at the same time with ref. to various sources.
