In the Shanti Parva of the Mahabharata, Bhishma gives advice to Yudishthira and the Pandavas concerning how to be a good king and how to be good person, while he's lying on a bed of arrows after the end of the Kurukshetra war. But in this chapter of the Shanti Parva, before he speaks to Bhishma Yudhisthira first speaks to the sage Vyasa, about various kinds of sins and how to repent for them. In particular, Vyasa says that although killing a Brahmana is unusually one of the worst sins, that does not apply to Brahmanas who are so evil that they take up arms in battle (which Brahmanas are prohibited from doing):
If a Brahmana well acquainted with the Vedas takes up arms and rushes against thee in battle for killing thee, thou mayst proceed against him for taking his life. By such an act the slayer does not become guilty of the slaughter of a Brahmana. There is a mantra in the Vedas, O son of Kunti, that lays this down, I declare unto thee only those practices that are sanctioned by the authority of the Vedas. One who slays a Brahmana that has fallen away from his own duties and that advances, weapon in hand, with intent to slaughter, does not truly become the slayer of a Brahmana. In such a case it is the wrath of the slayer that proceeds against the wrath of the slain.
My question is, what is the "mantra in the Vedas" that speaks of killing Brahmanas who have taken up arms In battle?
As I discuss in this answer, each of the four Vedas each of the four Vedas consists of four portions: Samhitas, the core part of the Vedas which consist of mantra heard from the gods; Brahmanas, which provide instructions on the proper conducting of important rituals; Aranyakas, which provide a guide to rituals meant for forest-dwellers and hermits; and Upanishads, which consist of conversations between teachers and students which clarify the philosophical message of the Vedas. So if it's a mantra it would presumably be in the Samhitas of the Vedas. But the Samhitas focus on praising the gods, not on discussing what humans should or should not do. So I assume Vyasa is just using mantra in a loose sense of "verse in the Vedas".
So are there any verses in the Brahmanas or Aranyakas of the Vedas which sanction killing Brahmanas who have taken up arms in battle? That's the part that provides instructions for humana, kthough it mostly revolvea around Yagnas. If it helps, here is Vyasa's statement to Yudhisthira in Sanskrit:
17 pragṛhya śastram āyāntam api vedāntagaṃ raṇe |
jighāṃsantaṃ nihatyājau na tena brahma hā bhavet ||
18 api cāpy atra kaunteya mantro vedeṣu paṭhyate |
veda pramāṇa vihitaṃ taṃ dharmaṃ prabravīmi te ||
19 apetaṃ brāhmaṇaṃ vṛttād yo hanyād ātatāyinam |
na tena brahma hā sa syān manyus taṃ manyum ṛcchati ||
Assuming Vyasa is actually saying the Vedic verse in the passage, it would probably be verse 19, since Vyasa first says that this is confirmed in a verse in the Vedas and then he says verse 19.
So is verse 19 found in the Vedas at all? Vyasa is the one who compiled the Vedas, so he would certainly know what's in it.