Before answering your question, we need to step back a little and dig into the Khāṇḍava-dahana episode, which, undoubtedly is the most controversial event in the life of the Pāṇḍavas and ask ourselves the following questions.
- Who exactly is Takṣaka? Is "it" really a snake or is "he" a person or is it the name of a forest-dwelling tribe?
- What were Kṛṣṇa and Arjuna really doing near the Khāṇḍava forest? Why did they burn down an entire forest, and barring a few animals (individuals?) incinerate most of its inhabitants?
This is the official account or what the most recent critical edition of the Mahābhārata would have us believe to justify the Khāṇḍava-dahana:
He then spoke to Arjuna and Vasudeva of
the Satvata lineage. "You two, who are now so near the
Khandava tract, are supreme in the world. I am a voracious
brahmana who always eats unlimited quantities. O descendants
of Vrishni and Pritha! I beg you. Give me enough food to satisfy
myself." Having been thus addressed, Krishna and Pandava told
him, "Tell us what kind of food will satisfy you. We will try to
bring it for you."
Having been thus addressed, the illustrious brahmana then
told the warriors, who had asked him about the kind of food that
should be prepared for him, "I do not eat ordinary food. Know
me to be the fire. Therefore, seek to bring me the food that is
appropriate for me. This Khandava tract is always protected by
Indra. So I am unable to burn it down because that great-souled
one protects it. His friend, the naga Takshaka lives there with his
kin and is protected by the wielder of the vajra and many other
beings are also incidentally protected. Though I always wish to
burn it down, I cannot do so because of Shakra's energy.
Whenever he sees me ablaze, he pours down floods of rain from
the clouds. Though I earnestly wish to consume it, I cannot thus
burn it down. Since the two of you are skilled in the use of arms,
I have now come to you for help. I will now be able to burn
Khandava down and that is the food I desire from you. You know
about supreme weapons. Restrain the showers on all sides.
Restrain all the creatures."
Ch. 215, Khandava-daha Parva, The Mahabharata: Volume 2, Bibek Debroy
This is how the story concludes (officially):
Even the greatest of
beings could not look upon the invincible
Arjuna in battle, not to speak of engaging him
in a fight. Like the god of death himself, he
pierced one with a hundred arrows and a
hundred with one, and dead, they descended
into the flames. They found no refuge along
the banks, or in the uneven plains, or in the
abodes of the ancestors and the gods. The
heat increased and thousands of herds of
beings cried out loudly in pain. Elephants,
deer and birds cried out and the sound scared
those who lived in the Ganga and the ocean.
No one dared gaze at the mighty-armed
Arjuna and the immensely strong Krishna, let
alone fighting with them. With his chakra,
Hari slew rakshasas, danavas and nagas and
those who ventured along solitamy paths. The
heads and trunks sliced with the force of the
chakra, the giant bodies fell into the mouth of
the blazing fire. Aided by the flesh, torrents of
blood and fat, the flames rose up into the sky,
without a trace of smoke. Agni's eyes blazed,
his tongue blazed and his wide-open mouth
also blazed. The hair stood up, drinking up
the fat of life, the eyes were tawny. The fire
fed on the nectar that Krishna and Arjuna had
provided and was extremely happy, satiated
and contented.
Ch. 219, Khandava-daha Parva, The Mahabharata: Volume 2, Bibek Debroy
The following is how Irawati Karve interprets the whole incident in her book Yugānta. She wrote a whole chapter The Palace of Maya on this topic, so I'm only quoting a tiny bit here:
Krishna and Arjuna were great warriors. They a
fought and won many battles. But in none of these
battles did they gain any land by conquest. The
Kshatriya life as presented in the Mahabharata had a
certain definite pattern. Each known house had its
small territory which passed from father to son. Wars
were fought, tribute was demanded, but no Kshatriya
house was deprived of its kingdom. An enemy was
spared if he asked for mercy. If he fought and was
killed, his son was put on the throne. A Kshatriya never
killed women and children. Nor was he supposed to
put to the sword any defenceless person. His most
sacred duty was to defend the helpless. The charge that
he had not done so was the worst that could be made
against him.
The need for expansion explains the burning of the
forest, but the question still remains: why was it burned
so mercilessly? There is a very curious contradiction in
the narration. When Agni first appeared, he said he
wanted to burn the forest. No specific mention is made
of his wanting to feed on the creatures in it. But when
we come to the end of the narration, we are told that
Agni went away satisfied with all the flesh and fat he
had devoured.
Moreover, this forest was not merely a forest with birds
and animals in it. We are told that Takshaka, the king of
the Nagas, lived there. But who were the Nagas? The
word naga is generally used for serpents. However, in
the Mahabharata, the Nagas seem to be human beings.
The Mahabharata also mentions a bird woman, who
had children from a Brahman, living in the same forest.
The bird might be the clan name of certain people
living there. In the same way, many of the animals may
not have been animals at all but people belonging to
clans having animal names. But only regarding the
Nagas is the word raja (king) used. Apparently the
Nagas represented the ruling class. The Mahabharata
has given the names of the various Naga rajas belonging
to different regions.
...
There were rules which applied to all animals, but
apparently no rules which applied to all human beings.
If you spared an animal today, you could always kill it
tomorrow. But if you spared a human being — even to
make a slave out of him — he would in the course of
time acquire certain rights. There was indeed great
danger in sparing the lives of those who owned the
land. Krishna and Arjuna, therefore, must have felt the
necessity of completely wiping out the enemy.
...
The burning of Khandava starts with the request of
Agni who had come in the form of a Brahman. It is
implied that being Kshatriyas, Krishna and Arjuna
could not refuse. Even this excuse is flimsy. Not every request of a Brahman was fulfilled by the Kshatriyas.
The Brahman Parashurama had ordered Bhishma to
marry Amba; Bhishma had refused. In the burning of
Khandava no rules of conduct seem to have been
observed. The sole aim was the acquisition of land and
the liquidation of the Nagas. But the cruel objective
was defeated. Just as Hitler found it impossible to wipe
out a whole people, so did the Pandavas. All that they
gained through this cruelty were the curses of hundreds
of victims, and three generations of enmity.
She revisits the topic in the chapter Paradharmo Bhayavahah:
Into the story of Takshaka's curse, too, is woven a
long, monotonous narrative about Brahmans.
Parikshita, when out hunting, came across a Brahman
in deep penance. As a joke, he hung a dead snake
around the Brahman's neck. A little later, the Brahman's
son came there and got very angry at this practical joke.
He cursed the king that in a few weeks' time he would
die of snake bite. When the Brahman woke from his
deep meditation, the son told him what had happened.
The Brahman scolded him for thus giving in to anger,
and, as he knew an antidote to snakebite, he hurried to
Hastinapura to save the king. On the way, Takshaka
met him and cunningly turned him back, thus
preventing him from saving Parikshita. Actually Arjuna,
the grandfather of Parikshita, had, without provocation,
burned the Khandava forest and massacred the
Takshakas, a Naga clan. A Takshaka later killed
Parikshita. Janamejaya, the son of Parikshita, in turn,
wrought great destruction among the Nagas. It is a
straightforward story of a three-generation feud. The
lengthy rigmarole about Brahmans seems to be a later
interpolation.
The late Professor V. S. Sukthankar has pointed out
that the Mahabharata saga came into the hands of the
Bhrigus, a Brahman clan. These Brahmans inserted the
stories of their own family into the narration of the
Mahabharata. All the Brahman stories referred to above
are part of these later interpolations. They have no
relationship whatsoever with the original story of the
Mahabharata. We can, therefore, dismiss them. If all
these accretions are dropped, the Mahabharata gains in
beauty, economy, and movement.
To answer your question:
Why is there such a glaring difference between the two scriptures?
If one accepts Karve's above interpretation then it's only but natural for later writers and narrators to weave stories upon stories to cover up the facts of the Khāṇḍava-dahana and its obvious fallout (the death of Parīkṣit) in whatever way they felt comfortable.
Personally, Karve's reasoning seems more logical to me so neither Mahābhārata nor Bhāgavata's account is the correct narrative of what actually happened.