Karna very well understood that he is the son of Pandu (in theory) and Krishna is talking about securing Dharma back. However, he had following reasons:
- Kunti abandoned him without thinking about his well being
- He was raised by Adhiratha's & his wife RAdhA in a Suta traditions, hence his loyalty was not bound with PAndava-s
- He enjoyed uninterrupted kingdom for years under Duryodhana & was
obliged by his friendship
- Karna was a good match for 1:1 combat with Arjuna; and hence he was the major confidence for Duryodhana to prepare his armies to fight against powerful PAndava-s
In Karna's own words from Udyoga Parva:
Karna said, 'Without doubt, O Kesava, thou hast said these words from thy love, affection, and friendship for me, as also in consequence of thy desire of doing me good, O thou of Vrishni's race. I know all that thou hast said unto me. Morally, I am the son of Pandu, as also in consequence of the injunctions of the scriptures, as thou, O Krishna, thinkest. My mother, while a maiden, bore me in her womb, O Janardana, through her connection with Surya. And at the command of Surya himself, she abandoned me as soon as I was born. Even thus, O Krishna, I came into the world. Morally, therefore, I am the son of Pandu. Kunti, however, abandoned me without thinking of my welfare. The Suta, Adhiratha, as soon as he beheld me, took me to his home, and from her affection for me, Radha's breasts were filled with milk that very day, and she, O Madhava, cleansed my urine and evacuations. How can one like us, conversant with duties and ever engaged in listening to scriptures deprive her of her Pinda? So also Adhiratha of the Suta class regardeth me as a son, and I too, from affection, always regard him as (my) father. O Madhava, that Adhiratha, O Janardana, from paternal affection caused all the rites of infancy to be performed on my person, according to the rules prescribed in the scriptures. It is that Adhiratha, again, who caused the name Vasusena to be bestowed upon me by the Brahmanas. When also I attained to youth, I married wives according to his selections. Through them have been born my sons and grandsons, O Janardana. My heart also, O Krishna, and all the bonds of affection and love, are fixed on them. From joy or fear. O Govinda. I cannot venture to destroy those bonds even for the sake of the whole earth or heaps of gold. In consequence also of my connection with Duryodhana of Dhritarashtra's race, I have, O Krishna, enjoyed sovereignty for thirteen years, without a thorn on my side. I have performed many sacrifices, always however in connection with persons of the Suta tribe. All my family rites and marriage rites have been performed with the Sutas. Obtaining me, O Krishna, Duryodhana hath, O thou of Vrishni's race, made this preparations for an armed encounter and provoked hostilities with the sons of Pandu. And it is for this, O Achyuta, that in the battle (that will ensue), I have been chosen as the great antagonist of Arjuna to advance against him in a single combat. For the sake of death, or the ties of blood, or fear, or temptation, I cannot venture, O Janardana, to behave falsely towards the intelligent son of Dhritarashtra. If I do not now engage in a single combat with Arjuna, this will, O Hrishikesa, be inglorious for both myself and Partha...
In subsequent conversation, Karna also suggests to Krishna not to disclose the secret of him being son of Kunti. Otherwise it would cause Yudhishtira not to accept won kingdom. However that is not the part of the question, so not adding that text.