My question is that the Varah incarnation of Vishnu took out the earth from the sea, but the sea is in the earth, then how was it taken out of the sea?
The Shiva Purana says that Hiranyaksha took the Earth to Patala.
Once, Hiranyaksha did penance for getting a son from Lord Shiva. Lord Shiva told him to accept Andhakasura as his son.
After obtaining the son, the demon took the Earth to Patala and Lord Vishnu, taking the form of a boar, fought with the daitya and put the Earth back to its place.
On hearing these words of the king, the kind-hearted Śiva was satisfied and spoke thus—“O ruler of Daityas, there may not be a son born of your semen. But I shall grant you a son.
My son Andhaka has a prowess equal to yours. He cannot be defeated by any. You choose him as your son. Cast off your distress and accept him as your son.”
After saying this, the delighted lord gave the son to Hiraṇyākṣa. Śiva, the great soul, the primordial lord of Bhūtas, the destroyer of Tripuras, the fierce god went away, accompanied by Pārvatī.
After getting a son from Śiva that Daitya circumambulated Śiva and worshipped him with many hymns. Joyously the noble Asura returned to kingdom.
42. Having obtained a son from Śiva, the demon of great and fierce valour conquered all the gods and took the earth to Pātāla.
43. Then the gods, sages and the Siddhas propitiated Viṣṇu of infinite vigour in the form of a Boar that constituted all sacrifices and all beings and was terrific in form.
44-46. He split the earth by beating and striking with his snout and entered Pātāla. He powdered hundreds of Daityas with his nose and the formidable curved fangs. He smashed the armies of the Asuras by kicking with his legs dazzling like lightning. He had a wonderfully fierce refulgence. With his Sudarśana dazzling like a crore of suns he chopped off the burning head of Hiraṇyākṣa and reduced the wicked Daityas to ashes. He was then delighted to crown his son Andhaka as the king of Daityas.
47.He returned to his abode. He lifted up the earth from the Pātāla by means of his fangs. He sustained the Earth as before.
The Padma Purana also talks about this incident indirectly. It says that the demon took the Earth and entered the lower world indicating that it could be Patala or some other like Rasatala.
That Hiraṇyākṣa of an immeasurable body and very haughty removed, with his thousands of arms, the earth along with mountains, oceans, islands, and all living beings. Having removed it out and put it on his head he entered the lower world. Then all the hosts of gods, afflicted by fear, cried.
15-19. They sought the shelter of Nārāyaṇa, Viṣṇu. Then knowing that wonder, he, Viṣṇu, the holder of a conch, a disc, and a mace, took up the Boar-form, existing everywhere and having no beginning, middle or end. The highest lord full of everything, having hands and feet on all sides, having large fangs and arms, struck the demon with one fang. The mean son of Diti, with his huge body pounded, died. Seeing the earth fallen (from the demon’s head), he lifted it with his fang, and putting it on Śeṣa’s head as before, took up the form of a Tortoise.
So we can say that Hiranyaksha must have taken the Earth to some lower world.
You can read the source for better understanding which I'm providing below:
Dharmarāja said :-...By Whose command the wind supports the water; the water supports the tortoise; the tortoise supports the Ananta and the Ananta supports the earth; the earth supports the oceans, mountain and all the jewels. The earth is of the nature of forgiveness, i.e., endures all. For this reason all things, moving and non-moving, rest on Her and again melt away in Her...(Srimad Devi Bhagwatam Maha Puranam 9:38:7-79).
Taittiriya 2.1.1. Matches the description here.:
तस्माद्वा एतस्मादात्मन आकाशः सम्भूतः । आकाशाद्वायुः । वायोरग्निः । अग्नेरापः । अद्भ्यः पृथिवी ।
From That, verily,—from This Self (Atman)—is ākāśa (ether) born; from ākāśa, the air; from the air, fire; from fire, water; from water, earth;.
I hope this clarifies all your queries.