The cause for your confusion is the literal interpretation of Manusmṛti.
See how Medhātithi interprets the same verse (2.23) in his manubhāṣya:
kṛṣṇasārastu carati mṛgo yatra svabhāvataḥ |
sa jñeyo yajñiyo deśo mlecchadeśastvataḥ paraḥ || 23 ||
But the region where the spotted deer roams by nature is to be known as the ‘land fit for sacrificial acts’; beyond that is the ‘land of the Mlecchas.’ (23)
Medhātithi’s commentary (manubhāṣya):
It is not meant that the sacrifices are to be performed on the very spot where the deer roams...
In the present instance there is no direct injunction, such as ‘one should perform sacrifices here (in this country)’; as the injunctive affix is found added to the root ‘to know’ (in the word ‘jñeyaḥ’), and not to the root ‘to sacrifice.’ All that is meant is that the country spoken of is ‘fit for sacrifices’; the meaning being that ‘this country is fit for sacrificial performances’; and this ‘fitness for sacrifices’ is possible even without a direct injunction (of the actual performance). The fact of the matter is that it is only in the countries mentioned that the several sacrificial accessories, in the shape of the kuśa-grass, the Palāśa, the Khadira and other trees, are mostly found; and sacrificial performers also, in the shape of persons belonging to the three higher castes and learned in the three Vedas, are found only in these countries; and it is on the basis of these facts that the countries have been described as ‘fit for sacrifices.’
And this for the simple reason that no land is by itself defective; it is only by association that it becomes defective, just as it is when soiled by impure things. Hence, even apart from the countries designated here as ‘fit for sacrifices,’ if, in a certain place, all the necessary conditions are available, one should perform his sacrifices, even though it be a place where the spotted deer does not roam.
So any part of the Indian subcontinent (and by extension, the whole world?) even places where the spotted deer doesn't roam but has a good supply of sacrificial accessories like kuśa-grass and people who can perform sacrifices (yajñas) is considered Āryāvarta.