This section of the Manusmriti on wisdomlib.org talks about what constitutes adultery.
In this section, it is clearly said that for a man, adultery constitutes engaging in sex with the wife of another man:
Those men who are addicted to intercourse with the wives of other men, the king shall banish after having branded them with terror-inspiring punishments.—(352)
A man who engages in secret conversation with the wife of another person,—if he is one who has been previously accused of similar offences,—should receive the penalty of the ‘first amercement.’—(354)
He who converses with ‘another’s woman’ at a watering place, or in a wilderness, or in a forest, or at the confluence of rivers,—incurs the guilt of ‘adultery.’—(356)
Nārada (14.6).—‘Indecent assault on another man’s wife is called violence of the highest order.’
One should not converse with the wives of other men, when forbidden. If, on being forbidden, he does converse, he becomes liable to be fined one ‘suvarṇa.’—(361)
Nowhere in this section is there a verse that explicitly says having sex with women other than your wife is a sin.
So, my question is, is it also adultery for the man when that man has sex with unmarried, promiscuous women?
This possible verse from the Narada Smriti permits sex with unmarried, promiscuous women:
12.78 Intercourse is permitted with a wanton woman, who belongs to another than the Brahman caste, or a prostitute, or a female slave, or a female not restrained by her master (nishkâsinî), if these women belong to a lower caste than oneself; but with a woman of superior caste, intercourse is prohibited.
But this doesn't mean any wanton women at any time, as there are caste and ashrama restrictions.
However, Medhatithi is of the opinion that it is a sin:
In the case of women who have not been wedded by any one, and live by prostitution, it is doubtful whether or not an expiation is necessary for having intercourse with them.
“What then is the right view on this point?”
The right view is that expiation is necessary.
Because the restriction (that one should he devoted to his own wife) has been directly enjoined, and expiation has been declared to be necessary in the case of one’s omitting to do what has been enjoined (11.41). ... The case of the ‘wanton’ woman has been already explained, and the prostitute also is an ‘unchaste woman.’
But the injunction "be devoted to your own wife" could simply be prohibitive to mean "avoid wives of other men."
Or, if the injunction "be devoted to your own wife" is actually meant to exclude all women except your own wife, the Narada Smriti verse that is permitting sex with promiscuous, unmarried women could only apply to unmarried Grihastha men, since if you don't have a wife, then you aren't bound by that injunction.
And a man actually enters Grihasthashrama when he takes his Samavartana bath, which is the final bath taken once you graduate from Brahmacharya.