Let me share my personal view, the central tenet of Vivaha is that, the husband and wife will together pursue to goals of Dharma, Artha, Kama, and Moksha.
So, adultery was considered as adharma, because a. it involved cheating, b. it involved violation of marital vow of pursuing purushartas together.
Manu Smriti (9.29-30) states - "She who, controlling her thoughts, speech, and acts, violates not her duty towards her lord, dwells with him (after death) in heaven, and in this world is called by the virtuous a faithful (wife, sadhvi).But for disloyalty to her husband a wife is censured among men, and (in her next life) she is born in the womb of a jackal and tormented by diseases, the punishment of her sin."
Now, consider a case of extra-marital affair with mutual consent. The question to ask is, whether the consent is really mutual? Whether the spouses are really ok with this arrangement? Now, assuming that the arrangement is mutual and without any grievance, then definitely, there is no issue of cheating from such an arrangement.
Regarding the second point of Purushartas, though at a first sight it does appear that there is a violation as both the partners are not fulfilling rati/Kama with each other.
Manu Smriti (3.60) says that husband and wife should please each other. Further, in Manu Smriti (9.101-102) it is stressed that, living together happily and faithfully with each other is the highest duty of a married couple. And both the partners must exert themselves to live happily together and pursue all actions together.
Therefore, it is clear that the the scriptures say that husband and wife should be dedicated towards each other and should be loyal as well. Fidelity in all sense of the word is upheld.
Yet, on a deeper consideration, it can be easily argued that, both partners have mutually arrived at this arrangement of fulfilling their own sexual desires. So, in this sense, they are still pursuing Kama together itself.
But, the question is, does the arrangement really work? Is there a guarantee that any of the spouse does not develop emotional relationship with another person? Is it practical to maintain a strictly casual extra-marital relationship without any love or attachment? Is it really possible to continue to love your partner in every way including sexual even after having casual affairs?
If the answer to these question is affirmative that the casual affairs do not affect the couple or their love, relationship, or their duty towards each other, and if such affairs indeed help a couple to come even closer, then there is no fault on the count of purusharthas.
But, there is one more issue, that of Sadharana-dharma that is common to everyone. In Manu Smriti (verse 10.63) - Manu lists Indriya nigraha as one of the common duties.
And there is a definite violation of this. Because, the partners are seeking sexual pleasure outside marriage even after deriving the same pleasure within marriage.
On the other hand, if there is no sexual pleasure within the marriage, it raises other issues related to purusharta. There is a failure on the part of couple to make the marriage work! But that's a different case altogether.
In my opinion, if the extra-marital arrangement is really mutual and without any grievances, and it does not hamper your relationships or your duties, then there is no adharma committed on the count of adultery, but an adharma on the count of non-performance of Indriya Nigraha does exist.