These are two different issues that Sri Rama is addressing.
Using the translation of Sri Bibek Debroy -
Abandoning eternal dharma, you had intercourse with your brother’s wife. The great-souled Sugriva is still alive. Ruma is like your daughter-in-law. Falling prey to desire, you have committed a wicked act by having intercourse with her. O ape! You followed kama and transgressed dharma. You have touched your brother’s wife and I have accordingly punished you.
Rama killed VAli because of VAli's behavior towards the wife of Sugriva. Rama is implying that rules of dharma pertaining to not having intercourse with Sugriva's wife when Sugriva is alive, are also applicable to apes.
Later, Rama says -
We are subject to our own norms and have thought of dharma when killing you. Hiding themselves, men are seen to capture a large number of animals with snares, nooses and various other hidden means. This is irrespective of whether they are running away, terrified, roaming around without fear, distracted, undistracted or retreating. Men who desire flesh will certainly kill them and there is no sin in this. Therefore, royal sages who know about dharma engage in hunting. O ape!
Accordingly, using an arrow, I have slain you in an encounter. Whether you are fighting or you are not fighting, you are no more than an ape.
Rama killed VAli indirectly because VAli is just an ape and rules of combat do not apply to animals like apes (when humans kill them).
So, in summary -
Rules of not engaging in adhArmic kAma apply to apes like VAli.
Rules of combat do not apply for humans killing animals (including apes like VAli).
Edits responding to comments
Among vAnara-s, only VAli was treated as sub-human by Sri Rama. This is because of the adharma of VAli.
Sri Rama had great respect and concern for vAnara-s. At the end of the war with Ravana, Sri Rama requests Indra to bring back all the dead vAnara-s to life.
...... the great Indra, the chastiser of Paka, spoke in an extremely happy voice to Raghava, who was standing there, his hands joined in salutation. ‘O Rama! O scorcher of enemies! Your seeing us should not be fruitless. I am full of affection for you. Tell me what you desire.’ Kakutstha was addressed in this way. With his brother, Lakshmana, and his wife, Sita, he joined his hands in salutation and said, ‘O lord of all the gods! O supreme among eloquent ones! If you are full of affection for me, I will tell you. Please act accordingly and make my words come true. There are valiant ones who have gone to Yama’s abode on my account. Let all those apes regain their lives and stand up. Devoted to me and seeking to ensure my pleasure, they did not think about death. Through your favours, let them be united again. This is the boon I ask for. Let them be free of their pains. Let them be free of their wounds. Let them be full of strength and manliness. O one who grants honours! I desire to see the golangulas and the Indras among the bears. Wherever the apes are, let there be sparkling rivers and the best of roots and fruits, irrespective of the season.’