Lord Varuna, the deity of the Oceans, is a deity who seems to follows the laws and rules very strictly. In the Vedas, he is known as a mighty King and Upholder of Rta, or the worldly order. Refer this page for a detailed description of Varuna as the Upholder of Rta.
In brief, I quote the following:
For instance, it is said;” By the law of Varuna heaven and earth are held apart; the planets rotate in their fixed orbits (ṛtena ṛtam apihitaṃ dhruvaṃ vāṃ sūryasya yatra vimucanty aśvān– RV 5.62.1). ....
Rta causes the rivers to flow into the ocean without over-filling it. Varuna the lord of Rta is the binder. He binds together the deep- space, the space between the earth and yonder, the winds, the clouds and the rays of light.”
1. Primary Reason:
This quality of Varuna as the upholder of Worldly order is reflected in his dialogue to Rama.
O, beloved Rama! Earth, wind ether, water and light remain fixed in their own nature, resorting to their eternal path.
Therefore, I am fathomless and my nature is that it is impossible of being swum across. It becomes unnatural if I am shallow. I am telling you the following device to cross me.
O, prince! Neither from desire nor ambition nor fear nor from affection, I am able to solidify my waters inhabited by alligators.
"O, Rama! I shall make it possible to see that you are able to cross over. I will arrange a place for the monkeys to cross me and bear with it. As far as the army crosses me, the crocodiles will not be aggressive to them. VR.6.25-28
Above, Varuna means that the sea is by nature deep and difficult to cross, so it would be against the laws of nature (Rta) (and interfering with ordinary worldly affairs) to allow an entire army to easily swim across to Lanka without being affected by the sea waves or aquatic creatures.
2. Secondary Motive
Varuna also has a secondary motive in kindling Rama's anger, which he reveals to Rama subsequently. When Rama, who has notched the Brahmastra onto his bow, asks Varuna for a suitable target for the weapon (since now Varuna's ocean domain has been spared from Rama's anger), Varuna replies,
Towards my northern side, there is a holy place. It is well known as Drumatulya, in the same way as you are well known to this world. Numerous robbers of fearful aspect and deeds, having the sinful Abhiras as their chief, drink my waters there. I am not able to bear that touch of those wicked people, the evil doers, O, Rama! Let this excellent arrow with out vain be released over them there. [VR.6.31-33]
(Slightly selfish of Varuna I guess, kindling another's anger to achieve his tasks, but then again as upholder of Rta one supposes he is forbidden from destroying a sect of population for his benefit.)
One may argue that had Varuna appeared to Rama immediately, Rama would not have agreed for a barter of way-over-the-sea for Brahmastra-destruction. Hence, only by making him wait for three days, the Ocean Lord's purpose would have been achieved.
Conclusion: difference between the situation with Hanuman and Rama
This is opposed to his helping hand in Hanuman's task, since asking for Mainaka mountain to rise up and provide a resting spot for Hanuman wasn't interfering with any worldly affairs, so Varuna was just offering the best help he could without affecting the rest of his natural domain.
As a side note, this strict, law abiding nature of Varuna is seen in Krsna's life as well, during two incidents, the first being the abduction of Nanda, described in Bhagavatam 10.28, where even though he knows Nanda wasn't committing a crime, he allowed his servant to drag him to his abode citing rules of Brahma Muhurta and Asuri Muhurta. Of course one may argue that he wanted a darshan of Lord Krsna.
The second incident is after Banasura's battle with Krsna, when Krsna proceeds to the abode of Varuna to secure one of Banasura's cows as per Satyabhama's request, narrated in the Harivamsa, Vishnu Parva, Chapter 127. Here too, instead of directly approaching Krsna as the Lord, Varuna first deals with him as a cattle thief and battles with Krsna and his army. It is subsequently that he decides to call it a truce and requests Krsna to stop fighting, citing that he doesn't wish to break his promise of protecting Banasura's cows. Once again, upholding laws.
