The story is told in this chapter of the Uttara Kanda of the Ramayana. Ravana is going around in the Patalalokas, trying to conquer everything in sight, when he comes across a palace, which he soon discovers to be the palace of the demon Mahabali. As I discuss in this answer, after Vishnu's incarnation Vamana the dwarf defeated Mahabali, Mahabali was pushed down to Sutalaloka, one of the seven Patalalokas, and Vamana became his guard. So Ravana offers to free Mahabali from his captivity under Vishnu, but Mahabali laughs and says that Vishnu is invincible. He asks Ravana to move a giant earring nearby, which belonged to Mahabali's great-grandfather Hiranyakashipu:
Having said this [Bali] the lord of Danavas again spoke unto [Ravana] the lord of Rakshas "O hero, O thou gifted with great strength, take that flaming discus which thou beholdest and come to my side. I shall then relate to thee the means of eternal liberation. Do what I have told thee, O thou having long arms. Delay not O Ravana." Hearing this the highly powerful Raksha proceeded, laughing, O descendant of Raghu, where that celestial kundala was. Ravana, proud of his prowess, easily took it up but could not move it by any means. And being ashamed that highly powerful one again attempted. As soon as it was uplifted the Rakshasa dropped down on ground, bathed in a pool of blood, like unto an uprooted Sala tree. In the meantime there arose a sound from Pushpaka, and the councillors of that lord of Rakshasas cried aloud. Regaining his sense that Rakshas rose up in a moment and lowered his head in shame.
Bali said to him "Do thou come, O foremost of Rakshasas and hear my words. O hero, the Kundala,crested with jewels, which thou didst assay to take up, is an ornament for the ear of one of my forefathers. This fell here on the ground, O thou gifted with great strength; another Kundala was thrown on the summit of the mountain. Besides these Kundalas his crown was also cast of on the ground before the altar during the encounter. Formerly none bore enmity towards my ancestor Hiranyakasipu, Time, death, or illness. He had no death during the day, night, evening or morning. O foremost of Rakshasas, he did not experience death from any weapon whatsoever. He created a dreadful enmity with Prahlada. This conflict having taken place with the high souled and heroic Prahlada there rose up a dreadful figure of Narasimha, a terror unto all, O foremost of Rakshasas. And that terrible figure casting his looks about, all the worlds were overwhelmed. Thereupon taking him up with his arms he destroyed him with his nails. The person, who is standing at the door, is this supreme Vasudeva, void of passion. I shalt now relate to thee the words of that supreme God; do than hear, if thy heart is filled with spiritual thoughts. The person, who is standing at the door, hath brought into subjection, in thousands of years, a thousand of Indras, an Ayuta of Devas and hundreds of great Rishis."
The implication is that if Ravana cannot even move Hiranyakashipu's earring, how can he possibly defeat Vishnu, who so easily killed Hiranyakashipu himself? Mahabali then praises Vishnu:
He is the lord Hari, Narayana, the protector of the three worlds. He is Ananta, Kapila, Jishnu and the highly effulgent Narasimha. He is Kratudhama, Sudhama, and hath the dreadful mace in his hand. He is like unto the twelve Adityas, Purana and the excellent Purusha ;he is like unto the red clouds the lord of gods and the best god. O thou having long arms, he is encircled by flames, a great devotee and fond devotees ;this lord preserveth the Universe and hath created it. And he, gifted with great strength, brought about destruction in the form of Time; and this Hari, with a discus in his hand, is sacrifice and is being worshipped in sacrifices. He is at one with all gods all beings, all worlds and all knowledge. He is all forms, the great form, Baladeva and hath long arms; he slayeth heroes, hath heroic eyes, is imperishable and the preceptor of the three worlds. All these sages, who long for final liberation, meditate upon him he, who can know this Purusha, is not sullied with sins. By remembering him, hearing of him and worshipping him, every thing can be obtained.
By the way, this chapter doesn't mention Ravana being the next birth of Hiranyakashipu, that's described in the Srimad Bhagavatam as I discuss here. Also there's a very good reason why Ravana was so much less powerful than his previous birth Hiranyakashipu: the successive births of Jaya and Vijaya became weaker and weaker. That's why Vishnu took two incarnations (Narasimha and Varaha) to defeat Hiranyaksha and Hiranyakashipu, but Vishnu defeated both Ravana and Kumbakarna in one incarnation as Rama, and by the time of Shishupala and Dantavakra, defeating Shishupala was just a minor part of Vishnu's incarnation as Krishna, and killing Dantavakra was an afterthought. As Jaya and Vijaya spent more time on Earth, they grew more eager to return to Vaikuntha, and so they wanted to be defeated as quickly as possible.
Not to keep you in suspense, Ravana did not actually get the opportunity to try to free Mahabali from Vamana, because Vamana disappeared before Ravana could challenge him to a battle. Vamana did this because if he agreed to fight he would easily kill Ravana, and that would violate Ravana's boon that he can't be killed by any god. (Vamana is one of the 12 Adityas and thus a god.) It's the same reason that Vishnu disappeared from Devaloka before Ravana's invasion of Devaloka, as I discuss here. That's why Vishnu incarnated as Rama, so that he could defeat Ravana in human form and thus keep Brahma's boon intact.
