This is how the story goes... It has been paraphrased by me, I have provided the references at the end and you can check those for more detailed reading.
Once upon a time, the sage Vibhāṇḍaka (son of Kāśyapa) went to a big lake to wash himself up, he saw the apsarā (celestial nymph) Urvaśī and got aroused. As a result, his reproductive fluid fell in the lake. A doe, who was cursed apsarā, while drinking water, swallowed it and became pregnant. Later, she gave birth to a boy with deer horns because of which he was named Ṛṣyaśṛṅga. Vibhāṇḍaka raised his son absolutely isolated from society in his āśrama in forests. And so, Ṛṣyaśṛṅga knew barely of any worldly pleasures and was totally unaware of existence of any female. As taught by his father and being filial and obedient to him, he practiced brahmacarya and attained miraculous powers due to his celibacy. The king of Añgas, Romapāda (at that time) offended a Brāhmaṇa due to which no Brāhmaṇa agrees to do a yajña, this angered Indra and caused a terrible drought endangering all creatures. Romapāda assembled the learned Brāhmaṇas and entreated for their help to suggest him a penance with which he could expiate his wrong-doing and thus stop the drought. The Brāhmaṇas advise him to bring Ṛṣyaśṛṅga to the kingdom and offer to him his own daughter Śāntā in marriage. The king came to decision after some consultation with his counselors and he commanded the dispatch of his purohita and ministers, to invite the son of Vibhāṇḍaka, but they pleaded with the king to not go as they were afraid of Vibhāṇḍaka. After they came to a consensus though, they decided to bring Ṛṣyaśṛṅga by employing some courtesans. They knew that Ṛṣyaśṛṅga was a forest dweller, devoted to monasticism and was wholly unacquainted with women and sensual pleasures, so they sought to use this very weakness to target him.
The best courtesans that could be mustered were sent to the forest and stayed near the āśrama to try to catch the glimpse of Ṛṣyaśṛṅga, who always dwelt with his father and never went outside the hermitage. But on one occassion, Ṛṣyaśṛṅga happened to come to the place where the courtesans resided, wearing beautiful clothes and singing with sweet voices, which was an extremely foreign and unknown thing to him. He approached the women, wanting to know more about them and attracted by their appearance, invited them to the āśrama and served them, in Vibhāṇḍaka's absence. They also served him delicacies which he had never eaten in his life. But then they gave an excuse and left the hermitage because they feared Vibhāṇḍaka might come soon. However, when they left, Ṛṣyaśṛṅga felt lonely and lovesick since he kinda enjoyed their company. So, the next day, he again went to the place, he had met them. The courtesans invited him to come to their āśrama and he resolved to go with them to the kingdom. As he went along, the god brought rain upon the lands, ending the drought. The king went to him and begged for the favor of Ṛṣyaśṛṅga, and gave him his daughter Śāntā in marriage. Later, Daśaratha who had an alliance with Romapāda, sought him to ask for the aid of Ṛṣyaśṛṅga to perform a yajña so that he could have sons. And thus, he invited both Ṛṣyaśṛṅga and Śāntā to the kingdom and then he performed the progeny begetting yajña in Ayodhyā.
This story appears both in Vālmīki Rāmāyaṇa (Sargas 8-10 in Bālakāṇḍa) and Mahābhārata (Tīrtha Yātrā Parva, Vana Parva). However, the part of the story of miraculous birth of Ṛṣyaśṛṅga is not present in Vālmīki Rāmāyaṇa. The story in Mahābhārata, recited by Lomaśa to Yudhiṣṭhira, is more detailed and covers lot of plot such as the birth of Rṣyaśr̥ṅga, conversation of father and son etc. which all isn't present in Vālmīki Rāmāyaṇa. However, the Mahābhārata story doesn't cover the part where Ṛṣyaśṛṅga is asked for help from Daśaratha and him going to Ayodhyā to perform a yajña.
It is believed in popular tradition that Ṛṣyaśṛṅga dwelt with his father near Śṛṅgerī, before he left to the kingdom. That is why probably you saw so in the temple.
The story of Ṛṣyaśṛṅga also features in the Jātaka (Niḷinikājātaka, Ja 526) with the name Isisiṅga (in Pāl̥i), but this story is not the same and is a bit different. Alambusājātaka (Ja 523) also recounts a story of Isisiṅga, which is quite similar to Ja 526 but different than the stories of Ṛṣyaśṛṅga in Mahābhārata and Rāmāyaṇa.
References