What is the historical origin of Yadavas?
Lord Brahma had a son named Atri and Atri had a son named Soma or Chandra (the Moon god). Budha was son of Soma.
Lord Brahmā had another son named Marīci and Marīci had son named Kaśyapa. Vivasvān (the Sun god) was son of Kaśyapa. From Vivasvān was born Vaivasvata Manu. Vaivasvata Manu got a daughter named Ilā.
Thus Budha begot in her (Ilā) womb a son named Purūravā.
In Purūravā's lineage then born kings like Nahusha and Yayati. Yayati's eldest son was Yadu.
Yadu's descendants came to be known as the Yadavas.
This lineage was also known as Mādhava and Vṛṣṇi after kings Madhu and Vṛṣṇi.
(Sources: SB 9.1, SB 9.17, SB 9.18, SB 9.23)
Yayati's youngest son was Puru. Puru's descendants came to be known as the Pauravas, the same lineage came to be known as Bharat Vanshi (after king Bharat), Kaurvas (after king Kuru), Pandavas (after king Pandu) etc.
From my reading of Mahabharata Yadavas do not appear to be a strong or prominent clan in India prior to the rise of Shri Krishna.
Yayati requested the eldest son, Yadu, to take his old age and transfer the youth of Yadu to him. Yadu declined his father's request. Yayati then cursed Yadu. Later on the curse was removed by Indra as mentioned in RV 4.30.17 which is explained in this answer. Yayati din't appoint Yadu as the chief king and gave few territories from South (druhyuṁ dakṣiṇato yadum ~ SB 9.19.22). This is same as how Yudhistra was sidelined earlier, he was not appointed as the king of Hastinapura but Indraprastha.
But Yadu gained popularity, fame by his own virtues deeds.
SB 9.23 mentions many famous Chakravarati kings in Yadavas prior to the rise of Lord Shri Krishna :
Kartavirya Arjuna (He [Kārtavīryārjuna also known as ShashtrAjuna] became the emperor of the entire world, consisting of seven islands ~SB 9.23.24. No other king in this world could equal Kārtavīryārjuna in sacrifices, charity, austerity, mystic power, education, strength or mercy ~SB 9.23.25. He defeated and put Ravana in Jail. He eventually met his death at
the hands of Lord Parashurama.)
Madhu (After him, his lineage is called as Madhavas ~
SB 9.23.29. Hence, Lord
Krishna is called as Madhav.)
Vrishni (After him, his lineage is called as Vrishneyas ~SB 9.23.29. Hence, Lord
Krishna is called as Vrisney. His name is mentioned in Bhagavad Gita
Śaśabindu (possessed fourteen opulences and was the owner of fourteen great jewels. Thus he became the emperor of the world.~SB 9.23.30-31)
So your understanding seems to be wrong, Yadavas were one of the most strongest and prominent clans.
Rather most other high houses of bharata seems to look at Yadavas with a bit of disdain. Is it true?
No. Who are high houses? Yadavas were also from Chndra Vanshi Kshatriyas and Yadu was elder brother of Puru (whose descendants were Kaurvas/Pandavas). So neither by birth nor by power (as described in the above section) Yadavas were inferior.