Yes, there is a scriptural basis for the restriction.
Here is what Krishna says to Uddhava in the twelfth chapter of the Uddhava Gita of the Srimad Bhagavatam:
One who desires to establish family life should marry a wife of his own caste, who is beyond reproach and younger in age. If one desires to accept many wives he must marry them after the first marriage, and each wife should be of a successively lower caste.