**Yes, there is a scriptural basis for the restriction.** 

Here is what Krishna says to Uddhava in the [twelfth chapter][1] of the Uddhava Gita of the Srimad Bhagavatam:

> One who desires to establish family life should marry a wife of his own caste, who is beyond reproach and **younger in age**. If one desires to accept many wives he must marry them after the first marriage, and each wife should be of a successively lower caste.


  [1]: http://www.vedabase.com/en/sb/11/17