What you heard definitely has a scriptural basis. This Matsya Purana chapter mentions the various types of soils suitable for various Varna people to build their houses on:
As regards the DAYS, all the days of a week are good, except Sundays
and Tuesdays. In regard to YOGAS, construction should not start on the
following yogas viz. Vyaaghat, Shula, Vyatipaat, Atiganda, Vishkambha,
Ganda, Parigha and Vajra. Following MUHURTAS are stated to be ideal
for contstruction viz. Sweta, Maitra, Maahendra, Gaandhawa, Abhijit,
Rouhini, Vairaja, and Savitri. To decide on the SHUBHA LAGNA, the
activity of construction should be commenced when the Chandra bala as
also Suryabala would be equally positve. The most important activities
apart, Stambharohana or the hoisting of the Pillar should be taken up
first; similarly the procedures should be the same as in respect of
the costruction of Praasaads, wells and such mini water bodies. SOIL
TEST would be the foremost activity before the actual construction is
taken up. Sweta (White), Lal (Red), Peeta (yellowish) and kaala
(black) may be the ideal soils for the Four Varnas of Brahmana,
Kshatriya, Vaisya and Shudra respectively. Taste-wise, the soil may be
sweetish for Brahmanas, sour for Kshatriyas, tiktha or bitter for
Vaishyas and Kasaili/ Kashaya / Pungent for Shudras. As a part of
the soil quality test, saplings are to be planted and the best soil
quality would yield the growth over three nights; medium growth over
five nights and minimal growth over a week. The best soil obviously
would give out the quickest yield
Update:
The Tantra SAra book (on pp 535) quotes some Gautamiya Tantram's verses that are related to the topic. Here are the verses:
Gautamiye--- tatrApi prathamam bhuminirnayah|
Bhumeh parigraham kuryAdyAvatAyatanam bhaveth |
ShuklamritasnA tu
yA bhumirbrAhmi sA parikirtitA |
Kshatriya raktamridu
bhumirharidvaishyA prakirtitA |
KrishnA bhumirbhavet shudrA
chaturdha bhuh prakirtita |
BrAhmi sarvArthasiddhi syAt kshatriyA
rAjyadA matA |
DhanadhAnyakAri vaishyA shudrA tu ninditA bhaveh |
The ground which has white colored soil is called BrAhmi Bhumi, the
one with red colored soil is called KshatriyA, the one with yellow
colored soil is Vaishya and the one with black colored soil is called
the Shudra Bhumi. The BrAhmi Bhumi fulfills all wishes, the KsatriyA
is the giver of kingdom, the Vaishya Bhumi is the bestower of wealth
but the Shudra Bhumi is censured.