I don't think it's about touching the blue color or wearing blue color clothes. It's about Indigo, that is used to create dyes etc. of the blue color.
If you go through the entire Manu Smriti, you won't find any restrictions in it regarding the Brahmins and the blue color.
But, regarding the use of indigo-dyed clothes, the cultivation of the same there are strict prohibitions in the scripture.
Quoting verses from the 1st chapter of the Angirasa Smriti:
.... regulations relating to the use of clothes dyed with indigo.
There is no sin [in using them] while lying down on bed for
co-habiting with one's wife. (12)
By rearing up indigo plants, by selling them, and acquiring his
livelihood therefrom, a Vipra becomes degraded ; and he is freed from
the sin by three distressing penances. (13) [Sacrificial] bath,
charity, recitation [of Mantrams],
the offering of oblations to the Sacred Fire, Vedic study and the
offering of oblations to the departed Manes as well as the [Five]
great Yajnas (Sacrifices) become profitless to a person, from the use
of a cloth dyed with indigo. ( 14).
If one, out of ignorance, puts on a cloth dyed with indigo, one, after
fasting for a day and night, becomes purified with the Panchagavya*
If by his carelessness an indigo wood pierces the body of a Brahmana,
and if blood is seen there, the twice-
-born person should perform the Chandrayana. (16)
If a twice-born person eat boiled rice with the sticks of indigo
plant, he should, by vomitting out the food be purified 'with the
The three Varnas should perform the Chandrayana if twice-born persons,
carelessly and out of error, eat indigo : for this is the rule. (18)
The food, that is brought by a person putting on a cloth dyed with
indigo, stands the- giver in good stead, [but] the partaker thereof
merely eats sin. (19)
Fasting for a day [is laid down! for Vipras fed with the rice which is
boiled by [a person putting on] a raiment dyed with indigo. (20)
Some further verses are as given below. The ground on which indigo has been cultivated is rendered impure for all purposes for as long as 12 years. So, indigo is considered as something very impure.
The husband of the woman, who on te demise of her consort, puts on a
raiment dyed with indigo, goes to hell. The woman, too, [goes to hell]
The corns, which one grows on a field sullied with the sowing of
indigo, are unworthy of being eaten by the twice-born. By eating
[them, one] should perform the Chdndrayana. (22)
Such a place [i.e., where indigo has been sown] should never be used
for a procession with idols, for the dedication of a bull, for
celebrating a sacrifice or for nicking gifts; for the earth [there]
has become contaminated. (23)
The ground, where indigo has been sown, remains impure for twelve
years and becomes pure thereafter. (24)
So, in my opinion, the prohibition is not regarding just the blue color but regarding the use of indigo and the blue color derived from it.