The Manu Smriti defines Sruti (that has been heard) and Smriti (that has been remembered) as follows:
Srutis tu Vedo Vijneyo DharmashAstram tu vai Smritihi...
........
By Sruti is meant the Veda, and by Smriti is meant the Dharma
ShAstra..
Manu Smriti 2.10.
And, the Parashara Smriti, describes in the following verse, how at the beginning of each Kalpa, the Vedas are revealed and the Smritis are remembered.
..............
. " The author of the Veda there is none ; (he) the fourfaced (God),
at each succeeding revolution of a Kalpa, recalls to mind the Veda ;
and so does Manu remember the law (at each succeeding revolution of a
Kalpa).
So, for the Smritis to be available to us they must be remembered and there must be some persons who will do this job of recalling.
And, these are the Sages, Deities (like Manu, Yajnavalkya, Brihaspati, ParAshara etc), who recall the laws, whom we know by the name of the Smriti-KartA or the Dharma ShAstra-KartA.
The names of 18 such SmritikartAs is mentioned in a Yajnavalkya Smriti verse.
Similarly, for the Srutis to be available to mankind, there has to be some persons, who hear them from God.
And, they are the Vedic Rishis like ViswAmitra, MadhucchandAh, Kanva Ghaura, Gotama etc etc.
So, for Barahmana, Upanishads etc to be also declared as Sruti they need to be also heard by some Rishis or the other.
But, do we have such Rishis? NO.
So, its fair to call only the Samhitas as Sruti or the Vedas and which is what most people will find reasonable to accept too.
From Paramacharya Sri Chandrashekarendra Saraswati's book The Hindu Dharma:
So far, in speaking of the Vedas, I have dealt mainly with the Samhita
part of each sakha or recension. We have already seen that the
Samhitas are the main text of the Vedas. Apart from them, each sakha
has a Brahmana and an Aranyaka.
The Brahmana lays down the various rites - karma - to be performed and
explains the procedure for the same. It interprets the words of the
mantras occuring in the Samhita, how they are to be understood in the
conduct of sacrifices. The Brahmanas constitute a guide for the
conduct of yajnas.
The word "Aranyaka" is derived from "aranya". You must have heard of
places like "Dandakaranya" and "Vedaranya". "Aranya"means a "forest".
Neither in the Samhita nor in the Brahmana is one urged to go and live
in a forest. Vedic rites like sacrifices are to be performed by the
householder (grhastha) living in a village. But after his mind is
rendered pure through such rites, he goes to a forest as a recluse to
engage himself in meditation. It is to qualify for this stage of
vanaprastha, to become inwardly pure and mellow, that Vedic practices
like sacrifices are to be followed.
The Aranyakas prepare one for one's stage in life as an anchorite.
They expound the concepts inherent in the mantras of the Samhitas and
the rites detailed in the Brahmanas. In other words, they explain the
hidden meaning of the Vedas, their metaphorical passages. Indeed, they
throw light on the esoteric message of our scripture.
...
According to present-day scholars, the Aranyakas incorporate the
metaphorical passages representing the metaphysical inquires conducted
by the inmates of forest hermitages
So, from the above passage , it's quite clear why the Brahamanas etc are added to each Veda. They simply serve as supplementary texts (just like appendices or commentaries do) to understand the Vedas which without them would have been impossible to understand.
However, saints like Rishi Aurobindo does not even subscribe to this line thought.
As per him, Vedas themselves are self-sufficient as far as understanding them is concerned. We don't even need those supplementary texts.
EDIT:
Here is the BridaranyAkopanishad's verse that mentions the four Vedas and Upanishads separately:
The King Janaka asks YajnavAlkya "KA prajnata yAjnavalkya|" OR "What is PrajnA, YajnavAlkya?"
To that YajnavAlkya replies:
VAgeva samrariti hovAchA |
VAchA vai samrAr vandhuh prajnAyata
rigvedo yajurvedha sAmavedah atharvangirasa
itihAsah purAnam vidyA
upanishad shlokAh sutrAnya anuvyAkhyAnAni ... vAgvai samrAt param
brahmam ||
BrihadAranyakopanishat 4.1.2
The meaning is:
O king, the vAk is the prajnA. By VAk a firend can be known. By VAk
the Rig Veda, the Yajur Veda, the SAma Veda and the Atharva Veda, the
ItihAsas, the PurAnas, the VidyAs, the Upanishads, the Shlokas,
Sutras, the explanations of them and the supplementary explanations
(anu vyAkhyAni) .. etc can be understood. That VAk is the Param
Brahman.
From the Upanishad's 4th adhyAya's 1st BrAhmana (called the ShadAchArya).