What are the six kinds of renunciation that are mentioned in the Vedas or other Hindu scriptures?
The six kinds of renunciation are described in this chapter of the Sanatsujatiya, which is a dialogue between Dhritarashtra and the sage Sanatsujata (often equated with Sanatkumara) before the start of the Mahabharata war. Here is what Sanatsujata says:
The six kinds of renunciation are all commendable. They are these: The first is never experiencing joy on occasions of prosperity. The second is the abandonment of sacrifices, prayers, and pious acts. That which is called the third, O king, is the abandonment of desire or withdrawing from the world. Indeed, it is in consequence of this third kind of renunciation of desire, which is evidenced by the abandonment of all objects of enjoyment (without enjoying them) and not their abandonment after having enjoyed them to the fill, nor by abandonment after acquisition, nor by abandonment only after one has become incompetent to enjoy from loss of appetite. The fourth kind of renunciation consists in this: One should not grieve nor suffer his self to be afflicted by grief when one's actions fail, notwithstanding one's possession of all the virtues and all kinds of wealth. Or, when anything disagreeable happens, one feeleth no pain. The fifth kind of renunciation consists in not soliciting even one's sons, wives, and others that may all be very dear. The sixth kind consists in giving away to a deserving person who solicits, which act of gifts is always productive of merit. By these again, one acquires the knowledge of Self. As regards this last attribute, it involves eight qualities. These are truth, meditation, distinction of subject and object, capacity for drawing inferences, withdrawal from the world, never taking what belongeth to others, the practices of Brahmacharya vows (abstinence), and non-acceptance (of gifts).
I'm reminded of the Sadhana Saptaka, the sevenfold discipline of Bhakti Yoga followed by Sri Vaishnavas. As I discuss in this question, it originates from Tanka's Chandogya Upanishad Vakya, which is an ancient pre-Shankara commentary on the Chandogya Upanishad that is now lost. Here is how Ramanujacharya describes the Sadhana Saptaka in this section of the Sri Bhashya, his commentary on the Brahma Sutras:
The Vâkyakâra also declares that steady remembrance results only from abstention, and so on; his words being 'This (viz. steady remembrance= meditation) is obtained through abstention (viveka), freeness of mind (vimoka), repetition (abhyâsa), works (kriyâ), virtuous conduct (kalyâna), freedom from dejection (anavasâda), absence of exultation (anuddharsha); according to feasibility and scriptural statement.' The Vâkyakâra also gives definitions of all these terms. Abstention (viveka) means keeping the body clean from all food, impure either owing to species (such as the flesh of certain animals), or abode (such as food belonging to a Kândâla or the like), or accidental cause (such as food into which a hair or the like has fallen). The scriptural passage authorising this point is Kh. Up. VII, 26, 'The food being pure, the mind becomes pure; the mind being pure, there results steady remembrance.'
Freeness of mind (vimoka) means absence of attachment to desires. The authoritative passage here is 'Let him meditate with a calm mind' (Kh. Up. III, 14, 1). Repetition means continued practice. For this point the Bhâshya-kâra quotes an authoritative text from Smriti, viz.: 'Having constantly been absorbed in the thought of that being' (sadâ tadbhâvabhâvitah; Bha. Gî.VIII, 6). By 'works' (kriyâ) is understood the performance, according to one's ability, of the five great sacrifices. The authoritative passages here are 'This person who performs works is the best of those who know Brahman' (Mu. Up. III, 1, 4); and 'Him Brâhmanas seek to know by recitation of the Veda, by sacrifice, by gifts, by penance, by fasting' (Bri. Up. IV, 4, 22).
By virtuous conduct (kalyânâni) are meant truthfulness, honesty, kindness, liberality, gentleness, absence of covetousness. Confirmatory texts are 'By truth he is to be obtained' (Mu. Up. III, 1, 5) and 'to them belongs that pure Brahman-world' (Pr. Up. I, 16).--That lowness of spirit or want of cheerfulness which results from unfavourable conditions of place or time and the remembrance of causes of sorrow, is denoted by the term 'dejection'; the contrary of this is 'freedom from dejection.' The relevant scriptural passage is 'This Self cannot be obtained by one lacking in strength' (Mu. Up. III, 2, 4). 'Exultation' is that satisfaction of mind which springs from circumstances opposite to those just mentioned; the contrary is 'absence of exultation.' Overgreat satisfaction also stands in the way (of meditation). The scriptural passage for this is 'Calm, subdued,' &c. (Bri. Up. IV, 4, 23).
The Vakyakara refers to Tanka, and the Bhashyakara refers to Dramidacharya who wrote a Bhashya or commentary on Tanka's Vakya. Both Tanka and Dramidacharya were ancient commentators who lived long before the time of Adi Shankaracharya.