For those who don't know, Niyoga is the practice where a woman has a man other than her husband father a child with her, either because her husband is unable to father children or because he's dead. The child is considered the adopted child of the husband. The Mahabharata describes two famous examples of Niyoga. The first is when fathered children with Vichitraviry's wives Ambika and Ambalika after Vichitravirya's premature death, as I discuss here. That is how Dhritarashtra and Pandu were born. The second example is when the gods Yama, Vayu, Indra, and the Ashwini Kumaras fathered children with Pandu's wives Kunti and Madri, since Pandu was unable to father children due to a curse he had received. That is how he Pandavas were born, as I discuss here.
In any case, to answer you question, the main place in Hindu scripture where the rules of Niyoga are described is in this chapter of the Manu Smriti:
On failure of issue (by her husband) a woman who has been authorised, may obtain, (in the) proper (manner prescribed), the desired offspring by (cohabitation with) a brother-in-law or (with some other) Sapinda (of the husband).
He (who is) appointed to (cohabit with) the widow shall (approach her) at night anointed with clarified butter and silent, (and) beget one son, by no means a second.
Some (sages), versed in the law, considering the purpose of the appointment not to have been attained by those two (on the birth of the first), think that a second (son) may be lawfully procreated on (such) women.
But when the purpose of the appointment to (cohabit with) the widow bas been attained in accordance with the law, those two shall behave towards each other like a father and a daughter-in-law.
If those two (being thus) appointed deviate from the rule and act from carnal desire, they will both become outcasts, (as men) who defile the bed of a daughter-in-law or of a Guru.
By twice-born men a widow must not be appointed to (cohabit with) any other (than her husband); for they who appoint (her) to another (man), will violate the eternal law.
In the sacred texts which refer to marriage the appointment (of widows) is nowhere mentioned, nor is the re-marriage of widows prescribed in the rules concerning marriage.
This practice which is reprehended by the learned of the twice-born castes as fit for cattle is said (to have occurred) even among men, while Vena ruled.
That chief of royal sages who formerly possessed the whole world, caused a confusion of the castes (varna), his intellect being destroyed by lust.
Since that (time) the virtuous censure that (man) who in his folly appoints a woman, whose husband died, to (bear) children (to another man).