What is a definition of a sin? What are the criteria by which an act becomes sinful?
Sin: Actions performed out of desire or anger.
If the One (Purusha / consciousness) is attached to the actions performed under the influence of rajas guna (mode of passion), would incur sin.
Rajas is directly related to Adharma.
BG 3.36 - Arjuna said, O scion of the Vrsni dynasty (Krsna), impelled by how does this person(Purusha) commit sin even against wish, as of constrained by force?
BG 3.37 - The blessed lord said: This desire(kAma), this anger(krodha), born out of passion(Rajas) -- are great devourers(consumers), great sinners; Know that to be enemy here.
BG 14.7 - Rajas is the nature of passion, which is produced from lust & attachment; That binds the embodied [soul] through attachment to action.
The result of sin / Rajas is sorrow(dukha).
BG 18.38 - From the combinations of sense subjects (desires), which appears nectar in the beginning, but results in poison, -- that enjoyment is considered from Rajas
Important to note that, one has to be attached to the actions of Rajas, to be able to incur sin, not otherwise.
To extend further, all initiations (Arambha) are faulty, one way or the other way. Hence one should act, depending on own nature & circumstances. Because outside the context of Prakruti, the supreme reality doesn't care about sins & good deeds:
BG 18.48 — O son of Kunti, trivial(natural) Karma should never be given up, even though [it is] faulty; All initiations [of Karma] are faulty, like how smoke covers the fire.
BG 5.15 - The Omnipresent neither accepts anybody's sin nor even virtue. Knowledge remains covered by ignorance. Thus the creatures become deluded.
What is the criteria by which an act becomes a sin?
There cannot be an exact criteria for sin (pāpa) simply because there isn't exact criteria for dharma (righteous action). According to the Mahābhārata (Arjuna speaking to Draupadī):
There is no act that is wholly meritorious, nor any that is wholly wicked. Right or wrong, in all acts, something of both is seen.
I've tried to explain in this answer how using a knife on someone, depending on the context, can be considered dharmic (righteous) or adharmic (sinful).
Devotee: What is yoga?
Bhagavan: Yoga (union) is necessary for one who is in a state of viyoga (separation). But really there is only One. If you realize the Self there will be no difference.
D: Is there any efficacy in bathing in the Ganges?
B: The Ganges is within you. Bathe in this Ganges; it will not make you shiver with cold.
D: Should we sometimes read the Bhagavad Gita?
D: May we read the Bible?
B: The Bible and the Gita are the same.
D: The Bible teaches that man is born in sin.
B: Man is sin. There is no feeling of being man in deep sleep. The body-thought brings out the idea of sin. The birth of thought itself is sin.
D: The Bible says that the human soul may be lost.
B: The 'I' thought is the ego and that is lost. The real 'I' is 'I am that I am.'
While its very easy to ask such questions but at the same time extremely difficult to answer IMO.
But, still i have managed to come up with an extremely short answer :D.
Te hlAdaparitApaphalAh punyApunyahetutvAt
Because Punya (merit) and PApa or Apunya (demerit) are their causes, their effects are respectively Ananda (happiness or bliss) and Dukkha (sadness).
PAtanjala Yoga Sutras, SAdhana PAda-14.
We don't need to know what "they" are in this verse because that's irrelevant here.
So, going in the reverse order, any act whose effect is Dukkha (sadness) is demerit or sin.
Now, if you say that this is highly subjective, then i would say even what's sin and what's not is also highly subjective and changes with person, context etc.
We can also see the following verse from Manu Smriti:
Manu Smriti 2.12. The Veda, the sacred tradition, the customs of virtuous men, and one’s own pleasure (or own conscience), they declare to be visibly the fourfold means of defining the sacred law.
VedAh smritih sadAchArah swayasya cha priyam Atmanah- These are the fourfold means.
1) So, anything that's being declared in Vedas or Smritis as sin is a sin.
2) Anything that's not according to the principles of righteous living (sadAchAra) is a sin.
3) Even anything that our own conscience is marking as sin is a sin.
From the direct inference from the verse which I am gonna post below,
Sin is the opposite of Yoga. Any act which breaks the bond of unity with the fellow creatures or the creator or the ultimate reality is sin.
From ShivRahasya chapter 11.I have answered on another answer about Explanations on the laws of unity.
- Verily, My Love for man manifests itself in that I give him life of my Own Life and that I offer him sustenance, guidance and protection. And man's love of Me shall be seen in his following of My Law. By the following thereof he abides in Unity with Me. Therefore, he who desires a long and happy life in this World and the next, shall ever abide by the Law of Unity. He who abides not thereby goes the Path of Separation which is the Path of Sin and Discord.