How is prāyaścitta different from daṇḍa? Are both complementary to each other?
Prayaschitta and regal punishments (danda) both serve the same purpose: destroying the sin of the crime or sinful activity that was committed.
According to Hinduism, the king's duty is to enforce dharma and prevent adharma (sins); there is no separation of church and state. So, a crime in a Vedic kingdom would be a sin. So all sins are crimes, and all crimes are sins. This is different from secular Western countries where the main duty of governments is to protect the rights of people and not interfere in religious practices.
Therefore, the punishment of the king serves as a type of prayaschitta:
Manu 7.25 - Where dark-complexioned and red-eyed Punishment stalks about, destroying sins, there the people are not misled, provided that the Governor discerns rightly."
Manu 11.99 & 11.100 - A Brāhmaṇa who has committed the theft of gold shall go to the King, and confessing his crime, shall say ‘sire, punish me.’, Taking up a club, the King himself shall strike him once. The thief becomes purified by death; but the Brāhmaṇa by penance alone.
So you can see from these verses that regal punishment serves as a prayaschitta. This is in accordance with the Vedic theory of governance; that the king should enforce dharma and prevent adharma.
Are both prāyaścitta and daṇḍa complementary to each other or are there sins where both apply?
When to apply prayaschitta and when to apply danda is determined by circumstance, type of sin, caste, etc. Many of the prayaschittas cannot be done by shudras or women because the prayaschittas require recitation of mantras and performance of yajnas, which shudras cannot do. So for them, their purification would be done by regal punishment.
If the person doesn't do prayaschitta, then the king is obligated to punish:
Manu 9.235 - The slayer of a Brāhmaṇa, the drinker of wine, the thief and the violator of the preceptor’s bed,—all these individually should be known as men who have committed heinous crimes.
9.236 - Even on all these four, if they do not perform the expiatory penance, the king shall inflict corporal punishment along with fine, in accordance with the law.
Medhatithi's commentary:
For the crime of ‘Brāhmaṇa-slaying,’ ‘corporal punishment’ has been already laid down above,—in the rule that—‘the king shall put to death those who kill a woman, an infant or a Brāhmaṇa.’
From what follows in the next verse it is clear that ‘corporal punishment’ here stands for branding.
So, if the sinner does not do prayaschitta, then he is to be branded as follows:
9.237 - For violating the preceptor’s bed the sign of the female organ shall be branded; for drinking wine that of the tavern; for theft that of the dog’s foot; and for killing a Brāhmaṇa that of a headless man.
And then they became homeless, outcasted, etc:
9.238 - Debarred from entertainments, debarred from sacrifices, debarred from education, excluded from all religious acts, these shall wander over the earth; abject and despised.
9.239 - Being branded, these shall be abandoned by Kinsmen and relations, deprived of all sympathy and greetings;—such is the teaching of manu.
However, the text states what happens if they do prayaschitta:
9.240 - But men of the senior castes [Dvijas], who perform the expiatory penances, as prescribed, shall not be branded on the forehead by the king; they shall be made to pay the highest amercement.
9.241 - For offences committed by the Brāhmaṇa the middle-most amercement shall be inflicted on him; or he shall be banished from the kingdom, along with his goods and chattels.
Medhatithi's commentary:
The condition of expiatory penances being performed does not apply to
what is asserted here.
In the case of all these offences—of Brāhmaṇa-slaughter and the
rest—the Brāhmaṇa shall be fined ‘the middle-most amercement’
The qualification ‘unintentionally’ of the next verse has to be
construed with this also.
After he has paid the fine, he should be made to perform the expiatory
penances.
‘Along with his goods and chattels.’—This is a special favour to be
granted in the case of highly qualified Brāhmaṇas.
In the case of the offence being unintentional, he may not be
banished.
So for the mortal sins, even when the sinner performs the prayaschitta, he is still to be fined, banished, property taken, etc. but he should not be branded on the forehead and become homeless. So this is a case where prayaschitta and regal punishment are both inflicted; this is for the most heinous sins.
So, there are many things to consider like whether the sin was done intentionally or unintentionally, the caste of the sinner, the person who was the target of the crime, qualification of the sinner, etc.
Also, there is a difference between sins committed that other people know about and sins done in secret. The expiation for sins done in secret is much milder compared to the non-secret sins.