When one performs an action according to one's swadharma, is there karmic impact of that and will the one accumulate zero karma, or positive/negative karma? What scriptures say on this matter. Or what can be inferred from the scriptures regarding this matter?
There will be acquisition of good Karmas (leading to higher births, heavens etc) when you perform Swadharma with the desire of getting fruits. But there won't be any acquisitions of any Karmas when you are not attached to the fruits (i.e when you perform the same without any desires).
12.88. The acts prescribed by the Veda are of two kinds, such as procure an increase of happiness and cause a continuation (of mundane existence, pravritta), and such as ensure supreme bliss and cause a cessation (of mundane existence, nivritta).
12.89. Acts which secure (the fulfilment of) wishes in this world or in the next are called pravritta (such as cause a continuation of mundane existence); but acts performed without any desire (for a reward), preceded by (the acquisition) of (true) knowledge, are declared to be nivritta (such as cause the cessation of mundane existence).
12.90. He who sedulously performs acts leading to future births (pravritta) becomes equal to the gods; but he who is intent on the performance of those causing the cessation (of existence, nivritta) indeed, passes beyond (the reach of) the five elements.
So, it depends on how the acts are performed (with or without attachment, desire). And, the verses are from Manu Smriti.
Swadharma are the acts that scriptures (like Sruti-Smritis) prescribe as per one's varna. That is referred to in the above verses as "acts prescribed by the Veda". And, they are of two kinds: Pravritta (done with the desire of getting the fruits of Karmas) and Nivritta (done without any such desires).
So, if Swadharma done in the Pravritta way, it will help accumulate good Karmas. But, if the same is done in the Nivritta way, it will lead to endless bliss (liberation). So, no acquisition of any Karmas in this case.
The Swadharma for a Kshatriya (the princely-class), for example, are as follows:
Manu Smriti 1.89. The Kshatriya he commanded to protect the people, to bestow gifts, to offer sacrifices, to study (the Veda), and to abstain from attaching himself to sensual pleasures;
So, if the Kshatriya does all these tasks perfectly well, but thinking of the rewards that he is entitled to get in mind, then that certainly leads to accumulation of Karmas (good Karmas in this case). But, if he is performing his duties, without any attachment/desires to/for the fruits (Karmaphala), then he does not accumulate Karmas.
Karma means action. Anything that we perform is a form of action, which is inevitable. Swadharma is attained upon reaching the state of non duality. In that state, the mind is evolved to such an extent that the person would not be attached to any of the karma he performs. Hence no karma is accumulated.