In Bhagavad Gita, Lord Krishna says that ParamAtma exists in the hearts of all living entities.
Question: Does he exist even in atoms of the non living material world?
You are mostly referring to below verse:
अहमात्मा गुडाकेश सर्वशयस्थितः। अहमादिश्च मध्यं च भूतानामन्त एव च
BG 10.20 - I am the Supersoul, O Arjuna, seated in the hearts of all living entities. I am the beginning, the middle and the end of all beings. -- translation ISKCON
The meaning of "भूत" also means "existence", dead or alive. This can be confirmed by below verse.
यच्चापि सर्वभूतानां बीजं तदहमर्जुन। न तदस्ति विना यत्स्यान्मया भूतं चर-अचरम्
BG 10.39 - Moreover, O Arjuna, whatsoever is the seed of all beings, that I am. There is no thing moving or non-moving which can exist without Me.
Where non-moving things can be interpreted as Those entities which can't move by themselves (i.e. electrons don't move with their own wish) -- hence this include non living entities as well.