This abridged version of SRSTIKHANDA of Padma Purana answers your all questions:
After killing the demon king Ravana, Sri Rama returned to Ayodhya. Many sages had arrived there to bless him on the occasion of his crowning ceremony (Pattabhishek). Sage Vashishtha was one of them. As Sri Rama was conversing with the sages a brahmin arrived there with the corpse of his son and wailed --- "O son! Your death is certainly due to some flaw of Sri Rama. Now, your mother and I have decided to give up our lives as there is no point in living."
Sri Rama's heart was filled with grief and he asked Sage Vashishtha as to how the brahmin could be helped. Sage Vashishtha [or Narada who was there in Vasistha's proximity] revealed to Sri Rama that the brahmin's son had died a premature death because of Shambuk. Vashishtha [or Narada who was there in Vasistha's proximity] said -- "Shambuk is a Shudra by birth, but he is doing an austere penance. Shudras are not entitled to do penance in all the three yugas, except the Kali Yuga. So, only his death can bring back the brahmin's son alive."
Sri Rama instructed both his younger brothers -- Lakshman and Bharata, to look after the state's affairs in his absence and went in search of Shambuk boarded on his Pushpak Vimana. Sri Rama saw a man doing penance [with his legs upwards and head downwards] at the shore of a reservoir. He landed his Pushpak Vimana and went near that man. He introduced himself and enquired about the man's identity. Sri Rama asked -- Why are you doing such an austere penance. Who are you?
Without moving the man replied that he was a shudra named Shambuk. He said --- I am desirous of attaining to devaloka [with my body] and hence I am doing this penance. Sri Rama took out his sword and severed Shambuk's head. All the deities hailed Sri Rama and the brahmin's child became alive once again.
Is Rāma justified in killing him without a dialogue or a second chance? Why didn't he just imprison him? What was the need to eliminate Śambūka altogether?
As mentioned above,
only his death can bring back the brahmin's son alive.
As this is an abridged version of Padma Purana. The detailed story can be found in PADMA-PURANA: SRSTIKHANDA: Chapter named The Killing of a Sudra Ascetic and in Valmiki Ramayan: UTTARA KANDA (Source1, Source2, Source3).
Here are relevant verses from the full version of these scriptures:
97-98. "O descendant of Kakutstha, give up your anxiety;
the only son of the brahmana has regained his life and is united
with his relatives. O descendant of Kakutstha, the boy suddenly
became united with (i.e. regained his) life, (just) at the moment
when this shudra was destroyed by you.
This implies that killing of Śambūka was needed in order to bring brahmin's son alive.
After seeing below verses Rāma was justified in killing him:
89-90a. [Gods:] "O you descendant of Raghu, O you good one, you
have done this mission of the gods. Rama of a great vow, take (i.e. choose) whichever boon you like."
Also as per next chapter named Conversation between Rama and Agastya of Padma Purana :
[Rama:] Obeying the words of gods, I have destroyed this sinful
sudra who was practising penance.
[Agstya:] The gods also proclaim that you, the killer of the sudra,
have come. You have righteously brought the brahmana's son
back to life.
So Rama was justified in the opinion of Markandeya, Maudgalya, Vamadeva, Kashyapa, Katyayana, Javali, Gautama, Narada (these 8 were present when Vasistha/Narada explained the story of Shambhuk to Raam), Gods and Agastya.
Why did Rāma kill Śambūka? Is it because Śambūka was a śūdra or is it because he was doing tapasya for the wrong reasons?
As mentioned above,
Shudras are not entitled to do penance in all the three yugas, except the Kali Yuga.
He was also doing tapasya for the wrong reasons.
Hearing the words of Rama of unwearied actions, the ascetic, with his face downwards, said "O highly illustrious Rama, I am born in the race of Sudras; and with a view to reach the region of the gods with my body I am going through these austere penances. O Kakuthstha, I shall never utter a falsehood since I am willing to conquer the the region of gods.
And due to this a Brahmin's son died prematurely. So he committed Brahmhatya sin, one of the gravest sins:
As per Padma Purana:
51-52a. O king, a sudra, the
tormentor of his enemies, and of a wicked mind, is practising a
very severe penance at the border of your country. He killed
As per Valmiki Ramayana:
O king, one Sudra, under the influence of vicious understanding, has begun devout penances within thy kingdom. And for that reason this boy hath met with death.
Were śūdras not allowed to become a ṛṣi?
As mentioned above, possibly not in Treta Yuga. However, allowed in Kali Yuga.