You misunderstood. Lord Rāma is known as Maryada Purushottama which means the best of men or the Supreme Purusa (personality), not as a 'perfect family man'.
Take the two incidents, exile of Sita and Lakshmana. Both of these incidents happened after Lord Rama becoming the ruler of Kingdom Ayodhya. The dharma of a king is to lead his people in the path of Dharma. King Rama was forced to exile Sita(and that was not husband Rama) for keeping the Dharma. The same Dharma exiled Lakshmana(and it was not brother Rama).
As a person, Rama personifies the characteristics of an ideal person (purushottama)3 who is to be emulated. He had within him all the desirable virtues that any individual would seek to aspire, and he fulfils all his moral obligations (maryada). Rama's purity and piety in his intentions and actions inspires affection and devotion for him from a variety of characters from different backgrounds. For example, he gave up his rightful claim to the throne, and agreed to go into exile for fourteen years, to fulfill the vow that his father had given to Kaikeyi, one of King Dashratha's wives. This is in spite of the fact that Kaikeyi's son, Bharat, begged him to return to Ayodhya and said that he did not want to rule in place of Rama. But Rama considered his dharma as a son above that of his own birthright and his life's ambition. For such supreme sacrifices, and many other qualities, Rama is considered a maryada purushottama(the best of upholders of Dharma).
Also while mentioning family you forgot to mention about the perfect son.
Some of his ideals are as follows:
At the time when it was normal for kings to have more than one wife, Rama gave the ideal of having a single wife. After Sita was banished, he was doing penance with a gold statue of Sita. In Balakanda of Valmiki Ramayana it is written that Rama and Sita resided in each other's heart.
Rama always followed his promise at any cost. In fact, he went to forest to make his father's promise to Kaikeyi true. Another instance was when, he had promised the Spirit of Time that during their conversation, if anyone was to intrude, Rama would have pronounce an instant death sentence upon the individual. They were intruded upon by his beloved younger brother Lakshmana, and to keep his part of the promise, pronounced the death sentence. There are many examples of Rama's promises which he kept. Most important are the promise to sages to save their lives from Rakshasas, getting back Sugreeva's kingdom, making Vibhishana the king of Lanka.
Most of these qualities make a perfect family man too. So one of my statement needed to be corrected, he was a perfect family man too. Your doubts can be cleared by reading the Ramayana. It's just explaining the qualities of Lord Rama and how should one live for Dharma.
Sri Rāma - in the words of Swami Vivekananda - is "the embodiment of truth, of morality, the ideal son, the ideal husband, and above all, the ideal king."
Vivekanada quoted it in the correct order. Son,husband and then the ideal king.