While we should always make accommodation for fallen human nature, we shouldn't hesitate to realize what we are doing is making an accommodation. These types of individuals do just as much damage to the gospel as the benefit they do from that positive work. St. Cyprian elaborates on this issue in On the Unity of the Church:
Therefore also the Holy Spirit came as a dove, a simple and joyous creature, not bitter with gall, not cruel in its bite, not violent with the rending of its claws, loving human dwellings, knowing the association of one home; when they have young, bringing forth their young together; when they fly abroad, remaining in their flights by the side of one another, spending their life in mutual intercourse, acknowledging the concord of peace with the kiss of the beak, in all things fulfilling the law of unanimity. This is the simplicity that ought to be known in the Church, this is the charity that ought to be attained, that so the love of the brotherhood may imitate the cloves, that their gentleness and meekness may be like the lambs and sheep. What does the fierceness of wolves do in the Christian breast? What the savageness of dogs, and the deadly venom of serpents, and the sanguinary cruelty of wild beasts? We are to be congratulated when such as these are separated from the Church, lest they should lay waste the doves and sheep of Christ with their cruel and envenomed contagion.The people that St. Cyprian has in mind are primarily heretics, but it is interesting that he first slams them for horrible morals in their lack of charity, not horrible doctrine. (He touches upon the doctrine in the next segment.) In the mind of this great saint, a lack of charity has no place within the Church. By her very nature as an evangelical church, charity is part of the Catholic Churches nature.
Some will counter "well sorry, I believe that error needs to be condemned as error, and we shouldn't be having a tea party with evil people." Of course nobody is saying you cannot condemn error. Indeed, St. Cyprian is very strong in condemning error in this and several other treatises. The point is instead to stick to issues, not the person as much as possible. Whenever you bring up the defects of another, ask yourself "is this relevant to the argument." If it isn't, then don't bring it up.
Another point is to show charity to those within your own house. You don't have to like them. Yet to hold those as less than Catholic than you are (especially when the Church has not specifically said so) is a sin against charity, and this damages the effectiveness of your message. To treat anyone as somehow less than human because of said disagreements is beyond shameful, and it doesn't matter how much good that individual has done over the years for the cause.