While I definitely have my limits in who I am willing to associate with, I certainly don't avoid interacting with all people I disagree with. This would be both impossible and pointless anyway, as I have never encountered a person who I don't have some strong disagreement with about something. I have strong opinions on a lot of different topics, and unfortunately for me, the particular combination of opinions I possess doesn't map cleanly to any ideological group I am familiar with, much less any popular philosophical or political group.
There is a caveat to this: I also try not to surround myself with people who all disagree with me on the same things, or disagree in the same direction. I think when someone does that, their perspective on opposing viewpoints gets narrowed, as does their perspective of 'reasonable people who I nonetheless disagree strongly with on some things'. I'm also going to say that there's a very specific way I often see this perspective narrow, and that's the 'people who are more conservative than me are more reasonable than people who are further (or differently) left than me'. This is likely because these conservative opinions are more mainstream (since they're pushed heavily by the weight of society).
This is not me trying to say that leftists need to forget or ignore their differences to work together. Sometimes that's really not realistic. However, I do get a little tired of the different varieties of leftists painting each other as fascists while they express more positive opinions of actual fascists than they do of other leftists who disagree with them. What I'm saying to other leftists is, don't let the most palatable variety of fascist to whatever one's ideology may be position themselves as someone to be defended against leftists you disagree with.
Additionally, it might also be good not to defend more mainstream imperialist and colonialist views from leftists one disagrees with, positioning these imperialist and colonialist views as necessarily less violent than those dangerous radical leftists one doesn't agree with. The degree of popularity of a belief does not determine its violence, and in this case, mainstream beliefs are incredibly violent. So are they really somehow less violent than the leftist one disagrees with, or are they simply violent in a more widely acceptable way which society has trained us all from birth not to see as violence?
Are the leftists you disagree with capable of and perpetuating harm? Yes. So are all of us, in all the myriad varieties of different ways that harm can be perpetuated. I'm a pacifist, but that doesn't exempt me from perpetuating harm. Indeed, it is easy to perpetuate harm by promoting nonviolence, especially the form that is often promoted by mainstream colonial and imperialist powers, where you, the protestor, where you, the activist agitating for change, are asked to perform nonviolence and allow yourself to be brutalized by the police, by the state. Where you allow yourself to have the terms of your protest dictated by the state. This is the conservative conception of pacifism, and it is, ironically, its own form of state sanctioned violence, this time with the protestor as willing participant.
But pacifism isn't the only ideology vulnerable to this sort of conservative coopting. Every ideology can and has been coopted in this way by the mainstream. Which is why, for anyone who realizes that change needs to happen, we need to be alert for the ways the status quo may use the inevitable divisions between leftists to enlist our defense of it against other leftists. Because no leftist needs to be wasting their energy on that.