The Unpopular One
During adolescence and even young adulthood, there is always a sense of who is popular, who is average, and who is shunned from the rest. As we grow and mature in age these distinctions lessen, but never really go away. Even as most fall into the middle of a bell curve, there are still outliers with whom we rather not associate.
As adults it's less of a status distinction or popularity contest, but we tend to avoid those who require extra effort to befriend. People that lack skills of casual conversation or get stuck on a particular (and possibly boring) topic are pushed to the side. Again, not because it would ruin our "popularity" to be seen with them, but we simply don't get anything out of that relationship.
It's a selfish reason, yes, but one that is easily used in justification. After all you don't have to make painful small talk with that person, or stand there for twenty minutes listening to a lecture on the Cold war. No one will probably see you do it and it won't further your career, but, it will build your character.
Just like sitting with the lone, unpopular kid at the lunch table in 5th Grade speaks to your character and helps you grow as a person, so will reaching out and investing time in those whom society has passed by.