Not only is speaking loshon hora a lowly act not befitting the divine image in which man was created, but to listen as someone else speaks loshson hora is lowly as well. The Torah states, Do not accept a false report (Shemos 23:1). The Chofetz Chaim is apparently of the opinion that merely paying attention to loshon hora being spoken constitutes giving it some degree of credence, and is in violation of this prohibition.
Just as it is wrong to select the shortcomings of others as subject matter for one’s own speech, so too it is wrong for one to focus his attention on negativity being expressed by others.
Just as it is wrong to cause harm to others by way of speech, so too it is wrong to serve as a listener as potentially harmful information is being conveyed.
In short, derogatory or harmful speech should never earn one’s attention. Listening to loshon hora, even if one does not believe it, is in violation of Torah prohibition.
