A lot of people believe that a racist action happens because a person (who is racist in every single encounter) gets up in the morning and says, “I sure do have an irrational hate of X race. How can I be more racist toward that group every day and every way?” And, when that person engages in a racist action, s/he says, “I am doing this to you purely because you are X race.” Thus, as long as someone isn’t deliberately hostile, or their hostility isn’t irrational, or they don’t explicitly mention race, they didn’t do something racist.
In that world, someone saying that what you did is racist is accusing you of being that kind of really awful person. And so you are offended, and then the conversation shifts to how you feel about getting accused of getting up every morning and ironing your hood. (This is called “white fragility.”) So, you point out that you didn’t use racist terms, you have friends of X race (which might or might not be true), you have done un- or anti-racist things. None of that–your feeling of having been disrespected, your avoiding racist terms, your friends, your past behavior–is actually evidence that this you just did was not racist.
From Trump’s racist tweets by Patricia Robert-Miller