Based on my experiences as a human rights lawyer, even contrary to commitments made by international organizations such as the OSCE and the EU FRA, I argue that the concept of hate crimes should be limited to hate incidents committed against members of minority communities. I see a substantive difference from anti-discrimination legislation, where “the more the better”-principle is in place. Here, less is more! In several countries, Hungary is one of them, in line with the above commitments, hate crime (and hate speech) legislation, and practice have been used to protect the majority from the minority. In several cases of violent conflicts, members of minority communities have systematically been charged with racially motivated hate crimes committed against the majority. These happened even if prosecutions when racially motivated hate incidents target minorities are rare, and even in cases, and this is particularly alarming, if actually members of racist hate groups are the victims of the incidents. Based on these, by far not isolated, experiences, I hereby propose to change the trend and restrict the concept of hate crimes to victimization of members of communities that bear the stigma of social inferiority, face some sort of social marginalization, discrimination, persecution, or a history of oppression. I propose for international instruments to identify and declare hate crimes as specifically minority protection mechanisms and install the requirement of vulnerability, the threat of potential or actual exclusion or marginalization in the concept. Otherwise, this institution, too, can be used in a way to systematically disfavor minorities.