The argument sounds plausible because there is a superficial logical connection between slavery and reparations. Reparations are proposed as a remedy for slavery. If someone opposes the remedy, it can seem as though they are minimising the wrong. The rhetorical move exploits a genuine feature of moral reasoning: that opposition to a remedy can sometimes reflect indifference to the underlying harm.
The argument also draws strength from the history of how arguments against reparations have sometimes been made. When opponents of reparations have used language that minimises the severity of slavery, or have focused entirely on technical objections without acknowledging the moral weight of the history, the accusation has had more force. Bad-faith opposition to reparations does exist, and it can look superficially similar to good-faith opposition.
The political effectiveness of the argument is also evidence of its apparent plausibility. The fact that it has successfully shifted the terms of debate in international forums suggests that it resonates with audiences who are not themselves engaging in bad faith.