Plantation owners in 17th century Barbados had a problem. They purchased white indentured servants and black slaves. At that time, there was little difference: life expectancy was so short that most indentured servants never saw freedom. The decision to buy servants or slaves was often dictated by price.
Black and white alike, the oppressed teamed up to rebel. Their oppressors concocted a strategy. Divide and rule: pit the two groups against each other by emphasizing race, and giving more privileges to the whites. For the first time, they crafted race laws that stripped black slaves of rights. And they deliberately starved them. "Tush, they can shift," they said: feed the blacks too little and they will steal from the whites. The strategy worked so well that it was imported to South Carolina, then to other English colonies in what was to become the United States.
White privilege was created not to benefit white people in general, but to benefit the masters. It was created to oppress black and white workers and slaves. The point was not privileges themselves: it was the message they sent that made one group feel superior, the other inferior, so that they would not work together. This strategy survived slavery, into the Jim Crow era when W. E. B. Du Bois called it a "psychological wage" that kept wages down for white workers and maintained racial antagonism to benefit elites.
We used to talk about prejudice and disadvantage: the implication was that disadvantages should be eliminated, raising everyone up. To call out privilege is to imply that it should be eliminated, lowering everyone down. Talk of white privilege is what it has always been: a strategy for domination that sets people against each other on the basis of skin colour to prevent them from resisting domination by elites.
See The Many-Headed Hydra: Sailors, Slaves, Commoners, and the Hidden History of the Revolutionary Atlantic, by Peter Linebaugh and Markus Rediker. See also Black Reconstruction in America, 1860-1880, by W. E. B. Du Bois.