For Nigel Farage, the lead campaigner for Brexit, June 23, 2016, marked Britain’s “independence day”: the moment when the country finally recovered its sovereignty.

The idea that Leavers sold to the British public was circular: that sovereignty meant control, and control meant sovereignty. A sovereign Britain was meant to decide who is allowed into the country, what international alliances it is part of, and the type of relationship Britain has with its European neighbors. Reasserting “sovereignty,” then, meant that the UK would regain power over its money, laws, and borders; by leaving the EU, it would feel like it had a firm grasp on its destiny as a nation again.

But as many of the “sovereign” nations of the Global South might have warned Britain years ago, sovereignty is not the same thing as control: a nation can be “independent” and decidedly unfree at once. And today, “taking back control” looks less and less… controlled.

Prime Minister Theresa May—who today, postponed a vote on her unpopular withdrawal deal in the House of Commons—has little support from the EU. Opposition leader Jeremy Corbyn describes her actions as an act of “national self-harm.” Members of May’s own party describe the deal as “surrender terms”; another famous Brexiteer, Boris Johnson, is looking to realize his Churchillian childhood fantasies by rallying the country to reject a deal he claims will give Britain “colony status.”

Making things worse is that Britain will be paying billions of pounds for a Leave deal that will likely give it even less control over its borders, laws and money than it had when it was an EU member. The dream of “sovereignty” appears further away than ever. How did it come to this?

A clue might be in the retrospective framing of the famous Brexit slogan: the promise to “take back control.” Brexit was supposed to return the country to some golden age when it was, as Liam Fox described it, “a small island perched on the edge of the European continent [that] became a leader of world trade.” This squares with the myth that Britain is a fearless island nation of inventors, explorers, and entrepreneurs that once achieved global supremacy through its own initiative.

Erased from this narrative is the reality that Great Britain used to “control” the world not because of its independent spirit but because of its ability to wield imperial violence. Britain’s pre-EU memory of sovereignty is the sovereignty of an empire, not that of a nation-state; it’s a memory of the British Parliament determining economic policy from Barbados to Bengal.

In Brexit’s worldview, sovereignty is remembered as hegemony: directing the global order rather than being subject to its whims. Britain remembers its empire not through glorious military victories and the seizures of land, but through the economic and technological innovations that its empire is credited with: railways, bridges, and industrial production.

So, in the popular imagination, Britain’s former period of “control” is not presented as the time when Britain took control by force, but as the inevitable consequence of independent and enlightened Britain trading freely across the globe. This attitude is perhaps best captured by the famous quote of British essayist John Robert Seely, who wrote that “we seem, as it were, to have conquered half the world in a fit of absence of mind.”

The image of benevolent British global prominence is what Brexiteers try to transport from the past into the future; for them, British control is not something that was fought for, seized and protected but, rather, the natural outcome of being a free and sovereign nation.

This giant historical misunderstanding about the lost era of British rule is what is fueling Brexiteers’ grievances as they see their dream of sovereignty disappearing over the horizon. The vision of a previously sovereign Britain as a homogeneous nation-state, rather than a multiracial, global empire fuels the illusion that a ‘pure’ Britain can return if we take back control.

As historian David Eggerton tells us, “British imperial power flowed from the barrel of a British gun, not the Anglican Bible or textbooks of liberal political economy.” The danger for Brexiteers imagining that that they can take back control is that by failing to understand that what they remember as control came from violence, not just independence. They also ignore that in 2018, in a world in which Britain’s military advantage over the likes of India and China has long disappeared, Britain would struggle to regain this type of control if it even tried.

Brexit serves as a warning sign of the dangers of misunderstanding the nature of sovereignty: both in the past, where it was more imperial than national, and in the present, where globalization limits the ability for any nation to truly feel “in control.” Until the Brexiteers realize that they are chasing a ghost, there will be little control, only more moments of national crisis to come.