For British territories, Brexit will amount to being cut off from Westminster

Leaving the EU will remove the connection that those living in places such as Gibraltar have with the UK

The Rock of Gibraltar as viewed from the Spanish town of La Linea de la Concepcion in Andalusia. Courtesy Marek Stepan

There is an old British cliche that something truly safe is “as solid as the Rock of Gibraltar". But Gibraltar itself does not entirely feel safe nowadays. Legally it is a British Overseas Territory, a tiny foothold of British imperial power since 1704, attached to the landmass of mainland Europe at the southern tip of Spain. At the top of the Rock there is a sign that says the monkeys – Barbary macaques originally from the Rif mountains of Morocco – will snatch plastic bags from visitors. The monkeys know the bags often contain food. Very clever survivors, these monkeys live on a fortress that has been fought over for centuries.

I am on a short visit, thinking what an important part the Rock and the seas around here have played in British history. The Battle of Trafalgar was fought – and won – by the British fleet in the seas not far from where I am staying. Gibraltar stood out against Nazism and fascism in the Second World War and survives as a British territory, despite occasional friction with Spain. Everywhere, there are monuments to past conflicts adapted for peaceful use. Near my hotel was an air-raid shelter with a sign that said it was built in 1939. A man in a Jaguar sports car was using it as a parking space.

Speaking to a local, I asked if the monkeys really do snatch plastic bags. She laughed and said one even stole the hat she was wearing from her head. I rather admire these monkeys – and the view from their home at the top of the Rock, from which you can see north Africa. The currency here remains the British pound, the language is English – although many workers come over the border from Spain every morning. The food is British, Spanish and Moroccan. But many locals fear that their way of life is perhaps being snatched from them by the distant political monkeys in Westminster.

Most Gibraltarians loathe the idea of Brexit. At the Gibraltar Literary Festival, I am discussing my latest book on the subject and listening to local people, who say Britain leaving the European Union will complicate their lives unnecessarily as a result of their often fractious relationship with Spain. The recent Spanish general election has proven a boost for the far-right party Vox. One group of Gibraltarians explained to me that Vox represented a section of Spanish politics that "hates us", was a party for "fascists" and cherished the memory of Francisco Franco, Spain's former dictator. Franco is controversial even in death. After years of protests, his body was exhumed last month from Spain's great war memorial and reburied in a private family plot.

In the 2016 Brexit referendum, an astonishing 96 per cent of Gibraltarians voted to remain in the EU. Every political party here and every member of Gibraltar's parliament supported Remain. All this points to a new strain in what it means to be part of the UK and its dependent territories. Scotland, Northern Ireland, London and Gibraltar voted Remain three years ago but they were overruled by the majority of English voters. Now, in my recent visits to Belfast, Edinburgh and Gibraltar, many citizens I have spoken to and heard from in public meetings say they feel resentful that their opinions do not count for much. And that is the flaw in having a referendum in the first place.

Brexit, above all, is about the politics of identity. Those still opposed to Brexit often manage to juggle several layers of identity – Scottish, Northern Irish or Ulster, Gibraltarian, plus British, plus a citizen of the EU. Brexit means being forced to drop one identity – that of EU citizen – and in Gibraltar that could at best be a significant inconvenience at the Spanish border. Those in favour of Brexit say losing freedom of movement in the EU is a price worth paying to restore "lost sovereignty", and avoiding "interference" from the EU.

But sovereignty, as I learned from the leading British jurist Sir David Edward, is a slippery idea. Sir David said that at its core sovereignty is about power. Within the EU, the British have accepted some dilution of sovereignty in order to achieve a greater economic and political power as part of the world’s richest trading bloc. But in Scotland, Northern Ireland and most notably here in Gibraltar, loss of sovereignty means something different. It means, some Gibraltarians said, a sense of powerlessness to influence the faraway British government in Westminster.

All this political turmoil adds up to a very profound change in the way the parts of the UK and dependencies like Gibraltar now see themselves, with many fearing that the union itself may be in jeopardy. It would be a truly great irony if the Conservative party of Boris Johnson, steeped in the history of the British empire and of the union of the UK, were to “get Brexit done" – as their slogan tells us – and then find that Brexit, like one of those cheeky Gibraltar monkeys, snatches away the historic structure of the country that they claim they wish to preserve.

Gavin Esler is a journalist, author and presenter