The legend of the Sibylline Books tells us that in an ancient city, a woman offered to sell its citizens 12 books containing all the knowledge and wisdom in the world, for a high price. They refused, thought her request ridiculous, so she burned half of the books right then and there, and then offered to sell the remaining six for double the price. The citizens laughed at her, a little uneasily this time. She burned three, offering the remainder, but doubling the price again. Somewhat reluctantly – times were hard, their troubles seemed to be multiplying – they dismissed her once more. Finally, when there was only one book left, the citizens paid the extraordinary price the woman now asked, and she left them alone, to manage as best they could with one-twelfth of all the knowledge and wisdom in the world.

Books carry knowledge. They are pollinators of our minds, spreading self-replicating ideas through space and time. We forget what a miracle it is that marks on a page or screen can enable communication from one brain to another on the far side of the globe, or the other end of the century.

Books are, as Stephen King put it, "a uniquely portable magic" – and the portable part is as important as the magic. A book can be taken away, kept hidden, your own private store of knowledge. (My son's personal diary has an ineffectual – but symbolically important – padlock.) The power of the words inside books is so great that it's long been custom for some words to be blanked out: such as swear words, as anyone encountering a "d—-d" in a 19th-Century novel will know; or words too dangerously powerful to be written down, like the name of God in some religious texts.

Books carry knowledge, and knowledge is power, which makes books a threat to authorities – governments and self-appointed leaders alike – who want to have a monopoly on knowledge and to control what their citizens think. And the most efficient way to exert this power over books is to ban them.

Banning books has a long and ignoble history, but it is not dead: it remains a thriving industry. This week is the 40th anniversary of Banned Books Week, an annual event "celebrating the freedom to read." Banned Books Week was launched in 1982 in response to a rise in challenges to books in schools, libraries and bookstores.

The Catholic Church for a long time discouraged people from possessing their own copies of the Bible, and approved only a Latin translation that few ordinary people could read anyway

In some ways we have to admire the energy and vigilance of those who want to ban books today: it used to be so much easier than this. Centuries ago, when most of the population could not read and books were not readily available, their knowledge could be constrained at source. For example, the Catholic Church for a long time discouraged people from possessing their own copies of the Bible, and approved only a Latin translation that few ordinary people could read anyway. Ostensibly this was to prevent the laity from misinterpreting the word of God, but it also ensured they could not challenge the authority of church leaders.

Ulysses by James Joyce – which turns 100 this year – was banned in Britain from 1922 until 1936

Even when literacy rates improved, as when Britain introduced education laws in the late 19th Century, books remained expensive, particularly those high-end literary works whose words and ideas were the most durable (and potentially the most dangerous). Only in the 1930s, with Albatross Books and Penguin Books, did the new mass audience for quality affordable books have its appetites satisfied.

And simultaneously, book banning was about to take on a new life, as would-be censors tried desperately to keep up with a proliferation of new titles that opened readers to new and disruptive ideas. But what was surprising about the expansion of book banning in the 20th Century is how widespread the lust was for this "protection" racket.

'Corrupting minds'

Today, the government of China, for example, continues to issue edicts against books in schools that are "not in line with the [country’s] socialist core values; that have deviant world views, life views and values" – classically flexible words that can be applied to any book the authorities disapprove of for any reason. (Although "the students don't really look at them anyway," observed one teacher in 2020, clearing the school library shelves of Animal Farms and Nineteen Eighty Fours.)

In Russia, the approach to book banning has been a remarkably public adventure, given the number of great writers the country has exported – willingly or not – to the rest of the world. During the Soviet era, the government tried to exert as much control over its citizens' reading habits as it did over the rest of their lives.

In 1958, Boris Pasternak was awarded the Nobel Prize in Literature for his novel Doctor Zhivago, which had been published in Italy the previous year, but not at home. His award angered the Soviet authorities so much (the state-controlled media called it an "artistically squalid, malicious work") that he was forced to turn down the award. The government despised the book as much for what it did not contain – it failed to celebrate the Russian revolution – as what it did: it contained religious overtones and celebrated the worth of the individual. (The CIA, seeing the "great propaganda value" of Doctor Zhivago, arranged for it to be printed in Russia.)

Dr Zhivago by Boris Pasternak was despised by the Soviet government: the state-controlled media called it an "artistically squalid, malicious work"

The banning of books in the Soviet Union led to the development of samizdat – or self-published – writing, to which we owe the continued existence of, for example, the poetry of Osip Mandelstam. Dissident writer Vladimir Bukovsky summed up samizdat: "I write it myself, edit it myself, censor it myself, publish it myself, distribute it myself, and spend jail time for it myself."

But those in the West flatter themselves if they think that it doesn't happen there. When books are banned, or attempts are made to ban, the argument is the same there as elsewhere: that it is to protect ordinary people, who are seemingly too feeble-minded to judge for themselves, from exposure to corrupting ideas.

In Britain, book banning has often been a tool against perceived sexual obscenity. It is, typically, an attempt to use the blunt force of the law to arrest social change: a tactic that always fails, but which is irresistible to short-termist authorities nonetheless.

Many writers have had their reputations burnished by their brushes with Britain's obscenity laws. James Joyce was perceptive when he said, while writing Ulysses, that "police notwithstanding, I should like to put everything into my novel" – the book was banned in Britain from 1922 until 1936, even though the legal officer responsible for the ban had read only 42 of the book's 732 pages. The "everything" that Joyce put into Ulysses included masturbation, swearing, sex and going to the toilet.

In Britain, book banning has often been a tool against perceived sexual obscenity – such as the famous trial about DH Lawrence's Lady Chatterley's Lover

DH Lawrence was a special case: his work, often containing sexual acts that Lawrence regarded with a spiritual reverence, had been the subject of a campaign by Britain's Director of Public Prosecutions for years: burning his book The Rainbow, intercepting his post to seize his poems Pansies, and raiding an exhibition of his art.

Six decades after it tried to ban the book, the British government prevented the judge's copy of Lady Chatterley's Lover from being sold overseas, so that 'a buyer can be found to keep this important part of our nation's history in the UK'

The vendetta continued beyond the grave when Penguin books published Lady Chatterley's Lover in 1960 and invited prosecution. The trial is famous: the publisher recruited dozens of writers and academics to attest to the book's literary qualities (though Enid Blyton turned them down), and the judge exemplified the state's distrust of ordinary readers when he cautioned the jury against relying on literary experts: "Is that how the girls working in the factory are going to read this book?"

(The coda to the case, in which the jury found unanimously in favour of Penguin, is a delicious irony. Three years ago, and six decades after it tried to ban the book, the British government prevented the judge's copy of Lady Chatterley's Lover from being sold overseas, so that "a buyer can be found to keep this important part of our nation's history in the UK".)

Keeping ideas alive

Meanwhile, in the US, it's a tribute of sorts to the enduring power of books that banning them remains so popular in a world where each new wave of technology, from TV to gaming and social media, attracts fears of "inappropriate" content. Schools are a particular hotbed of attempted censorship, partly because directing a child's malleable mind seems like an efficient way to cut off perceived dangers; but also because (unlike bookshops) school boards have some degree of community input.

In 1982, the year that Banned Books Week was launched, a case of attempted school censorship (Island Trees School District) reached the US Supreme Court. Here, the school board argued that "it is our moral duty to protect the children in our schools from this moral danger as surely as from physical and medical dangers." The danger they referred to were books that were "anti-American, anti-Christian, anti-Semitic and just plain filthy." (The charge of antisemitism was laid against the Jewish novelist Bernard Malamud's great novel The Fixer.) The court concluded however, in alignment with the First Amendment, that "local school boards may not remove books from school libraries simply because they dislike the ideas contained in those books."

The three most challenged books of 2021 were objected to because of their LGBTQI+ content

That has not stopped them. Foremost among the hot topics for challenges and bans on books in US schools and libraries is sex. "America seems to be very exercised about sex," as James LaRue, then director of the American Libraries Association's Office for Intellectual Freedom, put it in 2017.

Sex has traditionally meant obscenity, which led to US judge Potter Stewart's famous attempt to pin down the definition of "hardcore pornography" in a court case in 1964: "I know it when I see it." But today "sex" in book bans is more likely to mean sexuality and gender identity: the three most challenged books of 2021 in the US were objected to because of their LGBTQI+ content. This brings into question the idea that book bans are done to protect young people rather than as an attempted ideological purge, and shows a lack of imagination by censors, holding that the depiction (for example, of transgender people) causes the phenomenon rather than the reverse. This is connected to the belief that things we dislike can be safely ignored as long as we don't see them on the page: a regular top 10 member of the Banned Books list is Toni Morrison's modern classic The Bluest Eye, for its depiction of child sexual abuse.

A regular on the most challenged books lists is the modern classic The Bluest Eye by Toni Morrison

But then book censorship in the US has a long history. Its most famous early victim was Harriet Beecher Stowe's 1852 anti-slavery novel Uncle Tom's Cabin. In 1857, a black Ohio man, Sam Green, was "tried, convicted and sentenced to 10 years' imprisonment in the penitentiary" for "having in his possession Uncle Tom's Cabin." It is a twist of history that the book is now more often criticised from the progressive end of the political spectrum, for its stereotypical portrayal of black characters.

The more prominent a book, the more likely it is to attract the attention of censors. JD Salinger's The Catcher in the Rye has been challenged regularly – a teacher was sacked for teaching it in 1960, and the book was removed from schools in Wyoming, North Dakota and California in the 1980s. The argument for banning Salinger's novel is typically for profanity and vulgar language, even though its opening line today – with "all that David Copperfield kind of crap" – seems quaint.

An ultra-rare signed Catcher in the Rye was on display at the Saatchi Gallery's Banned Books fair this month

Book banning is a broad church, bringing together books that are not normally bedfellows. It encompasses everything from popular fiction (Peter Benchley, Sidney Sheldon, Jodi Picoult) to established classics (Kurt Vonnegut, Harper Lee, Kate Chopin). It has more targets than an archery contest, from occult worship (the Harry Potter series) to atheism (The Curious Incident of the Dog in the Night-Time).

There is hope, of course. The publicity from Banned Books Week keeps these books and the issue of censorship in the public eye. And there is the Streisand Effect – trying to ban books makes people more aware of them. In the US, some Barnes and Noble stores have tables of banned books, and its website has a separate category for them. In the UK, a rare book fair at the Saatchi Gallery this month displayed and sold scarce editions of banned books, from an ultra-rare signed Catcher in the Rye (£225,000 / $264,000) to Copernicus's On the Revolution of the Heavenly Spheres, which outraged the Church in 1543 by suggesting that the Earth was not at the centre of the solar system (£2m).

But eternal vigilance, not just by the American Libraries Association but by all readers everywhere, is the price of keeping our ideas alive. As the story of the Sibylline Books tells us, books can burn, their knowledge can be lost, and nothing is forever.

Banned Books Week runs from 18 to 24 September.