Senior Catholics said Cardinal Keith O’Brien's resignation was intended to stop the allegations turning into a crisis. Photograph: Angus Blackburn/Rex Features
The pope has forced the abrupt resignation of Britain's most senior Roman Catholic as the church made a frantic attempt to minimise the impact of allegations of "inappropriate acts" committed by Cardinal Keith O'Brien against fellow priests.
O'Brien stood down as archbishop of St Andrews and Edinburgh the day after the Observer published accusations by three serving priests and a former priest about his conduct towards them during the 1980s.
He issued a statement in which he ambiguously apologised for "any failures" and to those he had "offended", and announced that he would no longer travel to the Vatican to help select a successor to Pope Benedict XVI, who retires at 8pm on Thursday. O'Brien had been due to be the only British cardinal with a vote.
The cardinal revealed in his statement that he had been asked by the outgoing pope to stand down immediately. Already due to retire next month, the cardinal stated: "The Holy Father has now decided that my resignation will take effect today."
Senior Catholics said his resignation was intended to stop the allegations turning into a crisis. The church is already under pressure over unrelated abuse and corruption scandals in other dioceses.
Professor John Haldane, one of Scotland's senior Catholic theologians and an adviser to the Vatican, said O'Brien's decision was "shocking and sad" but, given the timing of the allegations and the "inevitable" media interest, it was not a surprise. "He would not want that burden to fall upon the church and the pope at what is obviously a critical moment in the life of the Roman Catholic community," Haldane said.
But the move led critics to demand that other cardinals at the centre of scandals over failures to report sex abuse by priests – including Roger Mahony, emeritus archbishop of Los Angeles, and Seán Brady, the primate of all Ireland – "recuse" themselves from the papal conclave, citing O'Brien's decision as a precedent.
Insiders said O'Brien's abrupt departure had left the Scottish Catholic church, which he had led for 10 years, disoriented and shocked. One source said it meant that only three out of eight Scottish dioceses now have full-time, permanent bishops in charge.
In a detailed statement, O'Brien said: "I have valued the opportunity of serving the people of Scotland and overseas in various ways since becoming a priest. Looking back over my years of ministry: For any good I have been able to do, I thank God. For any failures, I apologise to all whom I have offended.
"I also ask God's blessing on my brother cardinals who will soon gather in Rome to elect his successor. I will not join them for this conclave in person. I do not wish media attention in Rome to be focused on me – but rather on Pope Benedict XVI and on his successor."
That statement did not repeat his earlier denials at the weekend rebutting the allegations. After he failed to appear on Sunday to take a mass at St Mary's cathedral celebrating Pope Benedict's eight years as pontiff, his deputy hinted that O'Brien was considering his future.
In another sign the Vatican is anxious to appoint Benedict's successor quickly and smoothly, the Vatican announced on Monday that the pope had changed the rules governing the conclave so that it could begin its deliberations immediately after he formally stands down on Thursday.
The move means cardinals no longer have to wait 15 days before beginning the conclave after the papacy becomes vacant on Thursday. That means a new pope could be elected before the end of Lent.
The four complainants went public with their allegations against O'Brien at the weekend in protest at his involvement in selecting the next pope. The four men had urged him to resign immediately, arguing that they wanted the conclave electing the new pope to be "clean".
In early February, the four submitted their detailed allegations against the cardinal, which date back to the 1980s, in a letter given by an intermediary to the pope's ambassador to the UK, Antonio Mennini.
On 11 February, Pope Benedict surprised the world by announcing he was stepping down, citing his own ailing health. On 18 February, O'Brien disclosed in his resignation statement, the pope accepted his retirement request but said it was "nunc pro tunc", in order words, "now, but to take effect later".
In a pre-recorded interview broadcast by BBC Scotland last Friday, in which he also surprised many in his church and outside by calling for priests to be allowed to marry, O'Brien confirmed that he expected to retire on St Patrick's day, his birthday.
The following evening, the Observer published the allegations against him. Those included claims by one man, then an 18-year-old seminarian, that O'Brien had made an inappropriate approach one night; allegations of "inappropriate contact" with a second man, a priest; and of "unwanted behaviour" after a late night drinks session by another priest. The third priest also alleged "inappropriate contact" after night prayers.
In his statement on Monday, O'Brien implied he had been told to resign immediately, stating: "The Holy Father has now decided that my resignation will take effect today, 25 February 2013, and that he will appoint an apostolic administrator to govern the archdiocese in my place until my successor as archbishop is appointed."
Many observers assumed O'Brien would press ahead with his plans to attend the conclave: the allegations were about 30 years old, unproven and he had denied them. Experts said he was obliged, as one of only 117 cardinals eligible to vote, to take part.
Professor Tom Devine, a prominent Catholic, said O'Brien's resignation was "the gravest single public crisis to hit the Catholic church in Scotland since the Reformation and its effects in the short term are incalculable".
O'Brien had been "a courageous leader of his flock, well liked and respected," he said. Devine added, however, that some perspective was needed: the church had survived crises for centuries and was larger than a single man.
He said O'Brien's accusers should make themselves known "in the cause of transparency and indeed fairness to all. [If] Catholicism in Scotland is to move on from this tragic affair, a number of serious questions urgently require frank and honest answers from all concerned".