A cohort of right-wingers in his party derailed an 11th attempt to elect him on the third day of voting.
Republicans took over the House in November's midterm elections, but the impasse has left the chamber unable to swear in members or pass bills.
The House has adjourned until Friday.
Not since 1860, when the United States' union was fraying over the issue of slavery, has the lower chamber of Congress voted so many times to pick a Speaker. Back then it took 44 rounds of ballots.
A group of 20 hard-line Republican lawmakers are refusing to give Mr McCarthy the necessary 218 votes.
The rebels are sceptical of the California congressman's conservative bona fides, despite his endorsement from former President Donald Trump.
One of the dissidents, Ralph Norman of South Carolina, told the BBC he simply does not trust Mr McCarthy.
The congressman said Mr McCarthy's team had threatened political retaliation against them if they did not fall in line, in the weeks leading up this deadlock.
"We were going to be thrown off committees," Mr Norman said. "We're going to lose every privilege we had.
"And we'd basically told them, 'If we can't ask questions, if we can't vet out the most powerful person that we're getting ready to put in office, then we're out.'"
Meanwhile, the minority Democrats continued to vote in unison for their leader, New York's Hakeem Jeffries, the first black person ever to lead a party in Congress. But it seems unlikely that he could win over six Republican defectors to become Speaker.
Lawmakers in the sharply divided chamber will reconvene at noon (17:00 GMT) on Friday, the second anniversary of a riot by Trump supporters at the US Capitol.
Despite the holdouts, Mr McCarthy - who has served as the top House Republican since 2019 - has won support from more than 200 Republicans, over 90% of his caucus. They are growing restless as their agenda stalls.
"I'm very worried about it and I'm on the intelligence committee," said Pennsylvania Republican Brian Fitzpatrick. He added that he and the other committee members are not able to received classified briefings until lawmakers are sworn in.
Rules do not require the speaker to be a member of the House, and on Thursday, Florida Republican rebel Matt Gaetz cast a protest ballot for Mr Trump to serve in the role.
"This ends in one of two ways: either Kevin McCarthy withdraws from the race or we construct a straitjacket that he is unwilling to evade," he said.
Colorado Republican Lauren Boebert nominated a lawmaker from Oklahoma, telling her colleagues to move past Mr McCarthy.
"It is not happening," she said, adding that Republicans "need to get to a point where we start evaluating what life after Kevin McCarthy looks like".
Mr McCarthy has offered a number of concessions to the rebels, including a seat on the influential rules committee, which sets the terms for debate on legislation in the chamber. He also agreed to lower the threshold for triggering a vote on whether to unseat the Speaker, to only one House member.
During Thursday's eight-hour session, he was seen huddling with aides and having animated one-on-one talks with colleagues.
The Speaker of the House is the second in line to the presidency, after Vice-President Kamala Harris. They set the agenda in the House, and no legislative business can be conducted there without them.
In November, Republicans won the House by a slender margin of 222 to 212 in the 435-seat chamber. Democrats retained control of the Senate.
Watch: Republican Gaetz nominates Trump for Speaker