Mr Sunak was filming a clip in Lancashire to promote his levelling-up funding plan when he says he made a “brief error of judgment” by removing the safety device.
He “fully accepts this was a mistake and apologises”, a spokesman for Downing Street said.
“The Prime Minister believes everyone should wear a seat belt,” the spokesman added.
“It was an error of judgment. He removed it for a short period of time to film a clip, which you’ve seen, but he accepts that was a mistake.”
Fines of up to £500 ($618) can be issued for failing to wear a seat belt when one is available.
There are a few exemptions, including when a car is being used for police, fire and rescue services and for certified medical issues.
But Downing Street did not believe there was an exception for travelling in a ministerial car.
In the video, police motorbikes can be seen escorting the car as Mr Sunak addresses the camera.
Rishi Sunak through the years — in pictures
“We are aware of the matter and we will be looking into it," Lancashire police said.
The government has considered toughening seat belt rules to ensure drivers not wearing them will receive penalty points.
Mr Sunak was also facing criticism for travelling to Blackpool in a taxpayer-funded RAF jet rather than using a train, and then taking a 28-minute flight to Darlington.
Labour said his video was adding to “endless painful viewing” after he was previously seen struggling to make a contactless payment with his card.
“Rishi Sunak doesn’t know how to manage a seat belt, his debit card, a train service, the economy, this country,” a Labour spokeswoman said.
“This list is growing every day and it’s making for endless painful viewing.”
Everything you need to know about Rishi Sunak — video
Liberal Democrat deputy leader Daisy Cooper said: “It seems like the PM is getting too used to flying around in private jets that he’s forgotten to wear a seat belt in a car.
“The fact he’s breaking a basic law is just embarrassing and frankly dangerous.”
The Automobile Association warned of the dangers of not wearing a seat belt.
The Royal Society for the Prevention of Accidents said: “Everyone should take seat belt laws seriously — whoever and wherever they are."