Although Mr Draghi won the vote in the upper house by 95 to 38, dozens of senators refused to take part, leaving his 18-month-old administration in tatters, with an early election in September or October the most likely outcome.
Italy had been enjoying months of relative stability, during which the respected former central banker helped to shape Europe's response to Russia's invasion of Ukraine and boosted the country's standing in financial markets.
"Politics have failed," said Foreign Minister Luigi Di Maio, who had repeatedly called for the coalition parties to rally behind Mr Draghi.
"The future of Italians was at stake. The effects of this tragic choice will be seen in history."
Mr Draghi is likely to tell the lower house of Parliament on Thursday that he intends to stad down, a source said.
He will then hand in his resignation to President Sergio Mattarella, who is expected to announce elections.
Mr Draghi tendered his resignation last week after one of his partners, the populist Five-Star Movement, failed to back him in a confidence vote on measures tackling the high cost of living.
Mr Mattarella rejected the resignation and told him to go before Parliament to see if he could keep the broad coalition going until the planned end of the legislature in early 2023.
In a speech to the Senate, Mr Draghi made a plea for unity and set out issues facing Italy ranging from the war in Ukraine to social inequality and rising prices.
But the Five-Star Movement again decided not to back him, claiming he had not addressed their core concerns.
The rightist Forza Italia and League parties also decided to shun the vote, saying they wanted a commitment that Mr Draghi was willing to forge a new administration without Five-Star and with fresh policy priorities.
But he appeared unwilling to lead a new coalition, telling the Senate that Italians backed his government, which was formed to help the country recover from the Covid pandemic.
"The support I saw in this country [for the coalition] ... is without precedent and convinced me to repropose a pact for government and ask you to vote on it," Mr Draghi told the Senate before the confidence vote.
"You will decide."
Mr Draghi is likely to remain in a caretaker capacity until the election, which polls say will be won by the conservative bloc including the far-right Brothers of Italy party, the only major group that refused to join the unity administration.
"Do I already have a team of ministers ready?" asked Brothers of Italy leader Giorgia Meloni, who is tipped to lead the polls in any early ballot and could be the next prime minister.
"I have my own ideas on how this nation should be governed, what should be done, what its industrial strategy should be."
The political turmoil comes at a tough time for debt-laden Italy, the third largest economy in the euro zone, where borrowing costs have risen sharply as the European Central Bank starts tightening its monetary policy.
European Economy Commissioner Paolo Gentiloni said on Twitter that the "irresponsible" move against Mr Draghi could lead to a "perfect storm" and warned of "difficult months ahead" for Italy.
The head of Italy's centre-left Democratic Party, another coalition party, bemoaned what he called a "mad day" and said Parliament had gone against the will of the people.
"Italians will show themselves to be wiser at the ballot box than their politicians," parry chief Enrico Letta wrote on Twitter.