Mr Scholz's centre-left Social Democratic Party (SPD), the Greens and the liberal Free Democratic Party all saw support fall in the elections on Sunday, according to projections.
The conservative opposition won in both polls, as expected, but the far-right Alternative for Germany (AfD) gained ground, causing concern about its growing appeal.
Nearly 14 million people were eligible to vote in the states, almost one in five of Germany's electorate. The polls were seen as a crucial indicator of the population's mood, with surging immigration and economic woes among key topics.
Two years after coming to power, the polls were an “interim report card” for Mr Scholz's coalition, news weekly Der Spiegel said in a commentary.
“The results are disastrous,” it said. “The coalition needs a reset if it wants to be re-elected in two years.”
For the anti-immigration AfD, the votes were the latest sign of growing momentum and showed their appeal was extending beyond their traditional strongholds in the east, observers said.
“Hesse and Bavaria are a sign that the AfD can build on solid ground in the west as it has in the east,” the daily Frankfurter Allgemeine said.
“The reason for this is widespread weariness with migration policy.”
The elections came after a torrid two years for Mr Scholz's government, which has had to contend with Russia's invasion of Ukraine and an ensuing energy crisis that plunged Germany into recession.
Adding to the problems, the chancellor's coalition has been consumed by bitter infighting on issues ranging from climate laws to spending cuts.
Not helping the cause of the SPD and its coalition partners, both states are conservative strongholds. Hesse had been ruled for 24 years by the main opposition Christian Democratic Union and Bavaria since 1957 by the Christian Social Union, headed by Markus Soeder.
The SPD had sought to gain ground in Hesse by fielding a heavyweight to run for state premier, Interior Minister Nancy Faeser.
But the party won just 15 per cent of the vote, almost five percentage points below its last result in 2018, according to projections late Sunday from broadcaster ARD.
The CDU maintained its first place in Hesse and extended its lead by over seven points to 34.5 per cent, the projection showed.
State premier Boris Rhein is set to retain his job, while Ms Faeser is left facing questions about her political future.
The AfD looked to have gained about five percentage points in both Bavaria and Hesse, building on recent local poll wins, although it is unlikely to enter government in either state.
Immigration was a central theme at the polls as Germany – like elsewhere in Europe – faces a surge of new arrivals, reviving memories of a major influx in 2015.
The victory of the CSU – the sister party of the CDU – in Bavaria was widely expected and state premier Mr Soeder will retain his post.
But with vote projections showing the party's worst result for decades, it could deal a blow to his ambitions to one day run for chancellor.