Mikel Arteta's men led the Premier League for 248 days last season before they were reeled in by Pep Guardiola's side, who then marched on to clinch a historic treble.
But the Gunners have reinforced this summer and new signings Declan Rice, Kai Havertz and Jurrien Timber all started the match.
It was City who drew first blood, though, as substitute Cole Plamer curled home a beautiful strike in the 77th minute.
Arsenal keeper Aaron Ramsdale made a couple of vital saves before Gunners' substitute Leandro Trossard got them back into the contest with a deflected equaliser deep into stoppage time.
In the resulting shoot-out, Kevin De Bruyne hit the bar and Ramsdale saved Rodri's spot kick before sub Fabio Vieira sealed it for the Londoners to spark wild celebrations.
City, who won the FA Cup and their first Champions League trophy as well as the league last season, begin the new campaign away at Burnley on Friday, August 11.
Arsenal, who finished as runners up, kick off the new season at home to Nottingham Forest the following day.
You can see our player ratings from the Wembley showpiece in the gallery above.