The Germany international scored the fastest goal in an FA Cup final, with just 12 seconds on the clock. Bruno Fernandes levelled from the penalty spot but Gundogan struck the winner early in the second half.
Gundogan's stupendous opener looked set to spark a derby rout. From the kick-off, City played the ball back to keeper Stefan Ortega, who hit it long for Erling Haaland to flick the ball on with his head.
United's Victor Lindelof nodded the ball clear as he battled with Kevin De Bruyne but it flew straight to Gundogan, who struck the sweetest of volleys to give his side the perfect start.
But United drew level against the run of play in the 33rd minute.
City were enraged by the decision to award handball against Jack Grealish as he got the faintest of touches on an Aaron Wan-Bissaka cross, but captain Fernandes kept his cool to slot home in front of the opposition support.
But Pep Guardiola's men would not be denied a seventh FA Cup triumph. Gundogan volleyed home what proved to be the winner from the edge of the box early in the second half.
Victory in arguably the biggest Manchester derby of all time to land the league and FA Cup double now means City can win the treble in next weekend's Champions League final against Inter Milan.
After the match, Gundogan said: "To win a final is always special. Every trophy is special. This group deserves it. We played really well, second half was amazing and we deserved to lift another trophy.
"Obviously we know how we want to play the kick off and we aim to do that, to go long for Erling and try to get a second ball. It just placed amazingly for me and I just had to hit it. Obviously it was a quite good strike."
The Germany international is out of contract in the summer and some sources are linking him with a move to Spanish giants Barcelona.
But the City captain said: "I don't need these kind of days to feel appreciated and feel special in this club. I know that and that is why I have been here seven years. Nothing is decided yet, we will see what will happen."
Manchester City manager Guardiola, said: "We can now talk about the treble. Of course we still have to win the Champions League. We performed so well for our city and our fans.
"They follow man to man so tight and we didn't expect them to defend so narrow. We found more spaces in the second half.
"Tomorrow and the next day they can have off. We've then got three or four sessions to train for Inter Milan. We'll travel on Thursday morning after training.
"It was so important for us today. The FA Cup is so nice.
"I'm a Barcelona fan but I will love this club for the rest of my life. I know today we gave our fans a good gift to enjoy against our neighbours."
On Ilkay Gundogan, he added: "What a season he has had. The skills are there but he also had the mentality. He plays the big games like they are friendlies, he can cope with the pressure."
Manchester United boss Erik ten Hag, said: "We are broken, disappointed of course, but I said I am proud of my team. We did very well, we conceded two soft goals but we were in the game. There was a great spirit and very good organisation.
"They were tough goals, they were so soft and avoidable. When you play against City and you don't concede almost nothing from open play it is a big credit for the team but if you concede goals like this it is disappointing.
"This team showed resilience, character and personality. We know we have a way to go but this will make us better. It was a test for us, we didn't succeed but we can take a lot of positives into next season."
United goalscorer Fernandes said: "Everyone is down and it's really difficult. We wanted to end the season in a different way but it wasn't possible.
"We've made big steps for next season. It wasn't a successful season overall but it was a good one. We have to come back next season and try to win bigger trophies."
Of Gundogan's stunning early opener, he said: "We were too soft and gave up too much space for the first goal but it's an amazing strike.
"If players are tighter then maybe they can stop it but it's an amazing strike."
The Portuguese midfielder said United's supporters deserved more.
"Our fans have been great all season and they deserved a different result today. We're disappointed that we couldn't give the fans the joy they deserve.
"We are grateful for everything they have done this season. I promise we will come and try again next season."