The extraordinary find is believed to be a first for a UK institution and was discovered by an X-ray taken of Van Gogh's Head of a Peasant Woman (1885) ahead of the forthcoming exhibition A Taste for Impressionism at the Royal Scottish Academy, Edinburgh.
Hidden from view for over a century, the portrait was found on the back of the canvas with Head of a Peasant Woman, covered by layers of glue and cardboard that are believed to have been applied ahead of an exhibition in the early 20th Century.
Van Gogh was known for reusing canvas to save money by turning it round and working on the opposite side.
The portrait shows a bearded sitter in a brimmed hat with a neckerchief loosely tied at the throat. His left ear, which he famously cut off in 1888, is clearly visible.
It is thought to be from his early work and his first exploration of self-portraits, which he later became known for.
Viewers will be able to see the sketch as an X-ray image through a specially crafted lightbox.
While it may be possible to separate the paintings, the process of removing the glue and cardboard will require delicate conservation work. Research is ongoing as to how that can be done without harming Head of a Peasant Woman.
The discovery has been described as "thrilling" by Professor Frances Fowle, senior curator of French art at the National Galleries of Scotland. She said: "Moments like this are incredibly rare.
"We have discovered an unknown work by Vincent Van Gogh, one of the most important and popular artists in the world.
"What an incredible gift for Scotland, and one that will forever be in the care of the National Galleries. We are very excited to share this thrilling discovery in our big summer exhibition A Taste for Impressionism, where the x-ray image of the self-portrait will be on view for all to see."
Lesley Stevenson, senior paintings conservator at the National Galleries said they are "thrilled to bits" to have discovered the portrait.
"When we saw the X-ray for the first time of course we were hugely excited," she said.
"This is a significant discovery because it adds to what we already know about Van Gogh's life.
"There is lots to think about with regards to the next steps, but for us it is another little nugget to get us a little bit closer to an incredible artist.
"Knowing that it's there in a painting that's in the National Galleries of Scotland in a collection that belongs to the people of Scotland is incredibly important and significant.
"Hopefully it will encourage people to come along and have a look."
Art enthusiasts will be able to view the portrait as part of the exhibition which will take place between July 30 and November 13.