They reached consensus that the system, in place since 2020, "should be fully used" to boost the number of migrants returning home after their asylum applications fail, said Sweden's Migration Minister, Maria Malmer Stenergard.
Sweden currently holds the EU presidency and chaired the meeting in Stockholm.
"Should intensified political and diplomatic efforts not produce the desired results, member states call on the Commission to come back to the Council with proposals on visa restrictions," Ms Malmer Stenergard said.
The tougher line was reflected in a letter European Commission President Ursula von der Leyen sent to leaders of EU countries on Thursday, before a February 9-10 summit that will discuss the issue.
Ms Von der Leyen said EU member states could sign up to a pilot scheme over the first half of this year to speed up screening and asylum procedures for eligible migrants, and "immediate return" for those not deemed to qualify.
Migrants cross from Morocco to Spanish enclave of Melilla - in pictures
She said she wanted the EU to draw up a list of "safe countries of origin", and for the bloc to strengthen border monitoring on the Mediterranean and western Balkans routes migrants use to get to Europe.
The EU planned to put in place migration deals with countries such as Bangladesh, Pakistan, Egypt, Morocco, Tunisia and Nigeria "to improve returns … and to prevent departures", Ms Von der Leyen said.
In Stockholm, EU home affairs commissioner Ylva Johansson said many of the bloc's member countries were under "huge pressure", receiving nearly one million asylum applications last year.
Migrant children rescued in French waters - in pictures
European Commission statistics show a low rate of effective returns.
In 2021, out of 340,500 orders for migrants to be returned to their countries of origin, only 21 per cent were carried out.
The EU funds various reintegration programmes in countries that readmit their citizens who have been denied asylum in Europe.
These are separate from deportations or forced returns based on a court or administrative order, which are often carried out under escort and typically do not include in-country assistance.
Sweden, the government of which relies on a far-right party, the Sweden Democrats, to stay in power, wants EU countries to use visas, foreign policy and development aid to press outside countries on the returns issue.
So far, the EU has applied the visa restriction against only one country: the Gambia, for whose citizens obtaining a Schengen visa is more difficult and costly.
Afghan migrants in France set sights on the UK – in pictures
The commission in 2021 proposed the mechanism be extended to Bangladesh and Iraq, but that has not happened.
Ms Johansson said after a November visit to Bangladesh that the threat of the visa sanctions had prompted Dhaka to become more "politically open" to accepting irregular migrants back from Europe.
The overall tone on migration has hardened in Europe since 2015-2016, when it took in more than a million asylum-seekers, most of them Syrians fleeing the war in their country.
The bloc in 2016 struck a deal with Turkey to prevent much of the onward passage of irregular migrants into Europe.
Afghan migrants detained in Turkey - in pictures
Austrian Chancellor Karl Nehammer said on Monday, during a visit to that border region, that the fence would cost about €2 billion ($2.18 billion) and called on the European Commission to fund it.
The commission has been reluctant to do that, emphasising instead the role of Frontex, the bloc's border patrol agency, which EU member states can call on.
"It's about strengthening the fence that is there," Mr Nehammer said in Vienna on Thursday.
"The commission categorically says, 'No, there is no money for fences'. That can't be the final word."
The current system to manage asylum and the visa-free Schengen zone had failed, he said.
Ms Johansson said she objected to the fence proposal on financial grounds, as the European Council representing member states had cut her department's budget for the 2021-2027 cycle.