Libyan authorities shot dead three Sudanese trying to flee detention late on Monday, after they failed to cross the Mediterranean to Europe, the UN migrant agency said.
"Staff from the International Organisation for Migration in Khums, reported that local authorities started shooting when the migrants attempted to escape from the disembarkation point," the agency said.
The migrants were among 70 getting off a vessel that was intercepted and sent back by the Libyan coastguard.
War-ravaged Libya is a major route for migrants seeking to reach Europe and it hosts an estimated 654,000 of them who are often living in cramped conditions with little access to health care.
In recent months, hundreds of migrants have been stopped at sea and their vessels sent back to Libya despite the risk of violence there.
The IOM and the UN refugee agency have said Libya should not be classified as a safe port for migrants and that they should not have to disembark there.
They want an alternative scheme to take people rescued or intercepted at sea to safe ports.
IOM Libya chief Federico Soda said 31 of the 70 migrants who left the boat were taken into detention and the others reportedly escaped.
Two died at the scene of the incident and a third on the way to hospital.
"The use of excessive violence results yet again in the senseless loss of life, amid a lack of action to change a system that often fails to provide any degree of protection," Mr Soda said.
In May, 30 mostly Bangladeshi migrants were abducted in Libya and shot dead in a southern city by an armed group.