The charity Medical Justice says there is evidence of the “harm inflicted” on detainees who have been told they will be sent to the East African nation.
It said that their health and well-being had been “severely” affected by the policy.
Several asylum seekers, the Public and Commercial Services Union and charities Care4Calais, Detention Action and Asylum Aid are questioning the legality of the Home Office policy, with the next court hearing scheduled for Monday.
“The policy is damaging in general for anyone, acutely so for such vulnerable torture and trafficking survivors who are already paying a high human cost even before any flights have taken off to Rwanda," Medical Justice said as it published a report on Thursday.
“The first removal flight to Rwanda was halted. Yet the health and well-being of those targeted has already been severely impacted.”
Inside a refugee camp in Rwanda - video
Director Emma Ginn said: “Medical Justice calls for the immediate and urgent release from indefinite immigration detention of all those targeted with removal for Rwanda, and for the policy to be abandoned."
The charity said its doctors, who visit and assess the health of detainees in immigration removal centres, had “described the severe impact of the threat of removal to Rwanda on mental health”.
Migrant crossings on the English Channel surge amid heatwave - in pictures
Medical Justice doctors found “extremely high rates of evidence of torture, trafficking and other vulnerabilities” in the group due to be removed, the findings show.
The report said the prospect of removal is “exacerbating” detained people’s mental health conditions, including depression, anxiety and post-traumatic stress disorder.
It said the prospect was causing them to experience “fear, confusion, uncertainty about their safety, and a loss of hope”.
For some it has “increased their risk of self-harm and suicide”.
UK minister says Britain will push ahead with migrant deportations to Rwanda - video
The charity said it had been contacted by 51 people being held in immigration detention who were told they could be sent to Rwanda, highlighting the cases of 36 in its report.
Of 17 people on whom Medical Justice doctors had conducted clinical assessments, 14 had evidence of a history of torture and six had “indicators of trafficking”, and 15 had a “diagnosis or symptoms of PTSD”, it said.
“One is likely to have a psychotic disorder and lack capacity to even instruct his solicitor," the report said.
"One requires urgent investigations to rule out recurrence of a previous brain tumour.
“Some 11 people had suicidal thoughts in detention, including one who attempted suicide twice.
"Some were clinically considered to be at high risk of suicide if threatened with removal to Rwanda.”
UK's first Rwanda deportation flight cancelled - video
The Home Office responded: “The Medical Justice report features a number of inaccuracies and misrepresentations about the policy which we have previously addressed.
“We have been clear from the start that no one will be relocated if it is unsafe or inappropriate for them, and our thorough assessment of Rwanda has found that it is a fundamentally safe and secure country, with a track record of supporting asylum seekers.
“Through this world-leading Migration Partnership, those arriving in the UK illegally via dangerous and unnecessary routes may be relocated to Rwanda to claim asylum and have the chance to rebuild their lives there.”