Search online for the phrase “immigration detention” and you’ll notice lots of hits over the past few months – mostly because the federal government now holds a record number of immigrants in detention facilities. That number will soon surpass 45,000 – a mid-sized city of immigrants locked in prison-like conditions. This week, over two hundred organizations, including the ACLU, wrote a letter to Homeland Security Secretary Jeh Johnson expressing alarm over record immigration detention rates.

The prevailing narrative surrounding immigration detention evokes images of young, mostly male, migrants, many of them “dangerous criminals,” who were caught either trying to enter the United States illegally or living here without authorization, and who are merely being held until they can be deported. But that view is false and leads to widespread public misperceptions about who these detainees really are and why they are entitled to due process protections that the Obama administration has denied them.

Many of the people held in immigration detention centers are in the country lawfully. Most have not committed any crime at all, and those that have are, with rare exceptions, not “dangerous criminals.” Nor are most immigration detainees merely awaiting deportation. Rather, many of them have strong claims for relief, which they often win.

The largest group of people currently in immigration detention are asylum seekers, almost all of whom have no criminal record. Many of them presented themselves at the border and requested protection from persecution and other human rights abuses. Another large group of immigration detainees are lawful permanent residents of the United States. Many of them have lived in this country nearly their entire lives. The government wants to deport them because of crimes they committed – often relatively minor and from long ago – for which they have already served their sentences. Like the asylum seekers, many of these detainees have strong legal defenses that they end up winning.

These distinctions are important both legally and morally. Under U.S. and international law, asylum seekers who flee persecution have a right to request protection and a right not to be deported until that request is adjudicated. They broke no law by asking for our aid. Those who argue otherwise are mistaken, and would do well to remember the lessons of World War II, when the U.S. closed its doors to Jewish refugees who were sent back to Germany and killed in the Holocaust.

As my colleague Carl Takei of the ACLU National Prison Project pointed out in a recent blog, the U.S. has a clear obligation under international law to treat people seeking refuge from violence fairly and humanely. Yet, the Obama administration has chosen to imprison them while their claims are investigated, often with little or no access to legal assistance. We betray our nation’s values when we treat these people as criminals.

We also betray our values by detaining many other immigrants, including longtime lawful residents of the U.S. Those people often face permanent banishment from the country they consider home, and permanent separation from their family members, many of whom are U.S. citizens. Because their cases are often complex, they take months and often years to decide. But again, the government locks them up, even though many of them will win their cases.

We believe all people – including undocumented immigrants and those who have lost their immigration cases and are awaiting deportation – are entitled to due process and should only be detained if absolutely necessary to ensure their removal. The Due Process Clause protects all “persons,” not just those who have their papers in order. But it is important to remember that many of the people whom the government imprisons in its immigration detention centers do have a right to stay in this country. They have broken no immigration law and will never be deported. While the government seeks to justify the inhumane treatment it affords such people by implying that they are on their way out of the country, that picture is false. These immigrants are not, as the prevailing narrative goes, awaiting deportation. They are awaiting justice.

In a few weeks, the ACLU will be arguing before the U.S. Supreme Court on behalf of thousands of immigrants who have been detained for months, and often years, without being afforded the most basic due process protection: a hearing before a judge who can consider whether their detention is necessary. The case, Jennings v. Rodriguez, will consider whether a 2015 ruling in our favor by the Ninth Circuit Court of Appeals was correct. The Ninth Circuit found that individuals are entitled to a bond hearing once their immigration detention exceeds six months. The Supreme Court will hear the case on Nov. 30. (Read more on the ACLU’s recommendations for reducing the immigration detainee population here.) A victory would not end immigration detention, but it would represent a modest improvement by ensuring that people detained for lengthy periods can at least ask a judge to release them while their cases remain ongoing.

More than a dozen leading legal and international human rights organizations, including the United Nations High Commissioner for Refugees (UNHCR), have joined the effort to obtain basic due process protections for immigration detainees by submitting amicus briefs in support of the ACLU’s argument that immigrant detainees deserve a hearing to determine whether they should be released on bond after six months of incarceration.

In today’s politically charged environment, thousands of people who seek nothing more than freedom from persecution or the chance to remain with their families in the country they consider home have been demonized as a threat to our nation’s core values. But imprisoning people without a hearing for months or years represents a far greater threat to those values. It is time to end the needless and immoral mass incarceration of immigrants. Ensuring basic due process to those detained for months or years would be a good place to start.

Ahilan T. Arulanantham is director of advocacy and legal director at the ACLU of Southern California. He will be arguing Jennings v. Rodriguez at the U.S. Supreme Court on Nov. 30.