Murray Rothbard, one of the most prominent libertarians of the 20th century, and one of the leading exponents of anarcho-capitalism, started out as sympathetic to free immigration. However, in his later life, he became opposed to the idea and fleshed out arguments against open borders based on the anarcho-capitalist counterfactual. The arguments were further developed by Hans-Hermann Hoppe and challenged by other libertarians (more information at the link).
Rothbard explained his thoughts in a paper titled Nations By Consent: Decomposing the Nation-State for the Journal of Libertarian Studies (11:1, Fall 1994). The paper is available online as a PDF here.
I lost a lot of respect for Rothbard around 1990 when he reversed his lifelong support for free immigration. If anything ever deserved Rothbard’s classic “monstrous!” denunciation, it is our “kinder, gentler” Berlin Wall built to keep people from living and working in the U.S. because they happened to be born elsewhere. Rothbard had always refused to justify one injustice with another, but overnight the welfare system became his rationale for cutting immigration below its already heavily restricted level. When Libertarian Party presidential candidate Ed Clark made the same argument in 1980, Rothbard was outraged, citing it as “probably the greatest (or perhaps the second greatest) single scandal of the Clark campaign”