"Nationalism" and "libertarianism" are pretty vague terms, but I can briefly explain the compatibility between the two.
If by "nationalism" you mean not letting foreign people into the nation and by "libertarianism" you mean having a small government that doesn't intervene in the economy and respects individual rights: yes, a government could simply protect the borders with a small military and then do nothing else. That would be libertarian and nationalistic.
If by "nationalism" you mean the government forcing ideals, funding businesses, prohibiting certain behaviour and other sorts of hyper-nationalistic stuff: no, there really isn't any form of libertarianism that would be consistent with a government that forces its beliefs on citizens and/or intervenes in the economy etc.
What about small nationalism (the first one) and libertarianism/anarchism (based on the NAP)?
Well, if 100% of the nation voluntarily supports nationalism, i.e. border protection, then yes, nationalism is completely valid. If everybody on a single piece of land decides that they collectively own that land and they want to make sure that nobody else invades, then they are free to protect it with force. If a family can prevent people from entering their house, then a nation can prevent people from entering its borders.
If 100% of a nation decides to enforce the exact same thing, but foreigners are already living on the land, can the nation deport those foreigners? No. The native people could segregate themselves or simply wait for the foreigners to leave/die.
As for when a country "democratically" decides to invite the 3rd world into its borders… Seems pretty aggressive to me. Forcing people to live with statistically violent immigrants seems unnecessary and destructive.