lsof: command not found in a packaged Mac app, and the port reclaim that never ran

Here is the whole of what the shipped app had to say about a port it could not free:

1WARN reclaim: lsof unavailable; port left occupied  error="lsof"

One line, one word of context. error="lsof" is a bare-name ENOENT wearing a coat — the process tried to run a program called lsof, the kernel said there is no such file, and we laundered that into a sentence about ports.

The port stayed occupied — 8090, the one our local MLX engine sidecar mounts on. The next mount refused, because the address was still in use. And the reclaim code that was supposed to fix exactly this had, in the packaged application, never once executed.

macOS keeps lsof somewhere your app cannot see

On this machine there is exactly one lsof:

1ls: /opt/homebrew/bin/lsof: No such file or directory
2ls: /usr/bin/lsof: No such file or directory
3-rwxr-xr-x  1 root  wheel  306768 Aug  2 02:15 /usr/sbin/lsof

/usr/sbin. Not /usr/bin, not Homebrew. And /usr/sbin is precisely the directory that a packaged app's environment tends to drop. The PATH our backend ran with under the app, measured, was:

1…/Resources/bin:…:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin

Read that list again and look for /usr/sbin. It is not there. So:

1$ env -i PATH=/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin sh -c 'lsof -ti TCP:1'
2sh: lsof: command not found
3EXIT=127

That is the entire bug. In a terminal your PATH is inherited from a login shell and contains /usr/sbin, so lsof resolves and every test passes. Under the packaged app it does not, so the spawn fails before it has done anything at all. Same code, same machine, two different worlds — and the one you develop in is the one where it works.

The part that should have caught it much earlier

I want to dwell on this, because the PATH detail is a fact you look up once and the other thing is a habit.

The reclaim had two branches: resolve the port's holders and signal them, or warn that lsof was unavailable. In the packaged app, the first branch was unreachable. Not rare, not flaky — unreachable. Every reclaim in every shipped build took the warning arm, one hundred per cent of the time.

Which means the log was not showing us an occasional failure. It was showing us the program's entire behaviour. The warning arm was not a fallback; it was the implementation, and the code we thought was doing the work had no happy path at all in the environment that mattered.

The signal was there and it was legible. It was not "this warning appears sometimes". It was "this warning is the only thing this subsystem has ever said". A branch that always fires is not an edge case, it is the design — and when the always-firing branch is the one that gives up, you have shipped the giving up.

I now read logs with that question in front: not is there an error, but is there anything here except the error. It is the same failure I hit building a benchmark scorer that could not flatter itself — a green result means nothing until you have proved the thing can go red.

Fixing it: look where the thing actually lives

The repair is unglamorous. Try the places the binary is known to live, in order, and only then fall back to walking PATH:

1const LSOF_KNOWN_LOCATIONS: [&str; 3] =
2    ["/usr/sbin/lsof", "/usr/bin/lsof", "/opt/homebrew/bin/lsof"];
3
4fn resolve_lsof_in(
5    known: &[std::path::PathBuf],
6    path_env: Option<&std::ffi::OsStr>,
7) -> Result<std::path::PathBuf, LsofUnavailable> {
8    let from_path = path_env
9        .into_iter()
10        .flat_map(std::env::split_paths)
11        .filter(|dir| !dir.as_os_str().is_empty())
12        .map(|dir| dir.join("lsof"));
13    let mut searched = Vec::new();
14    for candidate in known.iter().cloned().chain(from_path) {
15        if candidate.is_file() {
16            return Ok(candidate);
17        }
18        searched.push(candidate);
19    }
20    Err(LsofUnavailable { searched })
21}

Two things in there are deliberate.

The known locations come before the PATH walk, so the packaged app finds /usr/sbin/lsof without anyone having to repair its environment first. And the failure is a typed value carrying every path that was tried, in order, which renders as:

That is a message you can act on. error="lsof" was not. When a lookup fails, the thing the reader needs is not the name of what you wanted — they know that — it is where you looked.

The regression tests pin both halves, and the first one is named after the bug:

1test tests::the_packaged_apps_path_without_usr_sbin_still_resolves_lsof ... ok
2test tests::absence_names_every_path_searched_in_order ... ok
3test engine::tests::unmount_reclaims_an_unsupervised_listener ... ok
4
5test result: ok. 43 passed; 0 failed; 9 ignored; 0 measured; 0 filtered out

We also widened the environment on the way in, appending /usr/sbin:/sbin to the backend's PATH when they are missing, on POSIX only. That is belt and braces on top of the resolver rather than the fix itself. Environment repair is easy to regress silently; the resolver holds even if someone rebuilds the launch environment tomorrow.

The second trap, which is worse than the first

Once lsof resolves, you are about to send signals to whatever it returns. So it matters enormously what it returns.

1listener pid (the one to kill) : 42893
2client   pid (a bystander)     : 42894
3lsof -ti TCP:8874              : [42893, 42894]
4lsof -ti TCP:8874 -sTCP:LISTEN : [42893]
5bare list catches the bystander: True
6LISTEN list catches bystander  : False

That is a listener and a separate client process connected to it. lsof -ti TCP:8874 returns both. The bare query does not mean "who owns this port", it means "who currently has a socket touching this port", and a client holding an established connection qualifies.

Now put that in a port reclaim. You want to free port 8090, so you ask which processes are on 8090 and signal them. The list includes every client with an open connection — and in our case one of those clients is the backend's own keep-alive pool. The reclaim would have signalled the process running the reclaim.

-sTCP:LISTEN is the whole defence:

1// `-sTCP:LISTEN` is load-bearing: a bare `lsof -i :port` also lists every process
2// holding a CLIENT connection to the port — goosed's own keep-alive pool included.

I would not have found that by reading. I found it by starting a listener, connecting a separate process to it, and printing both lists side by side. The first version of that probe had the client and the listener in the same process, so both queries returned the same single pid and the test proved nothing while appearing to pass. Splitting them into two processes is what made the difference visible.

What to take from it

Neither of these is exotic. Both are the same shape: the environment you ship into is not the environment you built in, and a check that cannot fail is not a check.

If your app spawns a system binary by bare name, go and look at where that binary actually lives, then look at the PATH your shipped process really gets — print it from inside the packaged build, do not assume it. On macOS the two that go missing are /usr/sbin and /sbin. Checked on this machine: lsof, netstat, dtrace and arp are in /usr/sbin, while ifconfig and route are in /sbin — so appending only one of the two still leaves you a class of binaries short.

And if you are about to signal a list of pids, be certain the list contains only what you think it does. A port reclaim that kills clients is not a reclaim, it is an outage with good intentions — and the day it includes your own connection pool, the thing it takes down is the thing that was trying to help.