This started as simple job on one saturday morning. I often use the ranger file manager in my terminal. It works usually fine on Linux, but on macOS it occasionally freezes.
It happened to me again this morning, so I decided to finally dive in and find the problem. This article describes my debugging journey. Although it seemed pretty straightforward and barely deserving any attention in the beggining, the computer kept stabbing be in the back so I had to note down all the things that went wrong to relief my frustration.
When I do anything python-related, I just use the PyCharm IDE (more specifically, CLion with the python plugin). I really enjoy using these JetBrains product as they fit my needs the best (and it is also a Czech based company, kind of, which is nice).
I started the IDE as usual, expecting nothing more than a few minutes of debugging, loaded the ranger project, setup the virtualenv, started debugging… And boom! The ranger instance cannot run in the IDE embedded terminal, because it’s not a terminal at all.
_curses.error: setupterm: could not find terminfo database
Ok, this is not nice, but this shouldn’t be a problem. I will run the ranger in Terminal.app and attach to it remotely.
I do so, but what happens? The “Attach to remote process” is timing out. One moment with our friend Google and I found at least three issues reporting this behaviour. Patch was promised, but apparently never delivered.
The Python comes with a debugger called pdb. Invoking ranger in pdb however brings no results as the ranger takes over the terminal and I lost the access to the pdb console. Reading the documentation did not suggest that the console can be bound to a socket or something else so I gave up on this.
So when the first attempt failed, I decided to give a chance to builtin macOS tools such (those referenced from Activity Monitor.app).
I /usr/bin/sampled the frozen ranger few times and got some backtraces suggesting that the problem is in the 2411 read (in libsystem_kernel.dylib) + 10 [0x7ff813bf63ba] function.
This tells me what syscall is the app stuck at, but no information on the backtrace (I obviously got a backtrace in the sample output, but this is backtrace of the python interpreter, not the one of my python program).
Similar results were given by spindump of the process.
When all the above failed, I decided to get a core dump of the whole process, hoping that I will somehow read the python backtrace from it. So I started the ranger again, fired up lldb, wrote “attach –pid XXXX” and boom again:
error: attach failed: attach failed (Not allowed to attach to process. Look in the console messages (Console.app), near the debugserver entries, when the attach failed. The subsystem that denied the attach permission will likely have logged an informative message about why it was denied.)
I should’ve seen that coming. The omnipresent security of macOS does not allow me to simply debug my processes. Googling again and I found that when debugging a process is necessary, it must be codesigned with proper entitlements.
This is dumb, as the process running was not my binary, but the system-wide python interpreter which I cannot modify (and thus codesign). Luckily before I jumped to building my own python for this purpose, I realised that I also have the version from brew, which is owned by me. Signed it with the proper entitlements, launched ranger with the proper python binary and the lldb happily connected and I got my coredump.
Before I started digging in the produced coredump, I bumped to one reply on StackOverflow suggesting that there is a wrapper around PDB that is called RPDB which (although terribly outdated) does precisely what I needed: Bind the debugger console to the socket and let ranger use the console for ncurses interface.
After I spent more time than I wanted on finding the problem, I read through the related issue #1787 more carefully just to notice that there is linked pull request #2502 that also contains working solution. Let this blogpost be a reminder for me, that if one reads carefully, half of the weekend doesn’t have to be spent on discovering of already discovered.