Through the wilds of 🍏 dev

Debugging lldb Scripts in PyCharm

2023-03-07T00:00:00+00:00

Debugging in lldb can be great. But sometimes, running commands manually becomes tedious. Fortunately, lldb provides a way to automate any action: Python scripts!

However, writing Python scripts without IDE support and a debugger is beneath any developer’s dignity. In this tutorial, I will show you how to set up PyCharm to take your lldb scripting experience to the next level. There will be:

Python and PyCharm setup for lldb scripting
PyCharm fix for process attachment on an arm64 macOS
Python modules injection for IDE and runtime support
Debugging breakpoints and commands in PyCharm

Setting up PyCharm

Let’s start by creating a directory for the project:

$ mkdir lldb_scripts && cd lldb_scripts

Now we need a python environment. Create a virtual environment from the Xcode’s embedded Python used by the lldb:

$ sh -c 'exec "$(xcode-select -p)"/Library/Frameworks/Python3.framework/Versions/Current/bin/python3 -m venv python_env'

We also need lldb’s initialization file, so might as well link it to the project:

$ touch ~/.lldbinit && ln -s ~/.lldbinit lldbinit

Open the project in PyCharm. I am on a community edition:

$ open -a "PyCharm CE" .

Go to project settings and make sure the project picked up the correct virtual environment from the python_env directory:

Script to debug

Now we need a script to debug. Make a test.py file with the following contents:

import os  
import time  
  
def test_loop(debugger, command, result, dict):  
    print(f'lldb pid is: {os.getpid()}')  
    while True:  
        time.sleep(3)  
        print(f'The time is: {time.time()}')  
  
def __lldb_init_module(debugger, dict):  
    module = os.path.splitext(os.path.basename(__file__))[0]  
    function = test_loop.__name__  
    print(f'\nRegistering {module}.{function}. Call "{function}" from lldb to run the script')  
    debugger.HandleCommand(f'command script add -f {module}.{function} {function}')
    print(f'\nAttach to lldb at: {os.getpid()}')

And add the script import to the .lldbinit file:

command script import ~/lldb_scripts/test.py

The script is pretty simple:

__lldb_init_module function is called when lldb is loaded
The function registers the test_loop command in lldb
Calling test_loop prints lldb’s pid and spins in an eternal loop printing time every 3 seconds

Start lldb and call the test_loop:

$ lldb -o test_loop

Hopefully, this is your output:

Attaching PyCharm to the script

As always, things aren’t as simple as just running Attach To Process in PyCharm. lldb is signed to run with a hardened runtime, which means PyCharm won’t be able to attach. We need a copy of lldb with a stripped-down signature to bypass this restriction. Let’s make one:

$ cp "$(xcrun -f lldb)" unsigned_lldb && codesign --force --sign - unsigned_lldb

If you made the copy outside the Xcode’s bin directory, the rpath would be incorrect. To run this lldb, we must specify rpath explicitly. Run lldb again using:

$ DYLD_FRAMEWORK_PATH="$(dirname $(xcode-select -p))/SharedFrameworks" ./unsigned_lldb -o test_loop

To attach PyCharm, go to Run -> Attach To Process... and select the pid of the lldb process (i.e., 5203 from the output above).

If PyCharm doesn’t show any processes go to PyCharm -> Preferences... -> Build, Execution, Deployment -> Python Debugger and set the Attach To Process filter string to empty so that all processes are displayed.

On an Intel machine (or an arch -x86_64 lldb) PyCharm should attach and start outputting the timer in the debugger console. But many of us no longer work on Intel machines; on an arm machine, PyCharm will fail to attach.

Fixing a broken arm

Attaching to arm targets, as of PyCharm 2022.3.2, isn’t supported out of the box. Fortunately, the issue is relatively easy to workaround.

When PyCharm connects to a process, it injects and calls a library with a C function that links the debugger to the debuggee. PyCharm bundles only the x86_64 version of that library and, on attachment, futilely tries to inject the library into an arm process.

To fix the attachment process, we need to compile the library to run on arm and preload it into lldb before PyCharm tries to call the C function from the library. The sources for the library are shipped with PyCharm and can be found in:

$ cd "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd_attach_to_process/linux_and_mac"

A glance at compile_mac.sh reveals we need to substitute x86_64 for arm64, which should be enough to compile the library. Let’s compile and link:

g++ -fPIC -D_REENTRANT -std=c++11 -arch arm64 -c -o attach_arm64.o attach.cpp
g++ -dynamiclib -nostartfiles -arch arm64 -o attach_arm64.dylib attach_arm64.o -lc

We will use dlopen to preload the library straight from the test.py script. Add the following lines (make sure the dylib path matches your path):

import _ctypes  
import platform
import os
  
def load_pydevd_library():  
    processor = platform.processor()  
    if processor != 'arm':  
        print(f'lldb is running {processor} arch, skipping the arm fix')  
        return  
  
    library_handle = _ctypes.dlopen(  
        '/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd_attach_to_process/linux_and_mac/attach_arm64.dylib',  
        os.RTLD_NOW  
    )  
    if library_handle == 0:  
        print("Library didn't load")  
        return  
    print(f"Library handle is {hex(library_handle)}")  
  
    function = 'DoAttach'  
    do_attach_address = _ctypes.dlsym(library_handle, function)  
    if do_attach_address == 0:  
        print(f"Couldn't find {function} in library at {library_handle}")  
        return  
    print(f"{function} loaded at {hex(do_attach_address)}")  
  
load_pydevd_library()

If you now run lldb, the library handle and the DoAttach function address will print out:

And attaching in PyCharm should finally work as expected:

PyCharm still tries to inject the Intel library, but everything works out since we preloaded the DoAttach function.

Working in PyCharm

UPD: I was originally using pydevd.settrace() to break execution. Actually there is need no call pydevd API directly, just set a breakpoint in PyCharm and execution will break as usual.

But here is how you could tinkerer with pydevd directly from lldb

Building Swift Code Faster

2022-09-04T00:00:00+00:00

How do you speed up the build of your Swift project? Do you use -warn-long-function-bodies or maybe even -stats-output-dir? These can find some compilation performance issues, but how do you truly take your build times to the next level?

In this article, I want to show you how tweaking some build settings can drastically speed up your clean build times.

Let us start by building Avito with the default Xcode build settings. For the benchmark, I am using M1 Pro with 32 GB RAM. Here is the result:

So our clean build time is somewhere around 240 seconds. Is that good? Is that bad? Let us compute some stats for our project.

A way to measure the size of the project is to use a cloc utility.
Here is what we got:

1 240 000 lines of Swift code
100 000 lines of Objective-C code (mostly external dependencies)
6000 lines of C code

I would say that 240 seconds is not bad. But can we go faster? Let us switch SWIFT_COMPILATION_MODE from incremental to wholemodule and see where that leads.
This change gets us to the following build duration distribution:

Now the average build time is 179 seconds. Somehow our build got faster by 25%!

How well is your build parallelized?

To understand why the build became faster, we first need to know the difference between incremental and wholemodule compilation modes.

The summary is:

When we build with incremental, the build system uses a batch mode for the swift compiler. The batch mode splits each module compilation into multiple jobs and executes those jobs in parallel.
On the other hand, wholemodule runs a mostly single-threaded compilation of the module without splitting it in any way.

So incremental looks like it should be faster due to better parallelization, but why does wholemodule perform better in the end?

incremental build is slower because the compiler has to do more work compiling a module in batches rather than compiling a module as a whole which results in some overhead.
At the same time, even though the wholemodule compilation is single-threaded, many modules still build in parallel, making this mode efficient.

The compiler documentation also makes a note that wholemodule could be faster than incremental under circumstances where many modules build in parallel:

It is, therefore, possible that in certain cases (such as with limited available parallelism / many modules built in parallel), building in whole-module mode with optimization disabled can complete in less time than batched primary-file mode

So how well is build at Avito parallelized?
To understand that, we will employ a visualization similar to that introduced in Xcode 14. It allows us to see how many modules build in parallel at a particular time.

Let us first take a look at Avito built with incremental compilation mode:

Here colored rectangles represent heavy Xcode build system invocations: swiftc, ld, and some other tools such as actool. The build seems well parallelized with the incremental compilation mode.

Now take a look at the visualization produced with wholemodule compilation mode:

We can see where Avito build is faring well and where it could do better. The reason for this specific shape of the graph is our architecture. At first, we build different utilities which don’t have many dependencies, then follow the poorly parallelized parts of the build - monolithic modules. At last, we have feature modules that don’t depend on one another and build in parallel.

Another way to look at build performance is by looking at CPU utilization. The Instruments CPU Profiler trace maps to the wholemodule graph quite well:

As expected, there is a sag in CPU utilization where parallelization isn’t perfect.

Squeezing the last bits of build performance

In an ideal build scenario, all modules would build parallel across all available cores. Unfortunately, mistakes in architecture can make such a goal unattainable.

However, there is something we can do to make the build more efficient without reengineering everything from scratch.
What if we try to leverage the best of wholemodule and incremental simultaneously? We can keep using wholemodule where the build process is parallelized well and switch to incremental for those monolithic modules in the middle. That leads to the following build distribution:

This change gave us another 14 seconds! The CPU is loaded much more evenly, and gaps in the build graph are smaller.

What’s next?

Here we only looked at clean builds. Next time I will tell you all about the incremental builds at Avito!

Does tweaking compilation modes make your builds faster? How large is your project, and how swiftly does it build? Let me know in the comments!

Hacking the Finder with lldb and Hopper

2022-05-01T00:00:00+00:00

Do you want to never press Show Package Contents again? Or maybe you just want to learn a couple of practical lldb techniques? Either way, I invite you on the journey to discover a hidden feature within the Finder!

In this article I will be using:

Hopper disassembler
Multiple lldb features
Opcode manipulation to change a program behavior

I hope you will learn something new!

The backstory

Recently I had to work a lot with packages. What kind of packages? The most usual example of a package is an .app bundle, for example, the Xcode.app:

Imagine you are working on an .app with multiple .bundle and .appex directories inside it. If you want to inspect the contents of this hierarchy, you would normally use a Finder. You build using Xcode, go to the DerivedData/Build/Products, and… unfortunately, to view the contents of each package you have to click through Show Package Contents. This is inconvenient because Finder doesn’t allow you to easily walk in and out of the package and working with multiple packages simultaneously easily gives you vertigo.

But what if it was possible to view packages as regular directories? After all, package is just a regular directory. How does Finder even understand something is a package? It turns out some file extensions such as .app are hardcoded in LaunchServices to be recognized as packages. For example, if you simply create a directory:

$ mkdir Example.app

it will show up as a package in the Finder.

You can also register your own file extensions as packages:

Reverse engineering Finder using Hopper

Hopper is a great tool to learn about how a program works. Hopper’s basic utility is a binary disassembly tool but it also has many features which can give you a better insight into how a program operates. For example, it can turn assembly back into a pseudo-code which is often much easier to reason about than an assembly.

Let’s start by opening the Finder in Hopper and seeing if we can spot something of interest:

$ hopperv4 --executable /System/Library/CoreServices/Finder.app/Contents/MacOS/Finder

Now we can search for references to “packages”:

And thankfully there are some! Of particular interest to us is the procedure lurking under the symbol -[TBrowserContainerController allowsBrowsingPackages]. If we look at the pseudo-code there is something weird about this procedure:

int -[TBrowserContainerController allowsBrowsingPackages]() {
    return 0x0;
}

It seems that this is a method that returns a boolean flag but it always returns false. Why could that be? Most probably when Finder is compiled for internal testing at Apple, this method comes with some logic inside but for release build this logic is removed.

So what happens if we flip the output of this method to return 1 (i.e. true)? Let’s use lldb to find out. Inconveniently Finder is protected by SIP, so to attach to Finder you will have to disable it. After SIP is disabled, connect to Finder with lldb:

$ sudo lldb --attach-name Finder

The procedure that we are interested in starts at address 0x10009b788 (in different MacOS versions it might be different):

Let’s set a breakpoint there:

(lldb) breakpoint set --shlib Finder --address 0x10009b788
Breakpoint 1: where = Finder`___lldb_unnamed_symbol2354$$Finder, address = 0x0000000102f83788

Notice that the address 0x0000000102f83788 that the lldb outputs is different from the one we specified with --address due to ASLR. This is the actual slid address where the procedure is loaded and it will be different every time the Finder process is launched. Remember this address because it will be useful to us later.

Now click on /Applications in the Finder and it should freeze due to a breakpoint being hit in lldb. So how do we return true instead of false? Lldb has just the command for returning early from a call - thread return. Let’s use it returning 1 instead of 0:

(lldb) thread return 1
(lldb) continue

(you will have to repeat that for every package in the folder or breakpoint disable)

After the Finder execution continues you should finally be able to browse through any package as if it was a regular directory:

So after all the -[TBrowserContainerController allowsBrowsingPackages] is indeed responsible for displaying packages as regular directories! But how can we return 1 from this procedure without having to type commands in lldb after opening each directory in Finder?

Solutions that don’t work

Unfortunately simply scripting the breakpoint using --command 'thread return 1' and --auto-continue True crashes Finder for some reason and it would probably impede Finder’s performance. Before macOS 12 a fine solution using swizzling worked perfectly:

(lldb) expression --language swift --
Enter expressions, then terminate with an empty line to evaluate:
import Foundation

extension NSObject {
  @objc func swizzled_allowsBrowsingPackages() -> Bool { return true }
}

guard let originalMethod = class_getInstanceMethod(
  NSClassFromString("TBrowserContainerController"),
  Selector("allowsBrowsingPackages")
), let swizzledMethod = class_getInstanceMethod(
  NSObject.self,
  Selector("swizzled_allowsBrowsingPackages")
) else { return }

method_exchangeImplementations(originalMethod, swizzledMethod)

However, it also crashes at method_exchangeImplementations starting with Monterey.

That leaves us with the last reserve tool…

Manipulating opcodes

Underneath, the code that the CPU executes is just a sequence of numbers and those numbers can be manipulated. Let’s use memory read to make sense of the ARM64 assembly and the underlying opcodes. We will need the slid address of the procedure that the lldb gave us when we set the breakpoint (use breakpoint list to see previously set breakpoints):

(lldb) memory read --format instruction 0x0000000102acf788
    0x102acf788: 0x52800000   mov    w0, #0x0
    0x102acf78c: 0xd65f03c0   ret

We can see that mov w0, #0x0 is responsible for setting the output of the allowsBrowsingPackages method. So what would it take to turn #0x0 into #0x1? To figure out we can disassemble a similar function:

$ echo 'int foo() { return 0; }' | clang -c -xc -o /dev/stdout - | objdump -d /dev/stdin

the most important output here is the following sequence of bytes:

0: 00 00 80 52      mov w0, #0
4: c0 03 5f d6      ret

notice the output of objdump matches the disassembly from lldb perfectly, the only difference being that the disassembly from lldb is reversed due to endianness. Now, what happens if return 0 is changed to return 1? We get the following disassembly:

0: 20 00 80 52      mov w0, #1
4: c0 03 5f d6      ret

The difference is that the first byte changed from 0x00 to 0x20.

Now we can change the same byte in the Finder process and see what happens:

(lldb) memory write 0x0000000102acf788 0x20

and if we read the instructions now, the assembly should change as expected:

(lldb) memory read --format instruction 0x0000000102acf788
    0x102acf788: 0x52800020   mov    w0, #0x1
    0x102acf78c: 0xd65f03c0   ret

Now we can finally unpause the Finder using continue.

Yay! We can now browse any package in the Finder without having to Show Package Contents every time (who cares about SIP anyways).

In the future episodes...

While automating this solution I also had a chance to write some lldb python scripts and debug those scripts in PyCharm. In the next articles, I hope to cover those techniques as well.

Did you find any of the techniques used in this article useful? Do you have any questions or suggestions? Please let me know in the comments!