How to debug crashing C++ library loaded in python project

I am attempting to figure out why calling a function in a dynamically loaded lib crashes python. I’m doing the following, I have a C++ function in a dynamic library file, which is loaded in python using ctypes. I then call the function from python:

lib = cdll.LoadLibrary(libPath)

# Note: using c_char_p instead of POINTER(c_char) does not yield any difference in result
# Export const char* GetSection(const char* TilesetID, int32_t X0, int32_t Y0, int32_t X1, int32_t Y1, uint8_t*& OutData, uint64_t& OutDataSize)
lib.GetSection.argtypes = [POINTER(c_char), c_int32, c_int32, c_int32, c_int32, POINTER(c_void_p), POINTER(c_uint64)]
lib.GetSection.restype = POINTER(c_char)

output_data = c_void_p()
output_size = c_uint64()
str_data = lib.GetSection(id.encode('ascii'), x0, y0, x1, y1, byref(output_data), byref(output_size))

On MacOS, this works exactly as expected. Unfortunately on Windows 11, it does not. I’m running from a Jupyter notebook and the kernel crashes and restarts immediately after the lib.GetSection call.

I have attached the Visual Studio debugger to the process, and can see that on the C++ side of things, the function is being correctly called, all parameters are correct, and it returns without error. It is at this point that the python kernel crashes, deep in a python call stack that I don’t have symbols for.

How do I even approach debugging this? Does anything look wrong with the way I am calling the function?

Answer

Having a toy C++ function to demonstrate your problem would help. Below is a best guess C++ function with the same signature and the Python code to call it:

test.cpp

#include <cstdint>

#define API __declspec(dllexport)

extern "C" {

API const char* GetSection(const char* TilesetID, int32_t X0, int32_t Y0, int32_t X1,
                           int32_t Y1, uint8_t*& OutData, uint64_t& OutDataSize) {
    OutData = new uint8_t[5] { 1, 2, 3, 4, 5 };
    OutDataSize = 5;
    return "hello";
}

API void Delete(uint8_t* OutData) {
    delete [] OutData;
}

}

test.py

import ctypes as ct

dll = ct.CDLL('./test')
# Note change to 2nd to last argument.
dll.GetSection.argtypes = (ct.c_char_p, ct.c_int32, ct.c_int32, ct.c_int32, ct.c_int32,
                           ct.POINTER(ct.POINTER(ct.c_uint8)), ct.POINTER(ct.c_uint64))
dll.GetSection.restype = ct.c_char_p

def GetSection(tileid, x0, y0, x1, y1):
    output_data = ct.POINTER(ct.c_uint8)()
    output_size = ct.c_uint64()
    str_data = dll.GetSection(tileid, x0, y0, x1, y1,
                              ct.byref(output_data), ct.byref(output_size))
    out_data = output_data[:output_size.value] # create a Python list of the data
    dll.Delete(output_data)  # can delete the data now
    return str_data, out_data

print(GetSection(b'id', 1, 2, 3, 4))

Output:

(b'hello', [1, 2, 3, 4, 5])