Gtk.Spinner in Python GTK while importing large library

I have a GTK application in C that will spawn a Python GTK process to embed a matplotlib figure into a window in the C process using GtkSocket/GtkPlug (uses XEmbed Protocol). The problem I am having is that the import of the matplotlib library takes about 2.5 seconds and during that time the socket widget is simply transparent. I would like to place a Gtk.Spinner in the plug (so the Python side) before the matplotlib import and have the spinner animate asynchronously during the process of importing the matplotlib library. The problem is that in order for the widget to be placed in the plug, and subsequently, for the Gtk.Spinner to animate, it requires iterations of the GTK main loop. I have approached this from a ton of different angles:

(1) Using a thread. The first attempt was trying to run Gtk.main_iteration() via the thread, however, GTK can only be run on the main thread and this does not work. It stalls the program.

(2) Then I tried to use GObject.idle_add from the thread, where the main loop iterations would run from the idle function (apparently the function called via idle is done on the main thread?), but this didn’t work either.

(3) Then I tried to import the modules on the thread, while the main thread runs the Gtk.main_iteration()’s to allow the spinner to spin while the imports are taking place. The idea was once the imports are complete, a boolean flag would change to trigger a break from the loop of main iterations. In this case the spinner appears and spins but the plot never shows up. I get an X Server error:

Gdk-WARNING **: master: Fatal IO error 104 (Connection reset by peer) on X server :0.

(4) In lieu of threading, I tried to use GObject.timeout_add to call a function regularly that would perform the Gtk.main_iteration()’s, but doing that results in the original behavior where the socket/plug is transparent until the plot shows up (i.e. no spinner appears nor spins).

I have run out of ideas and now I am coming here hoping for an assist. They key idea is to get the Gtk.Spinner spinning while the Python script is loading the matplotlib library, and once that is done, replace the spinner widget with the figure (while all of this is taking place in a GtkSocket/Plug). I have not created a minimal reproducible example for this since it would be rather complex given the circumstances, but if anyone that is willing to help requests one I could come up with it. However, the relevant code section is below (with previous attempts commented out):

import sys
import gi
import time
import threading
gi.require_version('Gtk', '3.0')
from gi.repository import Gtk, Gdk, GObject
from gi.repository import Pango as pango

if sys.platform != "win32":

    GObject.threads_init()
    Gdk.threads_init()

    # Open socket ID file, read Socket ID into variable, close file
    socketFile = open('resources/com/gtkSocket', 'r+')
    gtkSock = socketFile.read()
    print("The ID of the sockets window in Python is: ", int(gtkSock))
    socketFile.close()
    # Create plug, create GTK box, add box to plug, add spinner to box
    spin_plug = Gtk.Plug.new(int(gtkSock))
    socketbox = Gtk.Box()
    spin_plug.add(socketbox)

    spinner = Gtk.Spinner()
    socketbox.pack_start(spinner, expand=True, fill=True, padding=False)
    spinner.start()

    finished = False

    def thread_run():
        time.sleep(4)
        '''
        # Loop for four seconds checking if gtk events are pending, and if so, main loop iterate
        t_end = time.time() + 4
        while time.time() < t_end:
            if (Gtk.events_pending()):
                Gtk.main_iteration()
                print("Events Pending...")
        '''

        '''
        import argparse
        import collections
        import csv
        import matplotlib
        import matplotlib.pyplot as plt
        import matplotlib.patches as patches
        import matplotlib.lines as mlines

        from collections import defaultdict
        from enum import Enum
        from matplotlib.backend_bases import MouseEvent
        from matplotlib.pyplot import draw
        from matplotlib.widgets  import SpanSelector
        from matplotlib.backends.backend_gtk3agg import FigureCanvasGTK3Agg as FC
        from matplotlib.backends.backend_gtk3 import NavigationToolbar2GTK3
        '''

        # You cannot run GTK Code on a separate thread from the one running the main loop
        # Idle add allows scheduling code to be executed on the main thread
        GObject.idle_add(cleanup)

    def cleanup():
        # Note: Trying to add the Gtk Main Iterations to the idle add function did not work...
        print("Closing Spinner Thread...")
        spinner.stop()
        finished = True
        thread.join()

    # start a separate thread and immediately return to main loop
    #thread = threading.Thread(target=thread_run)
    #thread.start()
    spin_plug.show_all()

    
    def spin():
        busy_wait = 0
        while (Gtk.events_pending() or busy_wait < 10):
            Gtk.main_iteration()
            if (not Gtk.events_pending()):
                busy_wait = busy_wait + 1
        print("Spin Call complete.")
        return True

    GObject.timeout_add(50, spin)
    
    '''
    # We cannot simply run an infinite Gtk.main() loop, so iterate until the plug has been filled
    busy_wait = 0
    while (Gtk.events_pending() or busy_wait < 10):
        if (finished):
            break
        print("Busy Wait: %d" % busy_wait)
        Gtk.main_iteration()
        if (not Gtk.events_pending()):
            busy_wait = busy_wait + 1
    print("Gtk Main Loop iterations complete.")
    '''

Any pointers or ideas would be greatly appreciated.

Answer

The solution was performing the imports on a thread while allowing the main thread to do the main loop iterations. Simply doing “import ” did not work. Some previous Stack Overflow posts that were useful are here:

Python thread for pre-importing modules

import a module from a thread does not work

Import python modules in the background in REPL

The solution looks like this:

GObject.threads_init()
Gdk.threads_init()

# Open socket ID file, read Socket ID into variable, close file
socketFile = open('resources/com/gtkSocket', 'r+')
gtkSock = socketFile.read()
print("The ID of the sockets window in Python is: ", int(gtkSock))
socketFile.close()
# Create plug, create GTK box, add box to plug, add figure to box
spin_plug = Gtk.Plug.new(int(gtkSock))
socketbox = Gtk.Box()
spin_plug.add(socketbox)
# Create a spinner, pack it, and start it spinning
spinner = Gtk.Spinner()
socketbox.pack_start(spinner, expand=True, fill=True, padding=False)
spinner.start()
spinner.show()
# Flag to break from the Gtk.events_pending() loop
finished = False

# This will load modules on a thread. A simple "import module" does not work
def do_import(module_name):
    thismodule = sys.modules[__name__]
    module = importlib.import_module(module_name)
    setattr(thismodule, module_name, module)
    print(module_name, 'imported')
    # Use the last module being imported so we know when to break from the Gtk.events_pending()
    if (module_name == "matplotlib.pyplot"):
        global finished
        finished = True

spin_plug.show_all()

modules_to_load = ['argparse', 'collections', 'csv', 'matplotlib', 'matplotlib.pyplot']

# Loop through and create a thread for each module to import from the list
for module_name in modules_to_load:
    thread = threading.Thread(target=do_import, args=(module_name,))
    thread.start()

# We cannot simply run an infinite Gtk.main() loop, so iterate until the plug has been filled
# Busy wait continues to allow the spinner to spin until the computer loads the modules. Since
# each computer will have a different loading speed, a busy wait of 300 should cover slower
# machines. We can break out of the loop early once the last module is loaded.
busy_wait = 0
while (Gtk.events_pending() or busy_wait < 300):
    #print("Busy Wait: %d" % busy_wait)
    #print ("finished: %d" % finished)
    if (finished):
        break
    Gtk.main_iteration()
    if (not Gtk.events_pending()):
        busy_wait = busy_wait + 1