Why is this tkinter program freezing?

I’m having an issue with my GUI freezing, and I don’t know why. The run method is not releasing the lock.

Demo program

import time
import threading
import Tkinter as tk
import ttk

LOCK = threading.Lock()

class Video(threading.Thread):
    def __init__(self):
        super(Video, self).__init__()
        self.daemon = True
        self.frame = tk.DoubleVar(root, value=0)
        self.frames = 1000

    def run(self):
        while True:                
            with LOCK:   
                position = self.frame.get()

                if position < self.frames:
                    position += 1
                else:
                    position = 0

                self.frame.set(position)                

            time.sleep(0.01)

root = tk.Tk()
video = Video()
root.minsize(500, 50)

def cb_scale(_):
    with LOCK:
        print('HELLO')

scale = ttk.Scale(
    root, from_=video.frame.get(), to=video.frames, variable=video.frame,
    command=cb_scale)

scale.grid(row=0, column=0, sticky=tk.EW)
root.columnconfigure(0, weight=1)

if __name__ == '__main__':
    video.start()
    root.mainloop()

Problem

Spam-clicking the progress bar freezes the program.

Attempts at debugging

1. I used mttkinter by adding import mttkinter to the import statements and the problem persists. The issue is the lock not being released.

2. I inserted print statements to find out where exactly the program freezes.

Program with print statements:

from __future__ import print_function

import time
import threading
import Tkinter as tk
import ttk

def whichthread(say=''):
    t = threading.current_thread()
    print('{}{}'.format(say, t))

LOCK = threading.Lock()

class Video(threading.Thread):
    def __init__(self):
        super(Video, self).__init__()
        self.daemon = True
        self.frame = tk.DoubleVar(root, value=0)
        self.frames = 1000

    def run(self):
        while True:
            whichthread('run tries to acquire lock in thread: ')
            with LOCK:
                whichthread('run acquired lock in thread: ')

                position = self.frame.get()

                if position < self.frames:
                    position += 1
                else:
                    position = 0

                self.frame.set(position)
            whichthread('run released lock in thread: ')

            time.sleep(0.01)

root = tk.Tk()
video = Video()
root.minsize(500, 50)

def cb_scale(_):
    whichthread('cb_scale tries to acquire lock in thread: ')
    with LOCK:
        whichthread('cb_scale acquired lock in thread: ')
        print('HELLO')
    whichthread('cb_scale released lock in thread: ')

scale = ttk.Scale(
    root, from_=video.frame.get(), to=video.frames, variable=video.frame,
    command=cb_scale)

scale.grid(row=0, column=0, sticky=tk.EW)
root.columnconfigure(0, weight=1)

if __name__ == '__main__':
    video.start()
    root.mainloop()

This produces the following output right before the program freezes:

...
run tries to acquire lock in thread: <Video(Thread-1, started daemon 140308329449216)>
run acquired lock in thread: <Video(Thread-1, started daemon 140308329449216)>
cb_scale tries to acquire lock in thread: <_MainThread(MainThread, started 140308415592256)>

This shows that for some reason, the run method does not release the lock.

3. I tried to comment out lines in order to narrow the problem down.

Removing any of the two with LOCK statements fixes the issue. Unfortunately, in my real program the run and cb_scale function do something meaningful that requires locking.

Commenting out both the calls to get and set in run fixes the issue.

… and this is where I am stuck! :)

EDIT

Thanks to Mike – SMT I was able to track the problem down further.

Using

class DummyDoubleVar(object):
    def get(self):
        return 500

    def set(self, _):
        pass

and

self.frame = DummyDoubleVar()

in Video.__init__ prevents the program from freezing.

(Remember that the original program reliably freezes even with mttkinter. I am stumped what’s going on here!)

Answer

In this post, I will show the solution to the problem and what led me to discover it. It involves going over CPython _tkinter.c code, so if that’s not something you are up for, you can just skip to the TL;DR section below. Now, let’s dive down the rabbit hole.

Lead-Up

The problem occurs only when moving the sliding bar manually. The MainThread and the Video-thread are then in dead-lock with each other over the LOCK, which I will call the user-lock. Now, the run method never releases the user-lock after is has acquired it, which implies that is hanging because it is waiting for another lock or some operation to complete which cannot. Now, looking at the log output of your verbose example, it becomes clear that the program does not hang consistently: It takes a few tries.

By adding more prints to the run method, you may discover that the problem is not consistently caused by either get or set. When the problem is caused, get may have already finished, or it may not have. This implies that the problem is not caused by get or set specifically, rather by some more generic mechanism.

Variable.set and Variable.get

For this section, I considered only Python 2.7 code, even though the problem is also present in Python 3.6. From the Variable-class in the Tkinter.py file of CPython 2.7:

def set(self, value):
    """Set the variable to VALUE."""
    return self._tk.globalsetvar(self._name, value)
def get(self):
    """Return value of variable."""
    return self._tk.globalgetvar(self._name)

The self._tk attribute is the Tk-object defined in the C-code of Tkinter, and for the code of globalgetvar we must jump back to _tkinter.c:

static PyObject *
Tkapp_GlobalGetVar(PyObject *self, PyObject *args)
{
    return var_invoke(GetVar, self, args, TCL_LEAVE_ERR_MSG | TCL_GLOBAL_ONLY);
}

Jumping to var_invoke:

static PyObject*
var_invoke(EventFunc func, PyObject *selfptr, PyObject *args, int flags)
{
    #ifdef WITH_THREAD
      // Between these brackets, Tkinter marshalls the call to the mainloop
    #endif
    return func(selfptr, args, flags);
}

Just to make sure: I compiled Python with thread support and the problem persists. The call is marshalled to the main thread, which I checked with a simple printf in that location. Now, is this done correctly? The function var_invoke will wait until the MainThread has resumed and has executed the requested call. What is the MainThread doing at this point? Well, it is executing its queue of events, in the sequence it got them. What sequence did it get them in? That depends on the timing. This is what causes the problem: In some cases, Tkinter will execute the call to the callback right before a get or set, but while the lock is held.

Regardless of whether mtTkinter is imported (as long as Python is compiled WITH_THREAD support), the call of get and set is marshalled off to the mainloop, but that mainloop may just be trying at that moment to call the callback, which also needs the lock… This is what causes the deadlock and your problem. So basically mtTkinter and plain Tkinter offer the same behaviour, though for mtTkinter this behaviour is caused in the Python code and for plain Tkinter it happes in C-code.

TL;DR; In short

The problem is caused by only the user-lock. Neither GIL nor the Tcl-interpreter lock is involved. The problem is caused by the get and set methods marshalling their actual invocation off to the MainThread and then waiting for completion of the call by this MainThread, while the MainThread tries to do the events in order and execute the callback first.

Is this intended behaviour? Maybe, I’m not sure. I sure can see that with all the ENTER_TCL and LEAVE_TCL macro’s in the _tkinter.c file, a better solution might be possible than the current one. For now though, there is no real work-around for this problem (bug? feature?) that I can see, apart from using Tk.after(0, Variable.set), so that the Video-thread does not hold the lock while the MainThread might need it. My suggestion would be removing the DoubleVar.get and set invocations from the code where the lock is held. After all, if your program does something meaningful, it might not need to hold the lock while it sets the DoubleVar. Or, if that is not an option, you will have to find some other means of synchronizing the value, like a subclass of the DoubleVar. Which suits your needs best highly depends on your actual application.