threading.Thread fails to pickle in python > 3.13.0a5

[mmckerns]

Starting in python 3.13.0a5, a threading.Thread instance is not serializable. This is due to the thread attribute pointing to an unserializable _thread._ThreadHandle instance, where in earlier versions the _thread instance pointed to None. Deleting the _ThreadHandle enables the Thread to be serializable again. Presumably, there's a good way to obtain an appropriate thread handle from the thread or using the threading module.

Python 3.13.0a5 (main, Mar 16 2024, 18:36:37) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import threading
>>> t = threading.Thread()
>>> t.start()
>>> import dill
>>> dill.dumps(t)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    dill.dumps(t)
    ~~~~~~~~~~^^^
  File "/Users/mmckerns/lib/python3.13/site-packages/dill/_dill.py", line 280, in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mmckerns/lib/python3.13/site-packages/dill/_dill.py", line 252, in dump
    Pickler(file, protocol, **_kwds).dump(obj)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/Users/mmckerns/lib/python3.13/site-packages/dill/_dill.py", line 420, in dump
    StockPickler.dump(self, obj)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pickle.py", line 483, in dump
    self.save(obj)
    ~~~~~~~~~^^^^^
  File "/Users/mmckerns/lib/python3.13/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pickle.py", line 599, in save
    self.save_reduce(obj=obj, *rv)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pickle.py", line 713, in save_reduce
    save(state)
    ~~~~^^^^^^^
  File "/Users/mmckerns/lib/python3.13/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pickle.py", line 556, in save
    f(self, obj)  # Call unbound method with explicit self
    ~^^^^^^^^^^^
  File "/Users/mmckerns/lib/python3.13/site-packages/dill/_dill.py", line 1217, in save_module_dict
    StockPickler.save_dict(pickler, obj)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pickle.py", line 968, in save_dict
    self._batch_setitems(obj.items())
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pickle.py", line 992, in _batch_setitems
    save(v)
    ~~~~^^^
  File "/Users/mmckerns/lib/python3.13/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pickle.py", line 574, in save
    rv = reduce(self.proto)
         ~~~~~~^^^^^^^^^^^^
TypeError: cannot pickle '_thread._ThreadHandle' object
>>> t.__dict__
{'_name': 'Thread-1', '_daemonic': False, '_ident': 123145514024960, '_native_id': 9483263, '_tstate_lock': <unlocked _thread.lock object at 0x1107a61c0>, '_handle': <_thread._ThreadHandle object: ident=123145514024960>, '_started': <threading.Event at 0x10fdfdf10: set>, '_is_stopped': False, '_initialized': True, '_stderr': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>, '_invoke_excepthook': <function _make_invoke_excepthook.<locals>.invoke_excepthook at 0x110772ca0>}
>>> t._handle = None
>>> pt = dill.dumps(t)
>>> _t = dill.loads(pt)
>>>

[mmckerns]

while it's not directly related, might be an opportunity to deal with objects in #334

[musicinmybrain]

The current failure on Python 3.13.0b1 looks like this:

$ gh repo clone uqfoundation/dill
$ cd dill
$ python3.13 -V
Python 3.13.0b1
$ tox -e py313
.pkg: _optional_hooks> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_wheel> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: prepare_metadata_for_build_wheel> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: build_sdist> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
py313: install_package> python -I -m pip install --force-reinstall --no-deps /home/ben/src/forks/dill/.tox/.tmp/package/2/dill-0.3.9.dev0.tar.gz
py313: commands[0]> .tox/py313/bin/python -m pip install .
Processing /home/ben/src/forks/dill
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: dill
  Building wheel for dill (pyproject.toml) ... done
  Created wheel for dill: filename=dill-0.3.9.dev0-py3-none-any.whl size=116162 sha256=86a3739b5a701f9aefd7840a31f695eefa0e8408a55028bfe497423a5d87d061
  Stored in directory: /tmp/pip-ephem-wheel-cache-sukup0uc/wheels/4b/69/4e/74fd16e96eb0e05af8c484f673696525e999353b5fec3ad50d
Successfully built dill
Installing collected packages: dill
  Attempting uninstall: dill
    Found existing installation: dill 0.3.9.dev0
    Uninstalling dill-0.3.9.dev0:
      Successfully uninstalled dill-0.3.9.dev0
Successfully installed dill-0.3.9.dev0
py313: commands[1]> .tox/py313/bin/python dill/tests/__main__.py
.<function <lambda> at 0x7fecfbe8be20>
<function <lambda> at 0x7f937fe8be20>
<function <lambda> at 0x7fed0408be20>
<function <lambda> at 0x7fbd8328be20>
<function <lambda> at 0x7f2a7ea8be20>
Traceback (most recent call last):
  File "/home/ben/src/forks/dill/dill/tests/test_detect.py", line 154, in <module>
    test_bad_things()
    ~~~~~~~~~~~~~~~^^
  File "/home/ben/src/forks/dill/dill/tests/test_detect.py", line 32, in test_bad_things
    assert len(s) is len(a) # TypeError (and possibly PicklingError)
           ^^^^^^^^^^^^^^^^
AssertionError
...F.......................
py313: exit 1 (1.79 seconds) /home/ben/src/forks/dill> .tox/py313/bin/python dill/tests/__main__.py pid=182079
  py313: FAIL code 1 (6.48=setup[2.72]+cmd[1.97,1.79] seconds)
  evaluation failed :( (6.53 seconds)

[musicinmybrain]

s is:

{('TypeError', "cannot pickle 'frame' object"), ('TypeError', "cannot pickle 'FrameLocalsProxy' object")}

and a is:

{'TypeError': "cannot pickle 'FrameLocalsProxy' object"}

which appears related in some way to PEP 667.

[mmckerns]

If I remember correctly -- from more than a decade ago -- the primary reason that a frame object couldn't be pickled is that the GIL is imbedded in the frame object and could only be written to in C. In python 3.13, there have been a lot of recent changes to support removal of the GIL. I expect it's related to that.

[musicinmybrain]

Fedora 41, which is already in beta freeze, will ship with Python 3.13. Final freeze is scheduled for 2024-10-15. There are quite a few packages that will be impacted, and may have to be retired from Fedora, if we can’t ship a working python-dill by that time.

I’m not sure how to fix this, or I would have already proposed a PR, but please let us know if there is anything we can do to help.

[mmckerns]

I know. I'm woking on it, and plan to cut a release this coming week. Fundamentally, there's only one or two fixes that are left to cover the remaining breaking changes made in python 3.13.

[musicinmybrain]

Thanks! That’s exciting news.

[mmckerns]

#668 is addressed by #677, thus all tests pass. However, I'm not closing this issue as the issue with a threading.Thread remains.

[mmckerns]

closed by #679