GIL? Where we are going, we don't need the GIL!

The free-threaded build of CPython 3.13 enables us to run Python without the GIL. In this talk, I will explain what the GIL does, how it hurts the Data Science & Analytics workloads in Python, previous efforts to remove the GIL, and the current GIL-ectomy. Finally, I’ll close with ecosystem changes.

PEP 703 - Making the GIL optional in CPython was accepted in 2023 and starting CPython 3.13, end users are able to test a free-threaded build of CPython that is unencumbered by the GIL. I’ll outline the key changes introduced by the PEP to the CPython interpreter i.e. changes in the following areas of the CPython interpreter

• Reference counting
• Garbage collection
• Memory management
• Container thread-safety
• Locking and Atomic APIs

Before diving into these, I’ll briefly outline previous efforts to remove the GIL and highlight why they weren’t successful i.e. the fact that they always caused a significant reduction in single-threaded performance. This includes efforts by Greg Stein in Python 1.4, Adam Olsen in Python 2.6 and Larry Hastings in 2016. I’ll mention some of the parallel efforts ongoing within the CPython ecosystem that are improving single-threaded performance, because of which the overall reduction in performance because of PEP 703 aren’t as significant as previous efforts. I’ll outline the overall plan of adoption agreed upon by the Python community of the free-threaded CPython build i.e.

1. Experimental phase
2. Supported-but-not-default phase
3. Default phase

where eventually the free-threaded build of CPython will be the default for the Python community.

Finally, in my call to action, I will highlight efforts within the broader Python ecosystem, especially within the Scientific Python ecosystem, to adopt the free-threaded build of CPython. This provides an opportunity for members of the audience who are interested in contributing to the FOSS Python ecosystem to drive the adoption.

As a Learning & Development consultant at Tiger Analytics, I have already created and delivered this talk internally to a large ML Engineering team. I’m happy to share the slides and maybe a recording of the talk to the review team if that helps.