Understanding multithreading by deciphering the cpython interpreter source code

By Chirag Shah

Elevator Pitch

In this talk, I will provide a concise understanding of Threading and Global Interpreter Lock(GIL) in Python.

Description

In the modern era of hybrid cores and processors, there is an in demand need for concurrent and parallel programming paradigms. Python, since its inception has amazing support for single threaded applications. The extensive use of Python in booming fields like Machine Learning has paved the way to constantly improve multi-threaded applications in Python.

I will speak from ground level covering very crucial aspects of Threading and Locks which will provide a better roadmap for community to develop better Python applications. I will provide a concise understanding of Threading and Global Interpreter Lock(GIL) in Python.

Program outcomes:

  • How threading can improve performance, its pros and cons.
  • What works best in which environment between threads and processes.
  • Why GIL matters the most in Python
  • How to leverage the power of open source source code to understand the crux of language.

Contents to be covered:

1. Threading for noobs:

  • Terminologies: Process, threads, multithreading, multiprocessing, types of threads, locks, mutex, CPU and I/O bound processes.
  • Multithreading in Python: Threading module (with example)
  • Comparative analysis of Sequential vs Multithreaded execution in Python (with example)

2. Understanding the global interpreter lock (GIL):

  • What and why of GIL
  • Impact of GIL on CPU and I/O Bound Processes
  • In-depth understanding of GIL using cpython interpreter source code
  • Reference counting
  • Ticks via context switching

3. Infamous concepts:

  • Cooperative vs Preemptive multitasking
  • Parallelism vs Concurrency
  • Thread Safety in Python

4. Removing the GIL:

  • Famous GIL removal patch
  • Guido on GIL, Larry Hastings Gilectomy

5. Questions

Agenda:

  1. 0 - 6 minutes: section 1, Threading for noobs

  2. 6 - 15 minutes: section 2, Understanding GIL

  3. 15 - 25 minutes: section 3, Infamous concepts

  4. 25 - 28 minutes: section 4, Removing the GIL

  5. 28 - 30 minutes: section 5, Questions

Notes

Slides (Initial Draft)

Todos to be added after decision:

  1. Concurrency vs Parallelism
  2. Code Snippets:
    a. Single vs multithreaded
    b. Concurrency and Parallelism
    c. More examples .. (IF POSSIBLE)
  3. Performance analysis. (Without GIL and other envs which are without GIL)
  4. Necessary to be thread safe
  5. Multithreading tips.