Playback speed
×
Share post
Share post at current time
0:00
/
0:00
Preview

CPython Memory Management Internals

A high-level but detailed explanation of how CPython implements memory management

Last weekend, we had a live session on the internals of memory management in CPython. Unfortunately, the recording of that session got corrupted, so I recorded a video on it.

CPython uses a combination of reference counting and a generational garbage collector (GC) to manage the memory of the Python programs. Reference counting is a fairly simple technique and I’ve written a very detailed article on it. So in this video, I give a basic overview of reference counting and its implementation, but the majority of the focus is on the garbage collector and its implementation.

The garbage collector in CPython is used for cleaning up memory of objects which have cyclic references and thus cannot be cleaned up reference counting.

The video covers the following topics:

  • How CPython manages memory

  • Reference counting & its implementation details in CPython at the interpreter level

  • In what situations reference counting does not work?

  • GC implementation details:

    • What is a generational GC?

    • How and when the GC is scheduled to run?

    • What does the GC do in CPython?

    • The cyclic reference detection algorithm in the CPython GC

  • Changes in reference counting and GC implementation due to removal of the global interpreter lock (GIL)

Upcoming Article

Although, this video is very detailed and tells you everything you need to know about theGC in CPython, it does not go into the source code (I do show some important struct definitions).

I’ve a very detailed article coming up in a few days on the GC internals in CPython, which goes into all the gory implementation details and walks you through the code to show how everything happens. So watch out for that as well. Perhaps having watched this video will help you get a better perspective when reading that article.

Upcoming Live Session on Building a Basic Remote Profiler for Python

I also have another live session coming up on 26th May where we will actually apply our knowledge of the CPython runtime internals to learn how tools like remote profilers or debuggers work. This will be very technical and will have lots of stuff to learn. I’m very excited about. You can checkout the details in the below post.

Support Confessions of a Code Addict

If you enjoy reading my articles and find value in them, please consider supporting Confessions of a Code Addict by becoming a paid subscriber. It takes several days of work to produce every piece that I write and your support will help keep it going. Although, I try to keep most of my articles open, I offer the following additional perks to the paid supporters:

  • Free access to the monthly live sessions

  • Access to past recordings of all the live sessions

  • Option to schedule a 30 minute meeting with me every month to discuss anything you wish

The monthly plan is at Indian Rupees (INR) 550 ($6) and annual plan is at INR 4999 ($60).

Alternative Ways to Support

Many people report that their transactions fail on Substack, so I also have a couple of alternate options.

You can support me on buymeacoffee.com. You can either buy me coffees or become a member and I will upgrade you to a paid subscription here for the equivalent period.

Support me on buymeacoffee.com

Alternatively, you can also support me through GitHub sponsorship:

Become a GitHub Sponsor

This post is for paid subscribers

Confessions of a Code Addict
Confessions of a Code Addict Podcast
Deep dives into varied topics on Computer Science including compilers, programming languages, database internals, AI and more. Subscribe for insights and advance your engineering skills!