This post is something new I’m trying where I will share a few updates about upcoming things at Confessions of a Code Addict, and some interesting resources that might be of interest to you.

Updates from Confessions of a Code Addict

I was completely wiped out mentally after writing the article on GPU computing and it has taken a while for me to get back on the saddle. I’m currently working on couple of new ideas for articles.

• There were few topics related to GPUs which I had to leave out in the last article because of space and time constraints. Those topics were not necessary to understand the fundamentals of GPU computing, but it’s good to know about them. I’m tentatively calling the article as “GPU Computing: The Missing Parts”. I plan to discuss topics such as:

  • Advanced grid and thread block layouts

  • Thread block clusters (newly introduced feature in Nvidia H100)

  • Control divergence: How warps execute in the presence of if/else conditions

  • Independent thread scheduling

  • Unified virtual memory between CPUs and GPUs

• Another exciting topic I am working on is concerned with CPython internals. It’s too soon to disclose the details but many of you have enjoyed the CPython internal articles and this one is going to be very hardcore.

I hope you are as excited about these as I am! Next week is quite busy for me, but I’m working as hard as I can to get these out for you as soon as possible. Stay tuned!

Resources for the Week

📰 Articles

• GPU Puzzles: Learn GPU Programming by solving these interactive GPU puzzles. This is the perfect next progression after reading the GPU computing article: https://github.com/srush/GPU-Puzzles

• Do Language Models really understand Language: Whether LLMs understand language or not is a hotly debated topic. In this article digs deep into the research in this area and breaks it down for you. Highly recommended read:

  Artificial Intelligence Made Simple

  Do Language Models really understand Language[Breakdowns]

  Hey, it’s Devansh 👋👋 In my series Breakdowns, I go through complicated literature on Machine Learning to extract the most valuable insights. Expect concise, jargon-free, but still useful analysis aimed at helping you understand the intricacies of Cutting-Edge AI Research and the applications of Deep Learning at the highest level…

  Read more

  2 years ago · 69 likes · 7 comments · Devansh

Read more

In this week’s digest:

• Unbounded memory usage by TCP for receive buffers, and how we fixed it

• Cgroups - deep dive into resource management in kubernetes

• A visual guide to how memory allocation works

• Squeezing a little more performance out of bytecode interpreters

• How GPU computing works

• Making PostgreSQL multi-threaded

• Train you own minimal GPT language model in Rust

• Effective Rust

• The Linux kernel module programming guide

• Introduction to Scala-3

• Distributed systems for fun and profit

📰 Articles

Unbounded memory usage by TCP for receive buffers, and how we fixed it

Cloudflare engages in intensive engineering work in the fields of networking and systems performance. Their blogs are highly recommended reading for all engineers, regardless of whether they directly work in those areas. By reading their blogs, you can gain valuable insights that can enhance your own work. In a recent article, Cloudflare discusses their process of debugging an issue in the Linux kernel's network stack. They discovered that the kernel was allocating excessive memory for certain TCP sessions, resulting in a significant increase in system-wide memory usage and subsequent performance degradation. This article serves as an invaluable lesson in debugging techniques and system performance analysis. It is a must-read for all engineers!

Cgroups - Deep Dive into Resource Management in Kubernetes

Everyone is developing microservices these days and deploying them with the help of Kubernetes. However, how does Kubenetes work? In this article Martin Heinz carefully explains what Cgroups are and how Kubernetes uses them to do resource allocation and management. It shows how are the pod manifests translated into cgroup files. Martin also talks about how the upcoming memory QoS feature is being implemented with the help of cgroups and how you would implement a container aware OOM-killer with cgroups.

A Visual Guide to How Memory Allocation Works

Memory is one of the most fundamental resource that every program works with. Even if you program in a language with automatic memory management, it’s still important to understand how memory allocators work. At the end of the day, every piece of software is using the memory allocator provided by the operating system. By understanding how these allocators work, you are in a better position to debug performance issues. And if it comes down to replacing the system provided memory allocator with a 3rd party allocator, you will be able to make an informed decision because you can understand their pros and cons. In this article, Sam in his usual visual story telling style explains the working of a simple memory allocator, and gives you enough details that you can go and write your own allocator.

Squeezing a Little More Performance Out of Bytecode Interpreters

Do you dabble in programming languages? I know many programmers who like to play around implementing their own interpreted languages. It’s a fun exercise, now only you gain a deeper understanding of how programming languages work, but you also learn weird tricks to improve the performance of your interpreter (or compiler). Stefan Marr is a well known researcher working in this area and in this short blog post he gives some tips to squeeze some extra performance in your bytecode interpreter. Do give it a read!

How GPU Computing Works

GPUs are at the forefront of all AI applications, they are powering the AI revolution. But, how do they work? A typical undergraduate computer science program covers in deep how CPUs work, but GPUs hardly get any coverage. There is a vast difference between the computing models of CPUs and GPUs. In this 40 minute talk Nvidia engineer Stephen Jones explains how exactly computation in GPU works.

Making PostgreSQL Multi-threaded

This is a discussion thread between the PostgreSQL developers on making PostgreSQL multi-threaded. Even though I typically share articles or books with you here, but this discussion is fascinating and very educational. The developers involved in the discussion are highly technical and you can pick up a ton of knowledge just by reading the nuances they point out in making a complex system such as PostgreSQL multi-threaded.

Train your own minimal GPT language model in Rust

Do you want to understand how the GPT models work? And do you want to train your own small GPT? In this article the author shows how he trained a minimal GPT model in Rust and explains the process involved, check it out!

Subscribe now

📚 Books and Courses

Effective Rust

In the style of the “Effective <your favorite programming language>” book series, such as Effective Java and Effective C++, we now have Effective Rust. These books are well known in the programming circles for giving best practices and techniques for using various language features in order to write idiomatic code. Check it out!

The Linux Kernel Module Programming Guide

Do you want to become a Linux kernel hacker? This book takes explains how the Linux kernel modules work and how to implement them. Linux kernel modules are pluggable extensions to the kernel which allow us to implement custom behavior and functionality at the kernel level. For example, you can implement device drivers as kernel modules. Check out this book and learn to implement your own kernel modules.

Introduction to Scala-3

This is a course on the basics of the Scala programming language. Scala is a popular functional programming language which runs on the JVM platform. It has been adopted by companies such as Twitter and LinkedIn, in order to meet the scale of their growing user bases. It comes with many modern features which have been widely appreciated by the programming community and some of these features are now slowly making their way into Java as well. This course is still under development, but check it out and keep an eye on it!

Distributed Systems for Fun and Profit

In the microservice world, the use of distributed systems such as Kafka, Cassandra, Redis etc. has become very common. These systems hide massive amount of complexity behind their easy to use APIs. In order to understand the trade-offs involved when configuring these systems and when debugging issues, we need to have a thorough understanding of how such systems work. The field of distributed systems is very wide and deep, and it’s easy to feel lost when trying to explore it. However, if you ask me for a resource to get started, I will recommend this book. It covers some of the most important topics of distributed systems within 74 pages.

Wrapping Up

I hope you enjoyed the articles and other resources from this week. Let me know in the comments if you enjoyed something in particular, or if something bugged you. Also, if you have anything interesting that you would like to share with me, then do so in the comments, I would love to read it.

If you'd like to stay connected with me through social media, you can find me on Twitter, LinkedIn, and Substack Notes. Given the declining quality of Twitter services, I warmly invite you to join me on Notes. It's a vibrant community that fosters real-time and amicable conversations among readers and writers.

Join me on Notes!

Share

In this week’s digest:

• CLI tools for interacting with ChatGPT APIs

• Debugging C and Python code using GDB

• Rust to Assembly: Understanding inner workings of Rust

• Learn 64-bit x86 assembly programming by writing a GUI from scratch

• AI does not help programmers

• Best Practices for Using Function Call Feature in OpenAI’s ChatGPT API

• An introductory course on deep learning using PyTorch

• Material from a workshop on vectorization from Cornell University

• Essentials of Compilation: A book teaching you writing a compiler for your own programming language

  Thanks for reading Confessions of a Code Addict! Subscribe for free to receive new posts and support my work.

📰 Articles

llm, ttok and strip-tags—CLI tools for working with ChatGPT and other LLMs

If you are a command line aficionado and appreciate the beauty of Unix pipes, then you will love this. Simon Willison has created a set of CLI tools for interacting with ChatGPT (via its APIs). For example, you can use these tools to get a summary of NYTimes' home page with this invocation:

curl -s https://www.nytimes.com/ \
  | strip-tags .story-wrapper \
  | ttok -t 4000 \
  | llm --system 'summary bullet points'

He has been adding more similar tools. I will leave the rest of the details to his article. Check it out here.

Debugging a Mixed Python and C Language Stack

Many of us are building complex data science stacks comprising tools and frameworks written in multiple languages, such as C, Python, and CUDA. Every now and then, we are bound to run into bugs, but how do we go about debugging such a mixed language stack? This article shows how to do that. It demonstrates how to debug C and Python code from GDB and provides some useful debugging techniques to find out the source of a deadlock among multiple threads of execution. Check it out here.

Learn x86-64 assembly by writing a GUI from scratch

Do you want to learn x86 assembly programming but find most of the resources on it dry and boring? Then this should excite you. In this long tutorial, the author takes you through the basics of assembly programming for x86 hardware by building an x11-based GUI from scratch. Check it out here.

Learn vim While Playing a Game

Do the key bindings of Vim seem unusual to you, and you can’t quite figure out how to quit Vim (just kidding)? Then check out this online game where you need to use Vim key bindings to cross the levels. As you progress through the levels, you discover and master more and more Vim shortcuts. Find it here.

Rust to Assembly: Understanding the Inner Workings of Rust

Some of the best C programmers I have known have told me that if you want to write better C code, you need to know what assembly code is being generated by the compiler for it. It helps you write more efficient code at the C level because you can guess how the compiler might translate it, and you can guide the compiler to generate more optimal code. Rust is also a systems programming language, and by understanding how the assembly code is being generated for a piece of Rust code, you can learn to write better Rust code. This is a series of short posts that show and explain the assembly code for different Rust features. Check it out here.

AI Does Not Help Programmers

There is an overwhelming set of people claiming that AI is here to replace programmers and a lot of is being written on this. However, as I have argued in one of my articles, AI is really going to empower programmers to work on another level of abstraction for solving bigger problems. Bertrand Meyer voices a similar opinion in his article in the communications of the ACM. Check out the full article here.

Best Practices for Using Function Call Feature in OpenAI’s ChatGPT API

And if you missed my latest article from last week, check it out. In this article I provide some guidelines and best-practices to get the best output out of the ChatGPT APIs when using the function call feature. Find the article linked below.

Subscribe now

📚 Books and Courses

Deep Learning Fundamentals

This is a hands-on course on deep learning by Sebastian Raschka, who is a researcher working in this field and has also published multiple books on this topic. The course uses PyTorch, so it’s also a good introduction to PyTorch. Find it here.

Introduction to Vectorization

This is material from a workshop on vectorization at Cornell University. It is not super detailed like a book or a course; however, it does provide a good introduction to the topic. Find it here

Essentials of Compilation (in Python and Racket)

I do have some readers who are deeply interested in compilers and programming languages. This book is a treat for them. Essentials of Compilation is an open-source book that covers how to compile a programming language down to assembly code. It does this by taking the readers through the steps of building a new programming language and a compiler for the language along with it. By the end of the book, you build a full-fledged compiler for your new language. The book comes in two flavors: one flavor uses the Racket programming language, while the other uses Python. Find the latest release of the book here.

Wrapping Up

I hope you enjoyed the articles and other resources from this week. Let me know in the comments if you enjoyed something in particular, or if something bugged you. Also, if you have anything interesting that you would like to share with me, then do so in the comments, I would love to read it. As always, thanks for reading Confessions of a Code Addict and do share it with your friends and colleagues.

Share

In this week's digest:

• Expected Performance of a Bloom Filter

• Implementing Cosine in C from Scratch (without using any math lib)

• Build a CI/CD Pipeline for a Serverless Application

• Designing Modern UIs with Rust

• JDK 21's Approach to Beginner-Friendly Programming in Java

• How Understanding CPUs can help Speed Up Numba and NumPy Code

• Introduction to the Tiny game engine

• Understanding DeepMind's AlphaDev Breakthrough in Optimizing Sorting Algorithms

• How to Use the Function Call Feature of OpenAI’s ChatGPT APIs to Write Your Own Plugins

• Books and Courses at the end of the article

• Read till the end of the article to find a bonus tip to improve your productivity

📰 Articles

Expected Performance of a Bloom Filter

Last week, I shared a link to an article on optimizing Bloom filters. This week, let's take a look at how to analyze the performance of a Bloom filter. This article is written by Daniel Lemire, a professor at Université du Québec who is well-known for his work on blazing-fast data structures and libraries such as simdjson. In this article, Daniel explains how a Bloom filter works and how to evaluate the expected performance for the hit and miss cases when doing a lookup in a typical Bloom filter. I hope you enjoy this article.

Implementing cosine in C from scratch

This is a high-quality article where the author takes you through the process of implementing the cosine function in C from scratch. “From scratch” means without using the standard math library. This is a fun read where you not only learn how cosine function is actually implemented in software, but also various ways to optimize it. Check it out here

Build a CI/CD Pipeline for a Serverless Application

If you are interested in learning how to automate the deployment of your application, then you are in luck. This well-written tutorial shows how to setup a complete CI/CD pipeline in Jenkins for deploying an AWS Lambda-based application. Check it out here.

Makepad: Designing modern UIs with Rust

Makepad is a toolkit for building native and web UIs in Rust. If you are a Rust fan and interested in building full-fledged UI-based applications with it, then checkout this tutorial.

JDK 21’s Approach to Beginner-Friendly Java Programming

Lately, Java has been gaining new features at such a rapid rate that sometimes it’s hard to keep track. One of the latest features coming in the JDK 21 release is the support for implicit classes and enhanced main methods. These changes are designed to make Java more beginner-friendly. This basically means that the canonical “hello, world” program in Java can just be written as:

void main() {
    System.out.println("Hello, World!");
}

Check out the article for full details on this.

Understanding CPUs can help speed up Numba and NumPy code

If you do data science work, then you will appreciate this. Sometimes, due to the scale of the data, the code we write can be quite slow and take a long time to run. Data science is an area where we want to iterate through experiments quickly in order to find the right solution to the problem, and slow code does not help with that. Not only that, once you have the solution, you can't put it into production if it is extremely slow. This is where tools like Numba come into the picture. Numba is a JIT for optimizing numerical processing code in Python. However, to use it effectively, we need to have a good understanding of how the CPU works so that we can write code that takes full advantage of it. This article takes you through a quick tour of CPU internals and then shows how to use Numba and NumPy to write optimized code. Check it out here.

Tinkering with Tiny

Are you interested in coding your own computer game? Well, creating games is one of the most complex projects one can undertake—it's not for the faint-hearted. However, it's not impossible either if someone is there to guide you through the steps. Tom Halligan has you covered with his latest article, where he gives an introduction to a game engine called Tiny. Check out the article linked below, and I hope you have fun with it.

Not A Robot

Tinkering with Tiny

While browsing Hacker News recently, I came across a post about a new game engine called Tiny. The pitch had me intrigued from the get-go: The virtual console that offers an easy and efficient way to build games and other applications with its Lua programming support, hot reloading, and 256-color maximum…

Read more

3 years ago · 5 likes · 6 comments · Tom Halligan

Understanding DeepMind's AlphaDev Breakthrough in Optimizing Sorting Algorithms

A little shameless plug here. I wrote a long-form article explaining the workings of the optimized sorting algorithms that were discovered by the AlphaDev model from DeepMind recently. This has been breaking news, and if you are interested in learning what the fuss is all about, then check out my article (if you haven't already).

Using the Function Call Feature of OpenAI’s ChatGPT APIs to Write Your Own Plugins

OpenAI recently introduced a new feature in their ChatGPT APIs (GPT-4 and GPT-3.5). This feature allows the ChatGPT model to accept a list of function calls as part of its input and it can generate a call to one of those functions in order to perform the task assigned to it. This opens up several new possibilities with ChatGPT. In my latest article I shared a tutorial on how to use this function call feature to build a plugin system in a ChatGPT like chat system. In the tutorial we build a toy chat app along with a web browsing and Python interpreter plugin. If this sounds exciting, then check it out.

📚 Books and Courses

The Little Book of Deep Learning

All of us want to learn deep learning, and why not? All the AI around is being built using it. However, it has become a vast field in itself and requires a lot of background for someone to get started in it. The Little Book of Deep Learning is an accessible introduction to the field. It provides you with enough background on the math, tools, and some of the basic models so that you can dive deeper. The book has been beautifully typeset and is designed so that you can read it on your phone. With just 158 small pages long, it is indeed a little book. Check it out here.

Learn Rust Programming

This is a 13-hour course that teaches Rust from the very beginning and covers all of the Rust syntax and the standard library. The course is from freeCodeCamp. Check it out.

Bonus Tip

As promised, here is a tip to improve your productivity.

Do you use Git from the command line? I know many people who prefer using Git from the CLI rather than through a UI. We normally use the git checkout command to switch Git branches on the command line. For example:

git checkout my-topical-branch

Then, once we are done working on the branch, we again use the checkout command to switch to the previous branch. However, this gets tedious. For example, I tend to frequently switch between master and my feature branch because sometimes I need to fix bugs. There is a handy shortcut to switch to the previous branch without typing its full name. Here it is:

git checkout -

The hyphen tells Git to switch to the previously checked-out branch.

Here's another bonus tip for you: the same trick works with the cd command. If you say `cd -`, the shell switches to the previous directory you were in.

Wrapping Up

I hope you enjoyed the articles and other resources from this week. Let me know in the comments if you enjoyed something in particular, or if something bugged you. Also, if you have anything interesting that you would like to share with me, then do so in the comments, I would love to read it. As always, thanks for reading Confessions of a Code Addict.

Share

As I continue working on a comprehensive long-form article that requires extensive research, I wanted to share some intriguing resources I discovered last week. This post will become a regular feature in my newsletter, providing you with a weekly dose of fascinating content. So, let's dive right in!

Articles

Inside GitHub: Working with the LLMs behind GitHub Copilot

In this article the folks at GitHub share their experience behind building GitHub Copilot, how they started experimenting with the GPT-3 APIs in the form of a static question and answering system, and iterated upon it to ultimately create an interactive plugin for the IDEs. They talk about how this led to the collaboration with OpenAI. They also share some interesting insights behind the prompt engineering they did for Copilot and also their plans for the future of Copilot. Check it out here.

Measuring Time: From Java to Kernel and back

If you are a Java programmer who is interested in performance tuning and measurements, you will find this article very enlightening. We tend to use things like System.currentTimeMillis() and System.nanoTime() to measure the time taken to execute a block of code. However, these calls themselves can have unreliable latency which can throw off our performance tuning efforts. This article dives deep, starting right from the implementation of these APIs in Java, and going deep into the Linux kernel code in order to determine the source of unreliability. Check out the full article here.

Real Multithreading is Coming to Python - Learn How You Can Use It Now

Now, some news for the Python devs out there. If you are a seasoned Python developer, you might know about the fact that Python does not have true concurrency. Even though Python has support for threads, due to the presence of the Global Interpreter Lock (GIL), the Python interpreter allows only one thread to run at any point in time. This means that even if your hardware has multiple CPU cores, they are not being utilized by Python. This has been a long-standing issue for Python developers. Now, there is an effort to mitigate the effects of the GIL. One such effort is discussed in this article. In Python 3.12 (under development), the Python developers have reduced the scope of the GIL to be at the per sub-interpreter level. A sub-interpreter is an instance of the Python interpreter which runs within the main Python interpreter. With this change, each sub-interpreter will have its own GIL, which means we can run multiple sub-interpreters concurrently and utilize multiple CPU cores. This is pretty cool work, read full details in the article here.

My first superoptimizer

Compiler optimization is the most fascinating topic for anyone who is into compilers. One of the optimization techniques is called superoptimization, whose goal is to optimize a given code snippet into its most optimized canonical form. This is a very hard problem and not something that real-world compilers can usually do. This article explores what superoptimization is and also builds a toy implementation. Check out the full article here to learn more about it.

Modern Bloom Filters: 22x Faster!

I also have something for the data structure and algorithm fans. Very few people are aware of a fascinating data structure called the bloom filter. It is a probabilistic data structure commonly used in large-scale big data systems to quickly check whether an item exists in a collection or not. Being a probabilistic data structure, it is not 100% accurate. If the bloom filter says an item does not exist in the collection, then that is 100% guaranteed not to exist in the collection. However, if it says that an item might exist, then that might or might not be true (false positive).

The advantage of using this data structure is that it only takes a fraction of memory to index a large amount of data because instead of storing the entire content in memory, it only stores a digest of the data, which is usually a few bits long.

You might use it in situations such as when you need to check if a password is part of a large set of compromised passwords or not. This article shows an implementation of a typical bloom filter and then goes on to show certain optimized implementations taking advantage of the hardware, which makes it 22x faster. These techniques were new to me, and I found them intriguing. Check out the article here

Is Avoiding Extinction from AI Really an Urgent Priority?

The news about AI experts raising concerns about the possibility of extinction because of AI has been trending this week. However, there is another set of experts who believe that these extinction claims are overhyped, and we have much more important issues to deal with. Seth Lazar, Jeremy Howard, and Arvind Narayanan share their views on this topic in this article. Check it out here.

Courses & Books

Computer Networks: A Systems Approach

If you are interested in learning about computer networks and protocols in a hands-on fashion, then you will love this book. It is an open-source book that teaches everything about computer networks using the systems approach. The authors define the systems approach as a technique that explores how different system components interact with each other, rather than studying each component in isolation. This is particularly important for computer networks, where each network layer works closely with other layers. This means going beyond a single layer and considering issues that span multiple layers. Congestion control is a prime example of this. The approach prioritizes real-world implementation, exemplified by the Internet as a complex and widely-used network system. Check out the book here.

Purely Functional Data Structures

If you are a fan of functional programming, I have something interesting for you. This book discusses the design and implementation of purely functional data structures. These data structures maintain functional purity by not allowing mutations or modifications in their internal state. In order to maintain this property of immutability, these data structures create a new copy with every operation. The main benefit of these data structures is that due to their immutability, they are well-suited for concurrent and parallel programming, and they are very easy to reason about. However, due to the fact that they need to make a new copy with every operation, they can incur high memory overhead. You can find the PDF version of the book here.

Performance Engineering of Software Systems

This is a course from MIT OpenCourseware on the topic of performance engineering. If you are interested in learning about building high-performance and scalable software systems that extract every ounce of performance from the hardware, then you should take this course.

Wrapping Up

I hope you enjoy some of these articles, books, and courses. Let me know in the comments if you found something particularly interesting in them and also if you have some interesting resources to share with me.

In another news, I’m travelling this week due to a family emergency, so my next article, which I’m very excited about, might be a little delayed. But it will reach your inbox soon. Thank you for reading Confessions of a Code Addict and supporting me.

Share