A "Huh" Moment with Python: The Quirk of str.join() and Generators

I love those unexpected "Huh, I didn't know that" moments, especially when they pop up in random conversations on Mastodon. Recently, I had one about Python—a language I've been coding in for 5 years, the last seven of which have been almost exclusively full-time. Despite my experience, Python still manages to surprise me, particularly with its quirks, especially in CPython, the most widely used implementation of the language.

Joining Strings in Python: The Standard Approach

The conversation that sparked this revelation began with a simple post about neat Python tricks. One of them involved the str.join() method—a standard tool for manipulating strings. Most of us know that Python strings are immutable, meaning once a string is created, it can't be altered. This immutability can make concatenating strings using the + operator inefficient, especially in long loops, because each operation creates a new string.

To improve efficiency, a common practice is to append strings to a list and join them at the end using str.join(). This approach is faster and often used when generating large texts, like HTML.

The Generator vs. List Comprehension Debate

Python also offers generators, which allow you to iterate over an object without loading the entire sequence into memory. This feature is useful for memory efficiency, especially when processing large data, like reading lines from a file.

Here's a typical example of filtering lines from a file:

f =f = open("input_file.txt") filtered_text = "\n".join(x for x in f if not x.startswith("#")) open("input_file.txt") filtered_text = "\n".join(x for x in f if not x.startswith("#"))        

This "textbook" solution uses a generator to filter out lines starting with #, then joins the remaining lines with \n.

Alternatively, you can achieve the same result with list comprehension:

filtered_text = "\n".join([x for x in f if not x.startswith("#")])        

While the syntax is nearly identical, the difference lies in those extra brackets. The generator approach is often preferred for memory efficiency. However, a recent exchange with Trey Hunner, a Python educator, made me realize that this might not always be the best approach—particularly when using str.join().

The Quirk of str.join() in CPython

Curious, I ran a series of tests to compare the memory usage and performance between generators and list comprehensions when used with str.join(). The results were surprising.

When you pass a generator to str.join(), CPython doesn't actually use it directly. Instead, it converts the generator into a list first! This extra step negates the memory efficiency benefits of using a generator in the first place.

Here’s a quick comparison of the performance:

%timeit " ".join(a for a in data if len(a) > 1) # Using generator 7.82 s ± 4.83 ms per loop %timeit " ".join([a for a in data if len(a) > 1]) # Using list comprehension 6.76 s ± 5.99 ms per loop        

The generator approach was consistently about 16% slower than list comprehension. The reason? CPython's str.join() method internally converts the generator into a list before proceeding, resulting in an additional step that slows down the process.

The Takeaway

This quirk is specific to CPython and doesn’t apply to all Python implementations. For example, in PyPy, an alternative Python implementation, the generator version might perform better. However, in CPython, if you're looking for speed with str.join(), list comprehensions are the way to go.

Have you encountered any other surprising Python quirks? Share your experiences below!