7 stupid things I still sometimes do after 16 years of writing Python

My partial life story: you can skip this part if you don’t care

That number 16 is a bit misleading, because yes, I first picked up Python in 2003 (I don’t have a computer science background; I studied music and chemistry!); I was actually trying to write a novel that year (and holy crap, did I fail miserably, as a creative writer I’m short stories only, but I’ve had some published, along with poetry). As anyone who’s ever been a writer knows, sometimes you just gotta get your head into something else to let your unconscious do some work for you, so instead of cleaning the house (reaaaally not my thing, I’m ashamed to admit) like many writers, I decided to teach myself how to program (I’d learned BASIC as a kid on a Texas Instruments home PC with an audio tape drive — yes I’m old — and took one Pascal class in university). I bought a big tome of C++ and was like, whoah, this is a bit much for a pastime. So I got Mark Lutz’s Learning Python (from Amazon, I think!) and that was a lot easier. I hated Windows and had an unreasonable Grudge against Mac for switching to OS X right after I bought an OS 9 iMac with a lot of software, so I tootled around with Python on Linux (Debian, IIRC), basically mostly using it as a replacement for bash scripting and awk so I wouldn’t have to learn those. In 2009, I was working in a biochemistry lab and overheard someone in the bioinformatics department say to a colleague, “Gee, it’s too bad nobody here knows how to use Python”, and I got co-opted part-time after tentatively raising my hand. I’ve now been coding full-time-plus (I get home from work and code some more!) for 8 years.


So after 8 or 16 years, however you want to look at it, I still find my stupid fingers doing the same stupid things and all I can say is thank god for pylint built into VSCode. But I thought these might be amusing to share.

Note that this isn’t your typical list of python “gotchas” like using mutables as default arguments or late-binding closure — those are real things that beginners should dive into and entirely grok and are waaaaay beyond the scope of this article. If you want some real python gotchas, here are four resources:

  • Matrin Chikilian on Toptal’s Top 10 mistakes that Python programmers make
  • Jibu James on Sayonetech’s Seven mistakes Python programmers make (he’s got a good __init__ best practice I don’t see talked about much, but he does call the method a constructor which is a pet peeve of mine, but it’s certainly not uncommon for people to make that mistake especially if they come from a Java background. Meh, it’s just a word, who cares?)
  • Pete Savage on Opensource’s 3 things I did wrong learning Python. Spoiler alert: all three are using mutables as default arguments, but he gives two ways that can be done that aren’t the usual def foo(bar=[]), so perhaps worth a read!
  • The “Common Gotchas” chapter in Kenneth Reitz’s online and paper O’Reilly book The Hitchhiker’s Guide to Python, which I mention again below

 

1. Confusing a string’s .find() method with re.search()

EDIT: So it turns out I’m evenĀ more stupid than I thought. If you want to find a literal match of a substring, it’s much better to just do:

if substring in string:

Which saves you all the problems I point out in this one. D’oh!

This is by far the most dangerous one because it causes errors to pass silently and there’s no way for a linter to read your mind.

Let’s say you have some logic that parses a bit of text and the condition is that text contains the word “dotard” (to pick a word at random).

There are two ways to do this, generally:

if mystring.find('dotard') != -1:
    do_stuff()

or

if re.search('dotard', mystring):
    do_stuff()

The problem is if you get a brain fart and mix up the two and write:

if mystring.find('dotard'):
    do_stuff()

The condition will always evaluate to true (-1 is truthy! as are 1,2,3… of course) unless ‘dotard’ are the first six letters of mystring, in which case it will evaluate to False (because the correctly returned index value, 0, is falsy!).

I found myself doing this occasionally while writing tests, which is a good place to make that mistake, because the tests don’t turn out as expected and you can figure out why. But I’m super paranoid about it now.

 

2. Forgetting to type ‘enumerate’

How many times have I written:

for i, item in mylist:

instead of

for i, item in enumerate(mylist):

I mean it’s easily caught, but it bothers me that I still do that occasionally to this day. My theory is that my lazy lizard brain finds “enumerate” too long a word to type (and it’s a weird word to type, one left-hand finger, than three right-hand fingers, then five left-hand fingers on keys that are very close together, I know I’m going deep into the weeds for an excuse here)

 

3. Forgetting to type ‘range’

I don’t have any good excuses for this except for sheer inattention. Sometimes I’ll write

n = some_function_that_returns_some_integer()
for i in n:
    do_something(i)

instead of

n = some_function_that_returns_some_integer()
for i in range(n):
do_something(i)

4. Misusing (or, really, ever using) assert statements

‘assert’ statements should only be used as sanity checks, i.e. to check for a condition that, in the normal, operation of a program, should be literally impossible. As a stupid example:

myint = 3
assert myint != 3, "I have made a huge mistake"

Why? Because you can make your .py script run a (tiny, tiny) bit faster by making it skip assertions by typing, e.g. python -O myscript.py at the terminal. So if you’re using assert to control program flow (which is sometimes done with exceptions) or to check an implausible but not impossible condition, someone running your program in the future might have errors passing silently.

The way around this is to raise an AssertionError instead:

myint = 3
if myint != 3:
    raise AssertionError("I have made a huge mistake")

It’s certainly worth checking the assumption that this is actually an assertion at all. If you’re checking for the value of variable, it seems that ValueError might be a more appropriate error to raise.

One other caveat: If you’re using pytest, some CI tools like codacity will object to all your assert statements. NO ONE in their right minds will run pytest -O, so you can ignore those, pytest is built all around bare asserts so you don’t have to use unittest’s assertEquals or whatever.

5. Forgetting the syntax of multiple Exceptions

This probably just happens to me. The correct syntax in Python 3.x is:

try:
    do_something()
except (ValueError, IndexError) as e:
    #handle the exception

My theory is I spent so long in Python 2 typing except ValueError, e that my brain still won’t let me type parentheses when it sees me type a comma in an except statement, I dunno.

6. Using _ as a throwaway in the REPL or Jupyter notebook

It is good and proper to use throwaway variables instead of cluttering up the namespace if they’re not actually being used. For example, if you need to do something n times but don’t need to know the index number of each time you do it, you generally do something like this:

for _ in range(n):
    do_my_thing()

No reason to clutter up the namespace with, say, i if it’s never going to be used, and pylint is good at warning you when you do this.

The problem is that in the REPL, _ is also used to represent the most recently evaluated expression. For example:

>>> def add_1(n):
...     return n + 1
>>> add_1(100)
101
>>> _
101

Although it’s never actually happened to me (I don’t use the REPL much, and I try to stay away from Jupyter notebooks after previously needing to use them in a job that was in danger of giving me some very lazy coding habits but if you want to use them, fine, go ahead, I don’t judge), it certainly has the potential to cause confusion.

Kenneth Reitz in The Hitchhiker’s Guide to Python has a great solution to this: use a double underscore __ instead of the single underscore _ for the throwaway. (Aside: that’s a free book Mr. Reitz spent a lot of time putting together, consider paying for the print version on O’Reilly. Also don’t take it uncritically, it pushes Reitz’s own pipenv project a bit too much for my comfort, and since pipenv vendorizes dependencies, the fact that the “Vendorizing Dependencies” section of the online book is TBD is a little ironic to me.) But to give credit where credit is due, it has the best explanation of late-binding closures I’m aware of, which unlike most others, (1) makes a point of pointing out it’s possible in any function, not just lambdas, even though in practice that’s where the mistake happens, (2) points out how you can use functools.partial to address it instead of the standard weird, hacky way, and (3) points out that sometimes late binding Python behavior is useful! I’ll just point out a very specific (4), this gotcha can getcha when using Python’s numpy library’s piecewise function.

7. Using normal string formatting syntax in logging

As everyone should know, you should log, especially when you’re developing! And when logging, you don’t use regular Python string syntax, it has its own *arg-unpacking-based syntax:

log.info('The function returned %d' % myint)  # wrong
log.info('The function returned {}'.format(myint))  # still wrong
log.info('The function returned {foo}'.format(foo=myint))  # wrong again
log.info(f'The function returned {myint}')  # sorry f-strings, still wrong

# it's actually:
log.info('The function returned %d', myint)

That said there are TONS of third-party logging modules that allow you to do regular string formatting. I like loguru myself, but only in while first developing, once the project is mature it’s back to the standard library logging module so it will be that way in prod.


So there it is! I’m not the most experienced programmer in the world (I consider myself on the cusp between Intermediate and Advanced, I mean I’ve overloaded the __new__ dunder for subclass validation a few times, that’s advanced, right? Spoiler alert: that’s my next tutorial subject). Just thought this might at least be an amusing, at most an informative read. Always good to learn from the mistakes of others, even the stupid ones (er, stupid mistakes, I don’t think I’m stupid as a person?) What’s frustrating is when you just can’t seem to shake the same damn mistakes you make over and over and over again, but I think they just call that life.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.