Learning Python - Day 6 - Exceptions

In most programs, there is a clear notion of the normal path through the code. But, of course, conditions can arise where this normal flow can't be followed. For example, if a program involves reading a file specified by the user, it may actually happen that the file doesn't exist. Conditions like this often need to be handled, and the standard mechanism for doing so in Python, as with many other languages, is with what are known as exceptions. In this module of Core Python: Getting Started, we will learn what exceptions are. We'll see how to introduce or raise an exception, and we'll look at how doing so interrupts the normal flow of a program. We'll learn about how you can catch exceptions to handle them, and we'll see what happens to your program if you choose not to handle exceptions. We'll talk a bit about Python's somewhat liberal approach to the use of exceptions. We'll explore some of Python's built?in exception types, and we'll see that some of these indicate programmer errors, while others represent various other kinds of conditions. And we'll look at an important mechanism for ensuring resource cleanup when exceptions are involved. Exception handling is a mechanism for stopping normal program flow and continuing at some surrounding context or code block. The event of interrupting normal flow is called the act of raising an exception. In some enclosing context, the raised exception must be handled upon which control flow is transferred to the exception handler. If an exception propagates up the callstack to the start of the program, then an unhandled exception will cause the program to terminate. An exception object containing information about where and why an exceptional event occurred is transported from the point at which the exception was raised to the exception handler so that the handler can interrogate the exception object and take appropriate action. If you've used exceptions in other popular imperative languages like C++ or Java, then you've already got a good idea of how exceptions work in Python. There have been long and tiresome debates over exactly what constitutes an exceptional event, the core issue being that exceptionality is in reality a matter of degree. Some things are more exceptional than others, whereas programming languages tend to impose a false dichotomy by insisting that an event is either entirely exceptional or not at all exceptional. The Python philosophy is at the liberal end of the spectrum when it comes to the use of exceptions. Exceptions are ubiquitous in Python, and it's crucial to understand how to handle them.

Exceptions and Control Flow

import sys

DIGIT_MAP = {
    'zero': '0',
    'one': '1',
    'two': '2',
    'three': '3',
    'four': '4',
    'five': '5',
    'six': '6',
    'seven': '7',
    'eight': '8',
    'nine': '9',

}


# def convert(s):
#     number = ''
#     for token in s:
#         number += DIGIT_MAP[token]
#     x = int(number)
#     return x


# def convert(s):
#     try:
#         number = ''
#         for token in s:
#             number += DIGIT_MAP[token]
#         x = int(number)
#         print(f"Conversion Succeeded! x = {x}")
#     except KeyError:
#         print("Conversion Failed")
#         x = -1
#     except TypeError:
#         print("Conversion Failed, Only use string values as input")
#         x = -1
#     return x

# def convert(s):
#     x = -1
#     try:
#         number = ''
#         for token in s:
#             number += DIGIT_MAP[token]
#         x = int(number)
#         # print(f"Conversion Succeeded! x = {x}")
#     except (KeyError, TypeError):
#         pass
#     return x

# def convert(s):
#     try:
#         number = ''
#         for token in s:
#             number += DIGIT_MAP[token]
#         x = int(number)
#         print(f"Conversion Succeeded! x = {x}")
#     except (KeyError, TypeError):
#         return -1

# def convert(s):
#     try:
#         number = ''
#         for token in s:
#             number += DIGIT_MAP[token]
#         x = int(number)
#         print(f"Conversion Succeeded! x = {x}")
#     except (KeyError, TypeError) as e:
#         print(f"Conversion error: {e!r}",
#               file=sys.stderr)
#         return -1

def convert(s):
    try:
        number = ''
        for token in s:
            number += DIGIT_MAP[token]
        x = int(number)
        print(f"Conversion Succeeded! x = {x}")
    except (KeyError, TypeError) as e:
        print(f"Conversion error: {e!r}",
              file=sys.stderr)
        raise`        

Since exceptions are a means of control flow, they can be clumsy to demonstrate at the REPL. So for this part of the course, we'll be using a Python module to contain our code. Let's start with a very simple module we can use for exploring these important concepts and behaviors. Place this code in a module called exceptional.py. We'll define a function called convert that attempts to construct an integer from a sequence of strings describing its decimal digits. It then returns that intege. Import the convert function from this module into the Python REPL and call our function to see that it has the desired effect. This seems to work, but if we call our function with an object that can't be converted to an integer, we get a trace back from the dictionary lookup. What's happened here is that digit map raised a key error when we tried to look up the string around in it. Of course, it did this because it doesn't have an entry for around. We didn't have a handler in place, so the exception was caught by the REPL and the stack trace was displayed. The key error referred to in the stack trace is the type of the exception object, and the error message, the string around, is part of the payload of the exception object that has been retrieved and printed at the REPL. Notice that the exception propagates across several levels in the call stack.

Handling Exceptions

def sqrt(x):

    """
    Compute square roots using the method of Heron of Alexandria.

    Args:
        param x:    The number for which the square root is to be computed.

    Returns:
        The square root of x.
    """

    guess = x
    i = 0
    while guess * guess != x and i < 20:
        guess = (guess + x / guess) / 2.0
        i += 1
    return guess


def main():
    try:
        print(sqrt(9))
        print(sqrt(2))
        print(sqrt(-1))
    except ZeroDivisionError:
        print("Can't compute square root of a negative number ")
    print("Program executions continue normally here")


if __name__ == '__main__':
    main()        

Let's make our convert function more robust by handling the KeyError using a try except construct. Both the try and except keywords introduce new blocks. The try block contains code that could raise an exception, and the except block contains the code, which performs error handling in the event that an exception is raised. Modify your convert function to look like this. We have decided that if an unconvertible string is supplied, will return ?1. To reinforce your understanding of the control flow here, we'll add a couple of print statements. Let's test this interactively after restarting the REPL. First, we import the convert function from a module named exceptional. Then, we convert the string three four into the number 34. Finally, we try to convert the word eleventeen, which, of course, fails. Note how the print in the try block, after the point at which the exception was raised, was not executed when we passed in eleventeen. Instead, execution was transferred directly to the first statement of the except block. Our function expects its argument s to be iterable, so let's see what happens if we pass an object that isn't, for example, an integer. This time our handler didn't intercept the exception. If we look closely at the trace, we can see that this time we received a TypeError, a different type of exception. Each try block can have multiple corresponding except blocks, which intercept exceptions of different types. Let's add a handler for TypeError, too. Now, if we rerun the same test in a fresh REPL, we find that the TypeError is handled as well. We've got some code duplication between our two exception handlers with that duplicated print statement and assignment. We'll move the assignment in front of the try block, which doesn't change the behavior of the program. Then we'll exploit the fact that both handlers do the same thing by collapsing them into one, using the ability of the except statement to accept a tuple of exception types. Now we see that everything still works as designed. We can convert the string two nine, but converting the word elephant or the integer 451 will fail and return ?1.

Exceptions and Programmer Errors

Now that we are confident with the control flow for exception behavior, we can remove the print statements. But now, when we try to import our program, we get yet another type of exception, an IndentationError, because our except block is now empty, and empty blocks are not permitted in Python programs. This is not an exception that it is ever useful to catch with an except block. Almost anything that goes wrong with a Python program results in an exception. But some, such as IndentationError, SyntaxError, and NameError are the result of programmer errors, which should be identified and corrected during development rather than handled it runtime. The fact that these things are exceptions is mostly useful if you're creating a Python development tool, such as a Python IDE, embedding Python itself in a larger system to support application scripting, or designing a plugin system which dynamically loads code. With that said, we still have the problem of what to do with our empty except block. The solution arrives in the form of the pass keyword, which is a special statement that does precisely nothing. It's a no?op, and its only purpose is to allow us to construct syntactically permissible blocks that are semantically empty. Perhaps in this case, though, it would be better to simplify further and just use multiple return statements and do away with the x variable completely. Sometimes we'd like to get hold of the exception object, in this case, an object of type of KeyError or AttributeError, and interrogate it for more details of what went wrong. We can get a named reference to the exception object by tacking an as clause onto the end of the except a statement. We'll modify our function to print a message with exception details to the standard error stream before returning. To print a standard error, we need to get a reference to the stream from the sys module, so at the top of our module we'll need to import sys. We can then pass sys.stderr as a keyword argument called file to print. Here we use a feature of f strings that we haven't seen before. If you put an !r after the expression, the repr representation of the value will be inserted into your string. In the case of exceptions, this gives us more detailed information about the type of the exception. Let's see that at the REPL.

Re-raising Exceptions

Let's add a second function, string_log, to our module, which calls our convert function and computes the natural log of the result. We've written this fairly innocuous looking bit of code to demonstrate the greatest folly of returning error codes, that they can be ignored by the caller, wreaking havoc amongst unsuspecting code later in the program. A slightly better program might test the value of v before proceeding to the log call. Without such a check, log will, of course, fail when passed the negative error code value. Naturally, the log failure causes the raising of another exception. Much better and altogether more Pythonic is to forget about error return codes completely and go back to raising an exception from convert. Instead of returning an un?Pythonic error code, we can simply omit our error message and re?raise the exception object we're currently handling. This can be done by replacing the return a ?1 with raise at the end of our exception handling block. Without a parameter, raise simply re?raises the exception that is being currently handled. Testing in the REPL, we can see that the original exception type is re?raised whether it's a key error or a type error, but our Conversion error message is printed to standard error along the way.

Exceptions Are Part of the API

Exceptions form an important aspect of the API of a function. Callers of a function need to know which exceptions to expect under various conditions so that they can ensure appropriate exception handlers are in place. We'll use square root finding as an example, using a homegrown square root function, courtesy of Heron of Alexandria, although he probably didn't use Python. Place the following code in a file named roots.py. There's only one language feature in this program we haven't met yet, the logical and operator, which we use in this case to test that two conditions are true on each iteration of the loop. Python also includes a logical or operator, which can be used to test whether either or both operands are true. Running our program, we can see that Heron was really onto something. Let's add a new line to the main function, which takes the square root of ?1. If we run that, we get a new exception. What has happened is that Python has intercepted a division by 0, which occurs on the second iteration of the loop and raised an exception, a ZeroDivisionError. Let's modify our code to catch the exception before it propagates up to the top of the call stack, thereby causing our program to stop, using the try?except construct. Now, when we run the script, we see that we're handling the exception cleanly. We should be careful to avoid a beginner's mistake of having too?tight scopes for exception handling blocks. We can easily use one try?except block for all of our calls to square root. We also add a third print statement to show how execution of the enclosed block is terminated. This is an improvement on what we started with but most likely, users of a square root function don't expect it to throw a ZeroDivisionError. Python provides us with several standard exception types to signal common errors. If a function parameter is supplied with an illegal value, it is customary to raise a ValueError. We can do this by using the raise keyword with a newly created exception object, which we can create by calling the ValueError constructor. There are two places we could deal with the division by 0. The first approach would be to wrap the root finding while loop in a try?except ZeroDivisionError construct and then raise a new ValueError exception from inside the exception handler. This would be wasteful, though. We know this routine will fail with negative numbers so we can detect this precondition early on and raise an exception at that point, The test is a simple if statement and a call to raise, passing the new exception object. The ValueError constructor accepts an error message. See how we can modify the doc string to make it plain which exception type will be raised by square root and under what circumstances. But look what happens if we run the program. We're still getting a traceback and an ungraceful program exit. This happens because we forgot to modify our exception handler to catch ValueError rather than ZeroDivisionError. Let's modify our calling code to catch the right exception class and also assign the caught exception object to a named variable so that we can interrogate it after it has been caught. In this case, our interrogation is simply to print the exception object, which knows how to display itself as the message to standard error. Running the program again, we can see that our exception is being gracefully handled.

Exceptions and Protocols

Exceptions are part of a function's API, and more broadly, are part of certain protocols. For example, objects which implement the sequence protocol should raise an IndexError exception for indices which are out of range. The exceptions which are raised are as much a part of a function's specification as the arguments it accepts, and as such, must be implemented and documented appropriately. There are a handful of common exception types in Python, and usually, when you need to raise an exception in your own code, one of the built?in types is a good choice. Much more rarely, you'll need to define a new exception type, but we don't cover that in this course. Often, if you're deciding what exceptions your code should raise, you should look for similar cases in existing code. The more your code follows existing patterns, the easier it will be for people to integrate and understand. For example, suppose you were writing a key value database. It would be natural to use KeyError to indicate a request for a non?existent key because this is how dict works, that is mapping in Python follows certain patterns, and exceptions are part of that pattern. Let's look at a few common exception types. IndexError is raised when an integer index is out of range. You can see this when you index past the end of the list. ValueError is raised when an object is of the right type, but contains an inappropriate value. We've seen that already when trying to construct an int from a non?numeric string. KeyError is raised when a lookup in a mapping fails. You can see that here when we look up a non?existent key in a dict.

Avoid Explicit Type Checks

We tend not to protect against type errors in Python. To do so runs against the grain of dynamic typing in Python and limits the reuse potential of code we write. For example, our convert function could test whether the argument was a list using the built?in isinstance function and raise a type error exception if it was not. But then we'd also want to allow arguments that are instances of tuple as well. It soon gets complicated if we want to check whether our function will work with types such as set, dict, or any other iterable type. And in any case, who is to say that it does? Alternatively, as we currently do, we could intercept TypeError inside our convert function and reraise it, but to what end? Usually in Python, it's not worth adding type checking to your functions. If a function works with a particular type, even one that you couldn't have known about when you designed the function, then that's all to the good. If not, execution will probably result in a TypeError anyway. Likewise, we tend not to catch TypeErrors very frequently.

It's Easier to Ask Forgiveness Than Permission

Now let's look at another tenant of Python philosophy and culture, the idea that it's easier to ask forgiveness than permission. There are only two approaches to dealing with a program operation that might fail. The first approach is to check that all the preconditions for a failure?prone operation are met in advance of attempting the operation. The second approach is to blindly hope for the best, but be prepared to deal with the consequences if it doesn't work out. In Python culture, these two philosophies are known as look before you leap, or LBYL, and it's easier to ask forgiveness than permission, EAFP, a term, which, incidentally, was coined by rear admiral Grace Hopper, inventor of the compiler. Python is strongly in favor of EAFP because it puts primary logic for the happy path in its most readable form with deviations from the normal flow handled separately rather than interspersed with the main flow. Let's consider an example, processing a file. The details of the processing aren't relevant. All we need to know is that the process_file function will open a file and read some data from it. First, the LBYL version. Before attempting to call process_file, we check that the file exists, and if it doesn't, we avoid making the call and print a helpful message instead. There are several problems with this approach, some obvious and some insidious. One obvious problem is that we only perform an existence check. What if the file exists, but contains garbage? What if the path refers to a directory instead of a file? According to LBYL, we should add preemptive tests for these too. A more subtle problem is there is a race condition here. It's possible for the file to be deleted, for example, by another process between the existence check and the process_file call, a classic issue of atomicity. There's really no good way to deal with this. Handling of errors from process_file will be needed in any case. Now consider the alternative using the more Pythonic EAFP approach. Here, we simply attempt the operation without checks in advance, but we have an exception handler in place to deal with any problems. We don't even need to know in a lot of detail exactly what might go wrong. Here we catch OSError, which covers all manner of conditions such as file not found and using directories where files are expected. EAFP is standard in Python, and that philosophy is enabled by exceptions. Without exceptions, that is, using error codes instead, you're forced to include error handling directly in the main flow of your logic. Since exceptions interrupt the main flow, they allow you to handle exceptional cases non?locally. Exceptions coupled with EAFP are also superior because, unlike error codes, exceptions cannot be easily ignored. By default, exceptions have a big effect, whereas error codes are silent by default. So the exception EAFP?based style makes it very difficult for problems to be silently ignored.

Cleanup Actions

Sometimes you need to perform a cleanup action, irrespective of whether an operation succeeds. In a later module, we'll introduce context managers, which are the modern solution to this common situation, but here we'll introduce the try finally construct, since creating a context manager can be overkill in simple cases. And in any case, an understanding of try finally is useful for making your own context managers. Consider this function, which uses various facilities of the standard OS module to change the current working directory, create a new directory at that location, and then restore it to the original working directory. At first sight, this seems reasonable, but should the call to os.mkdir fail for some reason, the current working directory of the Python process won't be restored to its original value, and the make_at function will have an unintended side effect. To fix this, we'd like the function to restore the original current working directory under all circumstances. We can achieve this with a try finally block. Code in the finally block is executed whether exception leaves the try block normally by reaching the end of the block, or exceptionally by an exception being raised. This construct can be combined with except blocks, here used to add a simple failure logging facility. Now, if os.mkdir raises an OS error, the OS error handler will be run, and the exception will be re?raised, but since the finally block is always run, no matter how the try block ends, we can be sure that the final directory change will take place in all circumstances. Errors should never pass silently, unless explicitly silenced. Errors are like bells, and if we make them silent, they are of no use.

Platform-Specific Code

Detecting a single key press from Python, such as the 'press any key to continue' functionality at the console, requires use of operating system specific modules. We can't use the built?in input function, because that waits for the user to press Return before giving us a string. To implement this, on Windows we need to use functionality from the Windows only msvcrt module, and on Linux and Mac OS X, we need to use functionality from the Unix only tty and termios modules, in addition to the sys module. This example is quite instructive, as it demonstrates many Python language features including import and def as statements, as opposed to declarations. Recall, The top?level module code is executed on first import. Within the first try block, we attempt to import msvcrt, the Microsoft Visual C Runtime. If this succeeds, we then proceed to define a function, getkey, which delegates to the msvcert getch function. Even though we're inside a try block at this point, the function will be declared at the current scope, which is the module scope. If, however, the import of msvcrt fails because we're not running on Windows, an import error will be raised, and execution will transfer to the except block. This is a case of an error being silenced explicitly because we're going to attempt an alternative course of action in the exception handler. Within the except block, we import three modules needed for a getkey implementation on Unix?like systems, and then proceed to the alternative definition of getkey, which, again, binds the function implementation to a name in the module scope. This Unix implementation of getkey uses a try finally construct to restore various terminal attributes after the terminal has been put into raw mode for the purposes of reading a single character. In the event that our program is running on neither Windows nor a Unix?like system, the import tty statement will raise a second import error. This time we make no attempt to intercept the exception. We allow it to propagate to our caller, which is whatever attempted to import this key press module. We know how to signal this error, but not how to handle it, so we defer that decision to our caller. The error will not pass silently. If the caller has more knowledge or alternative tactics available, it can in turn intercept this exception and take appropriate action, perhaps degrading to using Python's input built?in function and giving a different message to the user.

Summary

Exceptions are an essential topic in Python, and it's critical that you understand how to work with them as you develop your mastery of the language. In this article, we learned that the raising of an exception interrupts normal program flow and transfers control to an exception handler. Exception handlers are defined using the try except construct. Try blocks define a context in which exceptions can be detected. Corresponding except blocks define handlers for specific types of exceptions. Python uses exceptions pervasively, and many built?in language features depend on them. Except blocks can capture an exception object, which is often of a standard type such as value error, key error, or index error. Programmer errors, such as indentation error and syntax error, should not normally be handled. Exceptional conditions can be signaled using the raise keyword, which accepts a single parameter of an exception object. Raise without an argument within an except block re?raises the exception which is currently being processed. We tend not to routinely check for type errors. To do so would negate the flexibility afforded to us by Python's dynamic type system. Exception objects can be converted to strings using the str constructor for the purposes of printing message payloads. The exceptions thrown by a function form part of its API and should be appropriately documented. When raising exceptions, prefer to use the most appropriate built?in exception type. Cleanup and restorative actions can be performed using the try finally construct, which may optionally be used in conjunction with except blocks. Along the way, we saw that the output of the print function can be redirected to standard error using the optional file argument. The expressions in f strings can be suffixed with !r to use the repr representation of the inserted value. Python supports the logical operators and an or for combining Boolean expressions. Return codes are too easily ignored. Platform?specific actions can be implemented using an easier to ask forgiveness than permission approach facilitated by intercepting import errors and providing alternative implementations. In the next module of Core Python: Getting Started, we'll take a deeper look at the concept of iteration in Python, covering topics including comprehensions, Python's iteration protocols, and lazy evaluation.