Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
CPython segfault in 5 lines of code
class E(BaseException):
def __new__(cls, *args, **kwargs):
return cls
def a(): yield
a().throw(E)

CPython Segfault in 5 lines of code

(Works as described on at least CPython 3.6-3.8)

So this is pretty weird, right?

Let's look at why this happens. First, we define a subclass of Exception that returns a non-exception from its __new__ constructor. This could be anything, as long as it's not actually an instance of BaseException:

import random
class E(BaseException):
    def __new__(cls, *args, **kwargs):
        return random.choice([1, (), {}, ""])
def a(): yield
# will still always segfault:
a().throw(E)

Well, almost anything; a few things I've found that don't segfault are lists and generators- I'll come back to this later.

Then, we call the generator.throw() with E as its only argument. gen.throw() accepts up to 3 arguments, which mirror the arguments you'd pass to raise in Python 2:

raise TypeError, "Couldn't do thing", other_exc.__traceback__

We only pass it one argument, which because it's a type object (and a subclass of BaseException) it calls to get an exception object to throw to the generator (again, like raise). Usually, of course, type objects return an instance of themselves when called, but this one we've defined doesn't. But shouldn't it still just TypeError? And does raise do the same thing?

class E(BaseException):
    def __new__(cls, *args, **kwargs):
        return cls
# Python 3 still supports `raise ExceptionType`, just not the other arguments
raise E
Traceback (most recent call last):
  File "segfault.py", line 8, in <module>
    raise E
TypeError: calling <class '__main__.E'> should have returned an instance of BaseException, not <class 'type'>

raise works fine. Hmmm. Let's try looking up that error message in the CPython codebase.

// ceval.c, line 4238
    /* We support the following forms of raise:
       raise
       raise <instance>
       raise <type> */

     /* VVVVVVVVVVVVVVVVVVVVVVVVVVV is the argument a subclass of BaseException? */
    if (PyExceptionClass_Check(exc)) {
        type = exc;
     /* VVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV call the type object */
        value = _PyObject_CallNoArg(exc);
        if (value == NULL)
            goto raise_error;
          /* VVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV type check for instance of BaseException */
        if (!PyExceptionInstance_Check(value)) {
            _PyErr_Format(tstate, PyExc_TypeError,
                        /* VVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV search result for error message */
                          "calling %R should have returned an instance of "
                          "BaseException, not %R",
                          type, Py_TYPE(value));
             goto raise_error;
        }
    }
    else if (PyExceptionInstance_Check(exc)) {
        /* most common path, raising an already constructed exception */
        value = exc;
        type = PyExceptionInstance_Class(exc);
        Py_INCREF(type);
    }
    /* ... */

It looks like this type checking is done specifically in the code for executing the RAISE_VARARGS bytecode instruction, and not in any deeper machinery in the CPython interpreter. So does generator.throw() do this?

// genobject.c, line 483
     /* VVVVVVVVVVVVVVVVVVVVVVVVVVV check if it's a subclass of BaseException... */
    if (PyExceptionClass_Check(typ))
        PyErr_NormalizeException(&typ, &val, &tb);

          /* VVVVVVVVVVVVVVVVVVVVVVVVVVVVVV common path again */
    else if (PyExceptionInstance_Check(typ)) {
        /* Raising an instance.  The value should be a dummy. */
        /* ... */

Nope, it just passes it off to the interpreter function PyErr_NormalizeException. Does that validate the result of calling the exception type? Well, the source is pretty long and not that interesting, but you can read it for yourself and see that it does no checks whatsoever.

So that's why the segfault happens: CPython places a non-exception PyObject as the current exception for the thread, and when it tries to do an operation on it (like editing the traceback), it reaches past the size of the PyTupleObject struct that's actually stored there and segfaults.

I'm still not sure why lists and generators don't cause segfaults; it's not because they conform to the sequence protocol as strings and tuples do as well, and tuples and lists are usually very similar in functionality (besides, like, mutability). I'd assume that it's something in the interpreter code that catches that an exception value isn't actually an exception (but only for these types) and throws a proper error, but I'm not sure. If anyone does actually know, I'd love to hear about it.

Note: I came across this while trying to figure out the proper behavior for generator.throw() for RustPython. I suppose the proper behavior in order to be truly CPython compliant is to just dereference a null pointer! 😁

PyBaseExceptionRef::try_from_object(res).unwrap_or_else(|_| unsafe { *std::ptr::null() })

BPO issue

See discussion on HN or Reddit

@Hellkyte

This comment has been minimized.

Copy link

@Hellkyte Hellkyte commented Dec 18, 2019

Funnily enough this works as expected in cpython2.7.

@coolreader18

This comment has been minimized.

Copy link
Owner Author

@coolreader18 coolreader18 commented Dec 18, 2019

Yeah, some other people have also said it doesn't happen for them on other versions (or only as a script, not in the REPL). I've updated the post to include the version range it occurs in.

@eric-wieser

This comment has been minimized.

Copy link

@eric-wieser eric-wieser commented Dec 18, 2019

@coolreader18: If you haven't already, please file a bug at https://bugs.python.org

@sekrause

This comment has been minimized.

Copy link

@sekrause sekrause commented Dec 18, 2019

An issue has been created on BPO: https://bugs.python.org/issue39091

@divinity76

This comment has been minimized.

Copy link

@divinity76 divinity76 commented Dec 18, 2019

same in cygwin with 3.6.9,

$ python3 --version
Python 3.6.9
$ python3 -c "(i for i in []).throw(type('E', (BaseException,), dict(__new__=lambda cls, *args: cls))())"
Segmentation fault (core dumped)

neat

@P3GLEG

This comment has been minimized.

Copy link

@P3GLEG P3GLEG commented Dec 19, 2019

Wow man! Nice job that's an awesome find.

in /tmp
$ uname -s
Darwin

in /tmp 
$ python --version
Python 3.7.5

in /tmp
$ python segfault.py
[1]    87070 segmentation fault  python segfault.py
@pauldraper

This comment has been minimized.

Copy link

@pauldraper pauldraper commented Dec 19, 2019

67 chars

$ python3.7 -VV
Python 3.7.5 (default, Nov  7 2019, 10:50:52) 
[GCC 8.3.0]
$ python3.7 -c "(i for i in[]).throw(type('',(IOError,),{'__new__':lambda a,*b:a}))"
Segmentation fault (core dumped)
@Wheaties466

This comment has been minimized.

Copy link

@Wheaties466 Wheaties466 commented Dec 19, 2019

same on python 3.6

$ python3 -c "(i for i in[]).throw(type('',(IOError,),{'__new__':lambda a,*b:a}))"
Segmentation fault (core dumped)
$ python3 -V
Python 3.6.8
@bstaletic

This comment has been minimized.

Copy link

@bstaletic bstaletic commented Dec 19, 2019

This is fun!

% python3.5 -c "(i for i in []).throw(type('', (IOError,), dict(__new__=lambda cls, *args: cls))())"
zsh: segmentation fault  python3.5 -c
% python3.5 -V
Python 3.5.9
@horvatha

This comment has been minimized.

Copy link

@horvatha horvatha commented Dec 19, 2019

@bl-ue

This comment has been minimized.

Copy link

@bl-ue bl-ue commented Jun 17, 2021

Hey @coolreader18 — what's going on here? It looks like you fixed this, but in Python 3.9.5 it's still segfaulting:

python3
Python 3.9.5 (default, May  4 2021, 03:36:27) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class E(BaseException):
...     def __new__(cls, *args, **kwargs):
...         return cls
... 
>>> def a(): yield
... 
>>> 
>>> a().throw(E)
zsh: segmentation fault  python3

Edit: I see, python/cpython#17658 is not merged yet. 🤔

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment