python 2.7.15 Change Log

+++++++++++
Python News
+++++++++++

What's New in Python 2.7.15 final?
==================================

*Release date: 2018-04-29*

Core and Builtins
-----------------

- bpo-33374: Tweak the definition of PyGC_Head, so compilers do not believe
  it is always 16-byte aligned on x86. This prevents crashes with more
  aggressive optimizations present in GCC 8.


What's New in Python 2.7.15 release candidate 1?
================================================

*Release date: 2018-04-14*

Security
--------

- bpo-32997: A regex in fpformat was vulnerable to catastrophic
  backtracking. This regex was a potential DOS vector (REDOS). Based on
  typical uses of fpformat the risk seems low. The regex has been refactored
  and is now safe. Patch by Jamie Davis.

- bpo-32981: Regexes in difflib and poplib were vulnerable to catastrophic
  backtracking. These regexes formed potential DOS vectors (REDOS). They
  have been refactored. This resolves CVE-2018-1060 and CVE-2018-1061. Patch
  by Jamie Davis.

- bpo-31339: Rewrite time.asctime() and time.ctime(). Backport and adapt the
  _asctime() function from the master branch to not depend on the
  implementation of asctime() and ctime() from the external C library. This
  change fixes a bug when Python is run using the musl C library.

- bpo-30730: Prevent environment variables injection in subprocess on
  Windows.  Prevent passing other environment variables and command
  arguments.

- bpo-30694: Upgrade expat copy from 2.2.0 to 2.2.1 to get fixes of multiple
  security vulnerabilities including: CVE-2017-9233 (External entity
  infinite loop DoS), CVE-2016-9063 (Integer overflow, re-fix),
  CVE-2016-0718 (Fix regression bugs from 2.2.0's fix to CVE-2016-0718) and
  CVE-2012-0876 (Counter hash flooding with SipHash). Note: the
  CVE-2016-5300 (Use os- specific entropy sources like getrandom) doesn't
  impact Python, since Python already gets entropy from the OS to set the
  expat secret using ``XML_SetHashSalt()``.

- bpo-30500: Fix urllib.splithost() to correctly parse fragments. For
  example, ``splithost('//127.0.0.1#@evil.com/')`` now correctly returns the
  ``127.0.0.1`` host, instead of treating ``@evil.com`` as the host in an
  authentification (``login@host``).

- bpo-29591: Update expat copy from 2.1.1 to 2.2.0 to get fixes of
  CVE-2016-0718 and CVE-2016-4472. See
  https://sourceforge.net/p/expat/bugs/537/ for more information.

Core and Builtins
-----------------

- bpo-33026: Fixed jumping out of "with" block by setting f_lineno.

- bpo-17288: Prevent jumps from 'return' and 'exception' trace events.

- bpo-18533: ``repr()`` on a dict containing its own ``viewvalues()`` or
  ``viewitems()`` no longer raises ``RuntimeError``.  Instead, use ``...``,
  as for other recursive structures.  Patch by Ben North.

- bpo-10544: Yield expressions are now deprecated in comprehensions and
  generator expressions when checking Python 3 compatibility. They are still
  permitted in the definition of the outermost iterable, as that is
  evaluated directly in the enclosing scope.

- bpo-32137: The repr of deeply nested dict now raises a RecursionError
  instead of crashing due to a stack overflow.

- bpo-20047: Bytearray methods partition() and rpartition() now accept only
  bytes-like objects as separator, as documented.  In particular they now
  raise TypeError rather of returning a bogus result when an integer is
  passed as a separator.

- bpo-31733: Add a new PYTHONSHOWREFCOUNT environment variable. In debug
  mode, Python now only print the total reference count if
  PYTHONSHOWREFCOUNT is set.

- bpo-31692: Add a new PYTHONSHOWALLOCCOUNT environment variable. When
  Python is compiled with COUNT_ALLOCS, PYTHONSHOWALLOCCOUNT now has to be
  set to dump allocation counts into stderr on shutdown. Moreover,
  allocations statistics are now dumped into stderr rather than stdout.

- bpo-31478: Prevent unwanted behavior in `_random.Random.seed()` in case
  the argument has a bad ``__abs__()`` method. Patch by Oren Milman.

- bpo-31530: Fixed crashes when iterating over a file on multiple threads.

- bpo-31490: Fix an assertion failure in `ctypes` class definition, in case
  the class has an attribute whose name is specified in ``_anonymous_`` but
  not in ``_fields_``. Patch by Oren Milman.

- bpo-31411: Raise a TypeError instead of SystemError in case
  warnings.onceregistry is not a dictionary. Patch by Oren Milman.

- bpo-31343: Include sys/sysmacros.h for major(), minor(), and makedev().
  GNU C libray plans to remove the functions from sys/types.h.

- bpo-31311: Fix a crash in the ``__setstate__()`` method of
  `ctypes._CData`, in case of a bad ``__dict__``. Patch by Oren Milman.

- bpo-31243: Fix a crash in some methods of `io.TextIOWrapper`, when the
  decoder's state is invalid. Patch by Oren Milman.

- bpo-31095: Fix potential crash during GC caused by ``tp_dealloc`` which
  doesn't call ``PyObject_GC_UnTrack()``.

- bpo-30657: Fixed possible integer overflow in PyString_DecodeEscape. Patch
  by Jay Bosamiya.

- bpo-27945: Fixed various segfaults with dict when input collections are
  mutated during searching, inserting or comparing.  Based on patches by
  Duane Griffin and Tim Mitchell.

- bpo-25794: Fixed type.__setattr__() and type.__delattr__() for non-
  interned or unicode attribute names.  Based on patch by Eryk Sun.

- bpo-29935: Fixed error messages in the index() method of tuple and list
  when pass indices of wrong type.

- bpo-28598: Support __rmod__ for subclasses of str being called before
  str.__mod__. Patch by Martijn Pieters.

- bpo-29602: Fix incorrect handling of signed zeros in complex constructor
  for complex subclasses and for inputs having a __complex__ method. Patch
  by Serhiy Storchaka.

- bpo-29347: Fixed possibly dereferencing undefined pointers when creating
  weakref objects.

- bpo-14376: Allow sys.exit to accept longs as well as ints. Patch by Gareth
  Rees.

- bpo-29028: Fixed possible use-after-free bugs in the subscription of the
  buffer object with custom index object.

- bpo-29145: Fix overflow checks in string, bytearray and unicode. Patch by
  jan matejek and Xiang Zhang.

- bpo-28932: Do not include <sys/random.h> if it does not exist.

Library
-------

- bpo-33096: Allow ttk.Treeview.insert to insert iid that has a false
  boolean value. Note iid=0 and iid=False would be same. Patch by Garvit
  Khatri.

- bpo-33127: The ssl module now compiles with LibreSSL 2.7.1.

- bpo-30622: The ssl module now detects missing NPN support in LibreSSL.

- bpo-21060: Rewrite confusing message from setup.py upload from "No dist
  file created in earlier command" to the more helpful "Must create and
  upload files in one command".

- bpo-30157: Fixed guessing quote and delimiter in csv.Sniffer.sniff() when
  only the last field is quoted.  Patch by Jake Davis.

- bpo-32647: The ctypes module used to depend on indirect linking for
  dlopen. The shared extension is now explicitly linked against libdl on
  platforms with dl.

- bpo-32304: distutils' upload command no longer corrupts tar files ending
  with a CR byte, and no longer tries to convert CR to CRLF in any of the
  upload text fields.

- bpo-31848: Fix the error handling in Aifc_read.initfp() when the SSND
  chunk is not found. Patch by Zackery Spytz.

- bpo-32521: The nis module is now compatible with new libnsl and headers
  location.

- bpo-32539: Fix ``OSError`` for ``os.listdir`` with deep paths (starting
  with ``\\?\``) on windows.  Patch by Anthony Sottile.

- bpo-32521: glibc has removed Sun RPC. Use replacement libtirpc headers and
  library in nis module.

- bpo-18035: ``telnetlib``: ``select.error`` doesn't have an ``errno``
  attribute. Patch by Segev Finer.

- bpo-32185: The SSL module no longer sends IP addresses in SNI TLS
  extension on platforms with OpenSSL 1.0.2+ or inet_pton.

- bpo-32186: Creating io.FileIO() and builtin file() objects now release the
  GIL when checking the file descriptor. io.FileIO.readall(),
  io.FileIO.read(), and file.read() now release the GIL when getting the
  file size.  Fixed hang of all threads with inaccessible NFS server.  Patch
  by Nir Soffer.

- bpo-32110: ``codecs.StreamReader.read(n)`` now returns not more than *n*
  characters/bytes for non-negative *n*. This makes it compatible with
  ``read()`` methods of other file-like objects.

- bpo-21149: Silence a `'NoneType' object is not callable` in
  `_removeHandlerRef` error that could happen when a logging Handler is
  destroyed as part of cyclic garbage collection during process shutdown.

- bpo-31764: Prevent a crash in ``sqlite3.Cursor.close()`` in case the
  ``Cursor`` object is uninitialized. Patch by Oren Milman.

- bpo-31955: Fix CCompiler.set_executable() of distutils to handle properly
  Unicode strings.

- bpo-9678: Fixed determining the MAC address in the uuid module:

  * Using ifconfig on NetBSD and OpenBSD.
  * Using arp on Linux, FreeBSD, NetBSD and OpenBSD.

  Based on patch by Takayuki Shimizukawa.

- bpo-30057: Fix potential missed signal in signal.signal().

- bpo-31927: Fixed reading arbitrary data when parse a AF_BLUETOOTH address
  on NetBSD and DragonFly BSD.

- bpo-27666: Fixed stack corruption in curses.box() and curses.ungetmouse()
  when the size of types chtype or mmask_t is less than the size of C long.
  curses.box() now accepts characters as arguments.  Based on patch by Steve
  Fink.

- bpo-25720: Fix the method for checking pad state of curses WINDOW. Patch
  by Masayuki Yamamoto.

- bpo-31893: Fixed the layout of the kqueue_event structure on OpenBSD and
  NetBSD. Fixed the comparison of the kqueue_event objects.

- bpo-31891: Fixed building the curses module on NetBSD.

- bpo-30058: Fixed buffer overflow in select.kqueue.control().

- bpo-31770: Prevent a crash when calling the ``__init__()`` method of a
  ``sqlite3.Cursor`` object more than once. Patch by Oren Milman.

- bpo-31728: Prevent crashes in `_elementtree` due to unsafe cleanup of
  `Element.text` and `Element.tail`. Patch by Oren Milman.

- bpo-31752: Fix possible crash in timedelta constructor called with custom
  integers.

- bpo-31681: Fix pkgutil.get_data to avoid leaking open files.

- bpo-31675: Fixed memory leaks in Tkinter's methods splitlist() and split()
  when pass a string larger than 2 GiB.

- bpo-30806: Fix the string representation of a netrc object.

- bpo-30347: Stop crashes when concurrently iterate over itertools.groupby()
  iterators.

- bpo-25732: `functools.total_ordering()` now implements the `__ne__`
  method.

- bpo-31351: python -m ensurepip now exits with non-zero exit code if pip
  bootstrapping has failed.

- bpo-31544: The C accelerator module of ElementTree ignored exceptions
  raised when looking up TreeBuilder target methods in XMLParser().

- bpo-31455: The C accelerator module of ElementTree ignored exceptions
  raised when looking up TreeBuilder target methods in XMLParser().

- bpo-25404: SSLContext.load_dh_params() now supports non-ASCII path.

- bpo-28958: ssl.SSLContext() now uses OpenSSL error information when a
  context cannot be instantiated.

- bpo-27448: Work around a `gc.disable()` race condition in the `subprocess`
  module that could leave garbage collection disabled when multiple threads
  are spawning subprocesses at once.  Users are *strongly encouraged* to use
  the `subprocess32` module from PyPI on Python 2.7 instead, it is much more
  reliable.

- bpo-31170: expat: Update libexpat from 2.2.3 to 2.2.4. Fix copying of
  partial characters for UTF-8 input (libexpat bug 115):
  https://github.com/libexpat/libexpat/issues/115

- bpo-29136: Add TLS 1.3 cipher suites and OP_NO_TLSv1_3.

- bpo-31334: Fix ``poll.poll([timeout])`` in the ``select`` module for
  arbitrary negative timeouts on all OSes where it can only be a non-
  negative integer or -1. Patch by Riccardo Coccioli.

- bpo-10746: Fix ctypes producing wrong PEP 3118 type codes for integer
  types.

- bpo-30102: The ssl and hashlib modules now call
  OPENSSL_add_all_algorithms_noconf() on OpenSSL < 1.1.0. The function
  detects CPU features and enables optimizations on some CPU architectures
  such as POWER8. Patch is based on research from Gustavo Serra Scalet.

- bpo-30502: Fix handling of long oids in ssl.  Based on patch by Christian
  Heimes.

- bpo-25684: Change ``ttk.OptionMenu`` radiobuttons to be unique across
  instances of ``OptionMenu``.

- bpo-29169: Update zlib to 1.2.11.

- bpo-30746: Prohibited the '=' character in environment variable names in
  ``os.putenv()`` and ``os.spawn*()``.

- bpo-28994: The traceback no longer displayed for SystemExit raised in a
  callback registered by atexit.

- bpo-30418: On Windows, subprocess.Popen.communicate() now also ignore
  EINVAL on stdin.write() if the child process is still running but closed
  the pipe.

- bpo-30378: Fix the problem that logging.handlers.SysLogHandler cannot
  handle IPv6 addresses.

- bpo-29960: Preserve generator state when _random.Random.setstate() raises
  an exception. Patch by Bryan Olson.

- bpo-30310: tkFont now supports unicode options (e.g. font family).

- bpo-30414: multiprocessing.Queue._feed background running thread do not
  break from main loop on exception.

- bpo-30003: Fix handling escape characters in HZ codec.  Based on patch by
  Ma Lin.

- bpo-30375: Warnings emitted when compile a regular expression now always
  point to the line in the user code.  Previously they could point into
  inners of the re module if emitted from inside of groups or conditionals.

- bpo-30363: Running Python with the -3 option now warns about regular
  expression syntax that is invalid or has different semantic in Python 3 or
  will change the behavior in future Python versions.

- bpo-30365: Running Python with the -3 option now emits deprecation
  warnings for getchildren() and getiterator() methods of the Element class
  in the xml.etree.cElementTree module and when pass the html argument to
  xml.etree.ElementTree.XMLParser().

- bpo-30365: Fixed a deprecation warning about the doctype() method of the
  xml.etree.ElementTree.XMLParser class.  Now it is emitted only when define
  the doctype() method in the subclass of XMLParser.

- bpo-30329: imaplib now catchs the Windows socket WSAEINVAL error (code
  10022) on shutdown(SHUT_RDWR): An invalid operation was attempted. This
  error occurs sometimes on SSL connections.

- bpo-30342: Fix sysconfig.is_python_build() if Python is built with Visual
  Studio 2008 (VS 9.0).

- bpo-29990: Fix range checking in GB18030 decoder.  Original patch by Ma
  Lin.

- bpo-30243: Removed the __init__ methods of _json's scanner and encoder.
  Misusing them could cause memory leaks or crashes.  Now scanner and
  encoder objects are completely initialized in the __new__ methods.

- bpo-26293: Change resulted because of zipfile breakage. (See also:
  bpo-29094)

- bpo-30070: Fixed leaks and crashes in errors handling in the parser
  module.

- bpo-30061: Fixed crashes in IOBase methods next() and readlines() when
  readline() or next() respectively return non-sizeable object. Fixed
  possible other errors caused by not checking results of PyObject_Size(),
  PySequence_Size(), or PyMapping_Size().

- bpo-30011: Fixed race condition in HTMLParser.unescape().

- bpo-30068: _io._IOBase.readlines will check if it's closed first when hint
  is present.

- bpo-27863: Fixed multiple crashes in ElementTree caused by race conditions
  and wrong types.

- bpo-29942: Fix a crash in itertools.chain.from_iterable when encountering
  long runs of empty iterables.

- bpo-29861: Release references to tasks, their arguments and their results
  as soon as they are finished in multiprocessing.Pool.

- bpo-27880: Fixed integer overflow in cPickle when pickle large strings or
  too many objects.

- bpo-29110: Fix file object leak in aifc.open() when file is given as a
  filesystem path and is not in valid AIFF format. Original patch by Anthony
  Zhang.

- bpo-29354: Fixed inspect.getargs() for parameters which are cell
  variables.

- bpo-29335: Fix subprocess.Popen.wait() when the child process has exited
  to a stopped instead of terminated state (ex: when under ptrace).

- bpo-29219: Fixed infinite recursion in the repr of uninitialized
  ctypes.CDLL instances.

- bpo-29082: Fixed loading libraries in ctypes by unicode names on Windows.
  Original patch by Chi Hsuan Yen.

- bpo-29188: Support glibc 2.24 on Linux: don't use getentropy() function
  but read from /dev/urandom to get random bytes, for example in
  os.urandom(). On Linux, getentropy() is implemented which getrandom() is
  blocking mode, whereas os.urandom() should not block.

- bpo-29142: In urllib, suffixes in no_proxy environment variable with
  leading dots could match related hostnames again (e.g. .b.c matches
  a.b.c). Patch by Milan Oberkirch.

- bpo-13051: Fixed recursion errors in large or resized
  curses.textpad.Textbox.  Based on patch by Tycho Andersen.

- bpo-9770: curses.ascii predicates now work correctly with negative
  integers.

- bpo-28427: old keys should not remove new values from WeakValueDictionary
  when collecting from another thread.

- bpo-28998: More APIs now support longs as well as ints.

- bpo-28923: Remove editor artifacts from Tix.py, including encoding not
  recognized by codecs.lookup.

- bpo-29019: Fix dict.fromkeys(x) overallocates when x is sparce dict.
  Original patch by Rasmus Villemoes.

- bpo-19542: Fix bugs in WeakValueDictionary.setdefault() and
  WeakValueDictionary.pop() when a GC collection happens in another thread.

- bpo-28925: cPickle now correctly propagates errors when unpickle instances
  of old-style classes.

Documentation
-------------

- bpo-27212: Modify documentation for the :func:`islice` recipe to consume
  initial values up to the start index.

- bpo-32800: Update link to w3c doc for xml default namespaces.

- bpo-17799: Explain real behaviour of sys.settrace and sys.setprofile and
  their C-API counterparts regarding which type of events are received in
  each function. Patch by Pablo Galindo Salgado.

- bpo-8243: Add a note about curses.addch and curses.addstr exception
  behavior when writing outside a window, or pad.

- bpo-21649: Add RFC 7525 and Mozilla server side TLS links to SSL
  documentation.

- bpo-30176: Add missing attribute related constants in curses
  documentation.

- bpo-28929: Link the documentation to its source file on GitHub.

- bpo-26355: Add canonical header link on each page to corresponding major
  version of the documentation. Patch by Matthias Bussonnier.

- bpo-12067: Rewrite Comparisons section in the Expressions chapter of the
  language reference. Some of the details of comparing mixed types were
  incorrect or ambiguous. Added default behaviour and consistency
  suggestions for user- defined classes. Based on patch from Andy Maier.

Tests
-----

- bpo-31719: Fix test_regrtest.test_crashed() on s390x. Add a new
  _testcapi._read_null() function to crash Python in a reliable way on
  s390x. On s390x, ctypes.string_at(0) returns an empty string rather than
  crashing.

- bpo-31518: Debian Unstable has disabled TLS 1.0 and 1.1 for
  SSLv23_METHOD(). Change TLS/SSL protocol of some tests to PROTOCOL_TLS or
  PROTOCOL_TLSv1_2 to make them pass on Debian.

- bpo-25674: Remove sha256.tbs-internet.com ssl test

- bpo-11790: Fix sporadic failures in
  test_multiprocessing.WithProcessesTestCondition.

- bpo-30236: Backported test.regrtest options -m/--match and -G/--failfast
  from Python 3.

- bpo-30223: To unify running tests in Python 2.7 and Python 3, the test
  package can be run as a script.  This is equivalent to running the
  test.regrtest module as a script.

- bpo-30207: To simplify backports from Python 3, the test.test_support
  module was converted into a package and renamed to test.support.  The
  test.script_helper module was moved into the test.support package. Names
  test.test_support and test.script_helper are left as aliases to
  test.support and test.support.script_helper.

- bpo-30197: Enhanced function swap_attr() in the test.test_support module.
  It now works when delete replaced attribute inside the with statement.
  The old value of the attribute (or None if it doesn't exist) now will be
  assigned to the target of the "as" clause, if there is one. Also
  backported function swap_item().

- bpo-28087: Skip test_asyncore and test_eintr poll failures on macOS. Skip
  some tests of select.poll when running on macOS due to unresolved issues
  with the underlying system poll function on some macOS versions.

- bpo-15083: Convert ElementTree doctests to unittests.

Build
-----

- bpo-33163: Upgrade pip to 9.0.3 and setuptools to v39.0.1.

- bpo-32616: Disable computed gotos by default for clang < 5.0. It caused
  significant performance regression.

- bpo-32635: Fix segfault of the crypt module when libxcrypt is provided
  instead of libcrypt at the system.

- bpo-31934: Abort the build when building out of a not clean source tree.

- bpo-31474: Fix -Wint-in-bool-context warnings in PyMem_MALLOC and
  PyMem_REALLOC macros

- bpo-29243: Prevent unnecessary rebuilding of Python during ``make test``,
  ``make install`` and some other make targets when configured with
  ``--enable- optimizations``.

- bpo-23404: Don't regenerate generated files based on file modification
  time anymore: the action is now explicit. Replace ``make touch`` with
  ``make regen-all``.

- bpo-27593: sys.version and the platform module python_build(),
  python_branch(), and python_revision() functions now use git information
  rather than hg when building from a repo.

- bpo-29643: Fix ``--enable-optimization`` configure option didn't work.

- bpo-29572: Update Windows build and OS X installers to use OpenSSL 1.0.2k.

- bpo-28768: Fix implicit declaration of function _setmode. Patch by
  Masayuki Yamamoto

Windows
-------

- bpo-33184: Update Windows build to use OpenSSL 1.0.2o.

- bpo-32903: Fix a memory leak in os.chdir() on Windows if the current
  directory is set to a UNC path.

- bpo-30855: Bump Tcl/Tk to 8.5.19.

- bpo-30450: Pull build dependencies from GitHub rather than svn.python.org.

macOS
-----

- bpo-32726: Provide an additional, more modern macOS installer variant that
  supports macOS 10.9+ systems in 64-bit mode only. Upgrade the supplied
  third-party libraries to OpenSSL 1.0.2n and SQLite 3.22.0. The 10.9+
  installer now supplies its own private copy of Tcl/Tk 8.6.8.

- bpo-24414: Default macOS deployment target is now set by ``configure`` to
  the build system's OS version (as is done by Python 3), not ``10.4``;
  override with, for example, ``./configure MACOSX_DEPLOYMENT_TARGET=10.4``.

- bpo-17128: All 2.7 macOS installer variants now supply their own version
  of ``OpenSSL 1.0.2``; the Apple-supplied SSL libraries and root
  certificates are not longer used.  The ``Installer Certificate`` command
  in ``/Applications/Python 2.7`` may be used to download and install a
  default set of root certificates from the third-party ``certifi`` package.

- bpo-11485: python.org macOS Pythons no longer supply a default SDK value
  (e.g. ``-isysroot /``) or specific compiler version default (e.g.
  ``gcc-4.2``) when building extension modules.  Use ``CC``, ``SDKROOT``,
  and ``DEVELOPER_DIR`` environment variables to override compilers or to
  use an SDK.  See Apple's ``xcrun`` man page for more info.

- bpo-33184: Update macOS installer build to use OpenSSL 1.0.2o.

Tools/Demos
-----------

- bpo-31920: Fixed handling directories as arguments in the ``pygettext``
  script. Based on patch by Oleg Krasnikov.

- bpo-30109: Fixed Tools/scripts/reindent.py for non-ASCII files. It now
  processes files as binary streams. This also fixes "make reindent".

- bpo-24960: 2to3 and lib2to3 can now read pickled grammar files using
  pkgutil.get_data() rather than probing the filesystem. This lets 2to3 and
  lib2to3 work when run from a zipfile.

C API
-----

- bpo-20891: Fix PyGILState_Ensure(). When PyGILState_Ensure() is called in
  a non-Python thread before PyEval_InitThreads(), only call
  PyEval_InitThreads() after calling PyThreadState_New() to fix a crash.

- bpo-31626: When Python is built in debug mode, the memory debug hooks now
  fail with a fatal error if realloc() fails to shrink a memory block,
  because the debug hook just erased freed bytes without keeping a copy of
  them.