3.8.0a4.rst

Fixes mishandling of pre-normalization characters in urlsplit().

Address :cve:`2019-9740` by disallowing URL paths with embedded whitespace or control characters through into the underlying http client request. Such potentially malicious header injection URLs now cause an http.client.InvalidURL exception to be raised.

:func:`shutil.which` now uses os.confstr("CS_PATH") if available and if the :envvar:`PATH` environment variable is not set. Remove also the current directory from :data:`posixpath.defpath`. On Unix, :func:`shutil.which` and the :mod:`subprocess` module no longer search the executable in the current directory if the :envvar:`PATH` environment variable is not set.

In debug build, import now also looks for C extensions compiled in release mode and for C extensions compiled in the stable ABI.

Fix Python Initialization code on FreeBSD to detect properly when stdin file descriptor (fd 0) is invalid.

Remove parser headers and related function declarations that lack implementations after the removal of pgen.

dict.pop() is now up to 33% faster thanks to Argument Clinic. Patch by Inada Naoki.

Debug memory allocators: disable serialno field by default from debug hooks on Python memory allocators to reduce the memory footprint by 5%. Enable :mod:`tracemalloc` to get the traceback where a memory block has been allocated when a fatal memory error is logged to decide where to put a breakpoint. Compile Python with PYMEM_DEBUG_SERIALNO defined to get back the field.

On AIX, :data:`sys.platform` doesn't contain the major version anymore. Always return 'aix', instead of 'aix3' .. 'aix7'. Since older Python versions include the version number, it is recommended to always use sys.platform.startswith('aix'). Contributed by M. Felt.

Change str.capitalize to use titlecase for the first character instead of uppercase.

Implement PEP 570 (Python positional-only parameters). Patch by Pablo Galindo.

:c:func:`!PyEval_AcquireLock` and :c:func:`!PyEval_AcquireThread` now terminate the current thread if called while the interpreter is finalizing, making them consistent with :c:func:`PyEval_RestoreThread`, :c:func:`Py_END_ALLOW_THREADS`, and :c:func:`PyGILState_Ensure`.

Fix signed integer overflow in _ctypes.c's PyCArrayType_new().

Fix running script with encoding cookie and LF line ending may fail on Windows.

Fixed support of the surrogatepass error handler in the UTF-8 incremental decoder.

Changing dict keys during iteration of the dict itself, keys(), values(), or items() will now be detected in certain corner cases where keys are deleted/added so that the number of keys isn't changed. A RuntimeError will be raised after len(dict) iterations. Contributed by Thomas Perl.

Fix a possible double PyMem_FREE() due to tokenizer.c's tok_nextc().

Fixed TypeError message in classmethoddescr_call.

Fix a possible reference leak in :func:`itertools.count`.

Include node names in ParserError messages, instead of numeric IDs. Patch by A. Skrobov.

Regenerate :mod:`keyword` from the Grammar and Tokens file using pgen. Patch by Pablo Galindo.

Add missing :c:func:`PyObject_GC_Track` calls in the :mod:`pickle` module. Patch by Zackery Spytz.

Fix pythoninfo when the compiler is missing.

The .find*() methods of xml.etree.ElementTree can now search for wildcards like {*}tag and {ns}* that match a tag in any namespace or all tags in a namespace. Patch by Stefan Behnel.

pathlib.path.link_to() is now implemented. It creates a hard link pointing to a path.

:class:`fileinput.FileInput` now uses the input file mode to correctly set the output file mode (previously it was hardcoded to 'w') when inplace=True is passed to its constructor.

Fix compilation of faulthandler.c on HP-UX. Initialize stack_t current_stack to zero using memset().

The xml.etree.ElementTree packages gained support for C14N 2.0 serialisation. Patch by Stefan Behnel.

Add missing matrix multiplication operator support to weakref.proxy.

The XMLParser() in xml.etree.ElementTree provides namespace prefix context to the parser target if it defines the callback methods "start_ns()" and/or "end_ns()". Patch by Stefan Behnel.

The TreeBuilder and XMLPullParser in xml.etree.ElementTree gained support for parsing comments and processing instructions. Patch by Stefan Behnel.

The C version of functools.lru_cache() was treating calls with an empty **kwargs dictionary as being distinct from calls with no keywords at all. This did not result in an incorrect answer, but it did trigger an unexpected cache miss.

Fix distutils.sysconfig if :data:`sys.executable` is None or an empty string: use :func:`os.getcwd` to initialize project_base. Fix also the distutils build command: don't use :data:`sys.executable` if it is None or an empty string.

:func:`shutil.which` and distutils.spawn.find_executable now use os.confstr("CS_PATH") if available instead of :data:`os.defpath`, if the PATH environment variable is not set. Moreover, don't use os.confstr("CS_PATH") nor :data:`os.defpath` if the PATH environment variable is set to an empty string.

improve performance of IPNetwork.__contains__()

Path expressions in xml.etree.ElementTree can now avoid explicit namespace prefixes for tags (or the "{namespace}tag" notation) by passing a default namespace with an empty string prefix.

Fix :mod:`asyncio` wait() not removing callback if exception

Fix isinstance check for Mock objects with spec when the code is executed under tracing. Patch by Karthikeyan Singaravelan.

In development mode (:option:`-X` dev) and in debug build, the :class:`io.IOBase` destructor now logs close() exceptions. These exceptions are silent by default in release mode.

The _lsprof module now uses internal timer same to time.perf_counter() by default. gettimeofday(2) was used on Unix. New timer has better resolution on most Unix platforms and timings are no longer impacted by system clock updates since perf_counter() is monotonic. Patch by Inada Naoki.

json.loads now emits DeprecationWarning when encoding option is specified. Patch by Matthias Bussonnier.

The random module now prefers the lean internal _sha512 module over hashlib for seed(version=2) to optimize import time.

Set backlog=None as the default for socket.create_server.

Fix :func:`time.mktime` error handling on AIX for year before 1970.

Improve error message when trying to open existing DBM database that actually doesn't exist. Patch by Marco Rougeth.

Add statistics.quantiles()

Optimized http.client.HTTPResponse.read() for large response. Patch by Inada Naoki.

If debuglevel is set to >0 in :mod:`http.client`, print all values for headers with multiple values for the same header name. Patch by Matt Houglum.

Deprecated passing required arguments like func as keyword arguments in functions which should accept arbitrary keyword arguments and pass them to other function. Arbitrary keyword arguments (even with names "self" and "func") can now be passed to these functions if the required arguments are passed as positional arguments.

Add statistics.geometric_mean().

os.path.normcase() relies on os.fspath() to check the type of its argument. Redundant checks have been removed from its posixpath.normcase() and ntpath.normcase() implementations. Patch by Wolfgang Maier.

Stop rejecting IPv4 octets for being ambiguously octal. Leading zeros are ignored, and no longer are assumed to specify octal octets. Octets are always decimal numbers. Octets must still be no more than three digits, including leading zeroes.

Errors during writing to a ZIP file no longer prevent to properly close it.

Fixed wrong indentation writing for CDATA section in xml.dom.minidom. Patch by Vladimir Surjaninov.

inspect.getdoc() can now find docstrings for member objects when __slots__ is a dictionary.

Calling stop() on an unstarted or stopped :func:`unittest.mock.patch` object will now return None instead of raising :exc:`RuntimeError`, making the method idempotent. Patch by Karthikeyan Singaravelan.

The :meth:`imap.IMAP4.logout` method no longer ignores silently arbitrary exceptions.

Add time module support and fix test_time failures for VxWorks.

Added support for keyword arguments default_namespace and xml_declaration in functions ElementTree.tostring() and ElementTree.tostringlist().

Added new alternate constructors :meth:`datetime.date.fromisocalendar` and :meth:`datetime.datetime.fromisocalendar`, which construct date objects from ISO year, week number and weekday; these are the inverse of each class's isocalendar method. Patch by Paul Ganssle.

:mod:`modulefinder` no longer depends on the deprecated :mod:`imp` module, and the initializer for :class:`modulefinder.ModuleFinder` now has immutable default arguments. Patch by Brandt Bucher.

:mod:`modulefinder` correctly handles modules that have the same name as a bad package. Patch by Brandt Bucher.

:mod:`modulefinder` no longer crashes when encountering syntax errors in followed imports. Patch by Brandt Bucher.

Added :meth:`~socket.create_server()` and :meth:`~socket.has_dualstack_ipv6()` convenience functions to automate the necessary tasks usually involved when creating a server socket, including accepting both IPv4 and IPv6 connections on the same socket. (Contributed by Giampaolo Rodola in :issue:`17561`.)

Add support for :func:`classmethod` and :func:`staticmethod` to :func:`unittest.mock.create_autospec`. Initial patch by Felipe Ochoa.

Fix potential resource warnings in distutils. Patch by Mickaël Schoentgen.

Add transparency methods to :class:`tkinter.PhotoImage`. Patch by Zackery Spytz.

Don't return deleted attributes when calling dir on a :class:`unittest.mock.Mock`.

:class:`wsgiref.handlers.BaseHandler` now handles abrupt client connection terminations gracefully. Patch by Petter Strandmark.

:func:`xml.sax.parse` now supports :term:`path-like <path-like object>`. Patch by Mickaël Schoentgen.

Remove stale unix datagram socket before binding

Implemented Happy Eyeballs in asyncio.create_connection(). Added two new arguments, happy_eyeballs_delay and interleave, to specify Happy Eyeballs behavior.

Do not raise AttributeError when calling the inspect functions isgeneratorfunction, iscoroutinefunction, isasyncgenfunction on a method created from an arbitrary callable. Instead, return False.

Fix the multiprocessing.semaphore_tracker so it is reused by child processes

Fix setup.py check --restructuredtext for files containing include directives.

Remove obsolete comments from docstrings in fractions.Fraction

Document relative imports

Add docstring for io.IOBase.writelines().

New documentation translation: Simplified Chinese.

Avoid the duplication of code from Tools/scripts/serve.py in using the :rst:dir:`literalinclude` directive for the basic wsgiref-based web server in the documentation of :mod:`wsgiref`. Contributed by Stéphane Wirtel.

Using the code of the Tools/scripts/serve.py script as an example in the :mod:`wsgiref` documentation. Contributed by Stéphane Wirtel.

Added documentation for PyInterpreterState_Main().

Updates the docs.python.org page with the addition of a 'Contributing to Docs' link at the end of the page (between 'Reporting Bugs' and 'About Documentation'). Updates the 'Found a Bug' page with additional links and information in the Documentation Bugs section.

@typing.type_check_only now allows type stubs to mark functions and classes not available during runtime.

Add glossary entry for 'magic method'.

Added re.Match.groupdict example to regex HOWTO.

regrtest now always detects uncollectable objects. Previously, the check was only enabled by --findleaks. The check now also works with -jN/--multiprocess N. --findleaks becomes a deprecated alias to --fail-env-changed.

When using multiprocessing mode (-jN), regrtest now better reports errors if a worker process fails, and it exits immediately on a worker thread failure or when interrupted.

Change test_time.test_monotonic() to test only the lower bound of elapsed time after a sleep command rather than the upper bound. This prevents unnecessary test failures on slow buildbots. Patch by Victor Stinner.

Improve test coverage for xml.etree.ElementTree. Patch by Gordon P. Hemsley.

Fix typo in test_cyclic_gc() test for xml.etree.ElementTree. Patch by Gordon P. Hemsley.

Add a new :mod:`!_testinternalcapi` module to test the internal C API.

Fix test_imap4_host_default_value() of test_imaplib: catch also :const:`errno.ENETUNREACH` error.

Fix test_sys.test_getallocatedblocks() when :mod:`tracemalloc` is enabled.

Fix reference leak hunting in regrtest: compute also deltas (of reference count, allocated memory blocks, file descriptor count) during warmup, to ensure that everything is initialized before starting to hunt reference leaks.

Fix reference hunting (python3 -m test -R 3:3) when Python has no built-in abc module.

Port test_resource to VxWorks: skip tests cases setting RLIMIT_FSIZE and RLIMIT_CPU.

Fix test_tabnanny on VxWorks: adjust ENOENT error message.

Fix _testcapi.pymem_buffer_overflow(): handle memory allocation failure.

Fix test_utf8_mode on VxWorks: Python always use UTF-8 on VxWorks.

Fix tests that may fail with PermissionError upon calling bind() on AF_UNIX sockets.

Remove the stale scriptsinstall Makefile target.

On Unix, C extensions are no longer linked to libpython except on Android and Cygwin.

It is now possible for a statically linked Python to load a C extension built using a shared library Python.

When Python is embedded, libpython must not be loaded with RTLD_LOCAL, but RTLD_GLOBAL instead. Previously, using RTLD_LOCAL, it was already not possible to load C extensions which were not linked to libpython, such as C extensions of the standard library built by the *shared* section of Modules/Setup.

distutils, python-config and python-config.py have been modified.

./configure --with-pymalloc no longer adds the m flag to SOABI (sys.implementation.cache_tag). Enabling or disabling pymalloc has no impact on the ABI.

Change PyAPI_FUNC(type), PyAPI_DATA(type) and PyMODINIT_FUNC macros of pyport.h when Py_BUILD_CORE_MODULE is defined. The Py_BUILD_CORE_MODULE define must be now be used to build a C extension as a dynamic library accessing Python internals: export the :samp:`PyInit_{xxx}()` function in DLL exports on Windows.

Don't build the _crypt extension on VxWorks.

Add -fmax-type-align=8 to CFLAGS when clang compiler is detected. The pymalloc memory allocator aligns memory on 8 bytes. On x86-64, clang expects alignment on 16 bytes by default and so uses MOVAPS instruction which can lead to segmentation fault. Instruct clang that Python is limited to alignment on 8 bytes to use MOVUPS instruction instead: slower but don't trigger a SIGSEGV if the memory is not aligned on 16 bytes. Sadly, the flag must be added to CFLAGS and not just CFLAGS_NODIST, since third party C extensions can have the same issue.

make tags and make TAGS now also parse Modules/_io/*.c and Modules/_io/*.h.

Release builds and debug builds are now ABI compatible: defining the Py_DEBUG macro no longer implies the Py_TRACE_REFS macro, which introduces the only ABI incompatibility. The Py_TRACE_REFS macro, which adds the :func:`sys.getobjects` function and the :envvar:`PYTHONDUMPREFS` environment variable, can be set using the new ./configure --with-trace-refs build option.

setup.py now correctly reports missing OpenSSL headers and libraries again.

Fix regression introduced in bpo-36146 refactoring setup.py

python-config --ldflags no longer includes flags of the LINKFORSHARED variable. The LINKFORSHARED variable must only be used to build executables.

Remove references to "aix3" and "aix4". Patch by M. Felt.

Added platform.win32_edition() and platform.win32_is_iot(). Added support for cross-compiling packages for Windows ARM32. Skip tests that are not expected to work on Windows IoT Core ARM32.

Remove trailing spaces for registry keys when installed via the Store.

Fixed activate.bat to correctly update codepage when chcp.com returns dots in output. Patch by Lorenz Mende.

Added preset-iot layout for Windows IoT ARM containers. This layout doesn't contain UI components like tkinter or IDLE. It also doesn't contain files to support on-target builds since Windows ARM32 builds must be cross-compiled when using MSVC.

enum_certificates function of the ssl module now returns certificates from all available certificate stores inside windows in a query instead of returning only certificates from the system wide certificate store. This includes certificates from these certificate stores: local machine, local machine enterprise, local machine group policy, current user, current user group policy, services, users. ssl.enum_crls() function is changed in the same way to return all certificate revocation lists inside the windows certificate revocation list stores.

Fixes creating a venv when debug binaries are installed.

Enable better DLL resolution on Windows by using safe DLL search paths and adding :func:`os.add_dll_directory`.

Add the venv standard library module to the nuget distribution for Windows.

Add the following socket module constants on Windows: IPPROTO_AH IPPROTO_CBT IPPROTO_DSTOPTS IPPROTO_EGP IPPROTO_ESP IPPROTO_FRAGMENT IPPROTO_GGP IPPROTO_HOPOPTS IPPROTO_ICLFXBM IPPROTO_ICMPV6 IPPROTO_IDP IPPROTO_IGMP IPPROTO_IGP IPPROTO_IPV4 IPPROTO_IPV6 IPPROTO_L2TP IPPROTO_MAX IPPROTO_ND IPPROTO_NONE IPPROTO_PGM IPPROTO_PIM IPPROTO_PUP IPPROTO_RDP IPPROTO_ROUTING IPPROTO_SCTP IPPROTO_ST

Added current version of libffi to cpython-source-deps. Change _ctypes to use current version of libffi on Windows.

Report system load when running test suite on Windows. Patch by Ammar Askar. Based on prior work by Jeremy Kloth.

With the Windows 10 Creators Update, non-elevated users can now create symlinks as long as the computer has Developer Mode enabled.

Avoid failures setting macOS stack resource limit with resource.setrlimit. This reverts an earlier fix for bpo-18075 which forced a non-default stack size when building the interpreter executable on macOS.

Fix starting IDLE with pyshell. Add idlelib.pyshell alias at top; remove pyshell alias at bottom. Remove obsolete __name__=='__main__' command.

Fix the argument handling in Tools/scripts/lll.py.

Fix memory leak in :c:func:`!Py_SetStandardStreamEncoding`: release memory if the function is called twice.

:c:expr:`PyDoc_VAR(name)` and :c:expr:`PyDoc_STRVAR(name,str)` now create static const char name[] instead of static char name[]. Patch by Inada Naoki.

Change the value of CLEANBYTE, DEADDYTE and FORBIDDENBYTE internal constants used by debug hooks on Python memory allocators (:c:func:`PyMem_SetupDebugHooks` function). Byte patterns 0xCB, 0xDB and 0xFB have been replaced with 0xCD, 0xDD and 0xFD to use the same values than Windows CRT debug malloc() and free().

Since Python 3.7.0, calling :c:func:`Py_DecodeLocale` before :c:func:`Py_Initialize` produces mojibake if the LC_CTYPE locale is coerced and/or if the UTF-8 Mode is enabled by the user configuration. The LC_CTYPE coercion and UTF-8 Mode are now disabled by default to fix the mojibake issue. They must now be enabled explicitly (opt-in) using the new :c:func:`!_Py_PreInitialize` API with _PyPreConfig.

Fixed an accidental change to the datetime C API where the arguments to the :c:func:`PyDate_FromTimestamp` function were incorrectly interpreted as a single timestamp rather than an arguments tuple, which causes existing code to start raising :exc:`TypeError`. The backwards-incompatible change was only present in alpha releases of Python 3.8. Patch by Paul Ganssle.

Modify PyObject_Init to correctly increase the refcount of heap-allocated Type objects. Also fix the refcounts of the heap-allocated types that were either doing this manually or not decreasing the type's refcount in tp_dealloc