Merge 3.6 (issue python#28091)

Doc/glossary.rst

@@ -74,6 +74,34 @@ Glossary
       :keyword:`async with` statement by defining :meth:`__aenter__` and
       :meth:`__aexit__` methods.  Introduced by :pep:`492`.
 
+   asynchronous generator
+      A function which returns an :term:`asynchronous generator iterator`.  It
+      looks like a coroutine function defined with :keyword:`async def` except
+      that it contains :keyword:`yield` expressions for producing a series of
+      values usable in an :keyword:`async for` loop.
+
+      Usually refers to a asynchronous generator function, but may refer to an
+      *asynchronous generator iterator* in some contexts.  In cases where the
+      intended meaning isn't clear, using the full terms avoids ambiguity.
+
+      An asynchronous generator function may contain :keyword:`await`
+      expressions as well as :keyword:`async for`, and :keyword:`async with`
+      statements.
+
+   asynchronous generator iterator
+      An object created by a :term:`asynchronous generator` function.
+
+      This is an :term:`asynchronous iterator` which when called using the
+      :meth:`__anext__` method returns an awaitable object which will execute
+      that the body of the asynchronous generator function until the
+      next :keyword:`yield` expression.
+
+      Each :keyword:`yield` temporarily suspends processing, remembering the
+      location execution state (including local variables and pending
+      try-statements).  When the *asynchronous generator iterator* effectively
+      resumes with another awaitable returned by :meth:`__anext__`, it
+      picks-up where it left-off.  See :pep:`492` and :pep:`525`.
+
    asynchronous iterable
       An object, that can be used in an :keyword:`async for` statement.
       Must return an :term:`asynchronous iterator` from its

Doc/library/asyncio-eventloop.rst

@@ -88,6 +88,24 @@ Run an event loop
    This is idempotent and irreversible. No other methods should be called after
    this one.
 
+
+.. coroutinemethod:: AbstractEventLoop.shutdown_asyncgens()
+
+   Schedule all currently open :term:`asynchronous generator` objects to
+   close with an :meth:`~agen.aclose()` call.  After calling this method,
+   the event loop will issue a warning whenever a new asynchronous generator
+   is iterated.  Should be used to finalize all scheduled asynchronous
+   generators reliably.  Example::
+
+    try:
+        loop.run_forever()
+    finally:
+        loop.run_until_complete(loop.shutdown_asyncgens())
+        loop.close()
+
+   .. versionadded:: 3.6
+
+
 .. _asyncio-pass-keywords:
 
 Calls

Doc/library/inspect.rst

@@ -318,6 +318,27 @@ attributes:
    .. versionadded:: 3.5
 
 
+.. function:: isasyncgenfunction(object)
+
+   Return true if the object is an :term:`asynchronous generator` function,
+   for example::
+
+    >>> async def agen():
+    ...     yield 1
+    ...
+    >>> inspect.isasyncgenfunction(agen)
+    True
+
+   .. versionadded:: 3.6
+
+
+.. function:: isasyncgen(object)
+
+   Return true if the object is an :term:`asynchronous generator iterator`
+   created by an :term:`asynchronous generator` function.
+
+   .. versionadded:: 3.6
+
 .. function:: istraceback(object)
 
    Return true if the object is a traceback.

Doc/library/sys.rst

@@ -603,6 +603,24 @@ always available.
    .. versionchanged:: 3.6
       Added *platform_version*
 
+
+.. function:: get_asyncgen_hooks()
+
+   Returns an *asyncgen_hooks* object, which is similar to a
+   :class:`~collections.namedtuple` of the form `(firstiter, finalizer)`,
+   where *firstiter* and *finalizer* are expected to be either ``None`` or
+   functions which take an :term:`asynchronous generator iterator` as an
+   argument, and are used to schedule finalization of an asychronous
+   generator by an event loop.
+
+   .. versionadded:: 3.6
+      See :pep:`525` for more details.
+
+   .. note::
+      This function has been added on a provisional basis (see :pep:`411`
+      for details.)
+
+
 .. function:: get_coroutine_wrapper()
 
    Returns ``None``, or a wrapper set by :func:`set_coroutine_wrapper`.
@@ -1107,6 +1125,24 @@ always available.
       implementation platform, rather than part of the language definition, and
       thus may not be available in all Python implementations.
 
+.. function:: set_asyncgen_hooks(firstiter, finalizer)
+
+   Accepts two optional keyword arguments which are callables that accept an
+   :term:`asynchronous generator iterator` as an argument. The *firstiter*
+   callable will be called when an asynchronous generator is iterated for the
+   first time. The *finalizer* will be called when an asynchronous generator
+   is about to be garbage collected.
+
+   .. versionadded:: 3.6
+      See :pep:`525` for more details, and for a reference example of a
+      *finalizer* method see the implementation of
+      ``asyncio.Loop.shutdown_asyncgens`` in
+      :source:`Lib/asyncio/base_events.py`
+
+   .. note::
+      This function has been added on a provisional basis (see :pep:`411`
+      for details.)
+
 
 .. function:: set_coroutine_wrapper(wrapper)
 

Doc/library/types.rst

@@ -104,6 +104,14 @@ Standard names are defined for the following types:
    .. versionadded:: 3.5
 
 
+.. data:: AsyncGeneratorType
+
+   The type of :term:`asynchronous generator`-iterator objects, created by
+   asynchronous generator functions.
+
+   .. versionadded:: 3.6
+
+
 .. data:: CodeType
 
    .. index:: builtin: compile

Doc/reference/compound_stmts.rst

@@ -697,7 +697,7 @@ coroutine bodies.
 Functions defined with ``async def`` syntax are always coroutine functions,
 even if they do not contain ``await`` or ``async`` keywords.
 
-It is a :exc:`SyntaxError` to use :keyword:`yield` expressions in
+It is a :exc:`SyntaxError` to use ``yield from`` expressions in
 ``async def`` coroutines.
 
 An example of a coroutine function::

Doc/reference/datamodel.rst

@@ -627,6 +627,25 @@ Callable types
       as well as :keyword:`async with` and :keyword:`async for` statements. See
       also the :ref:`coroutine-objects` section.
 
+   Asynchronous generator functions
+      .. index::
+         single: asynchronous generator; function
+         single: asynchronous generator; asynchronous iterator
+
+      A function or method which is defined using :keyword:`async def` and
+      which uses the :keyword:`yield` statement is called a
+      :dfn:`asynchronous generator function`.  Such a function, when called,
+      returns an asynchronous iterator object which can be used in an
+      :keyword:`async for` statement to execute the body of the function.
+
+      Calling the asynchronous iterator's :meth:`aiterator.__anext__` method
+      will return an :term:`awaitable` which when awaited
+      will execute until it provides a value using the :keyword:`yield`
+      expression.  When the function executes an empty :keyword:`return`
+      statement or falls off the end, a :exc:`StopAsyncIteration` exception
+      is raised and the asynchronous iterator will have reached the end of
+      the set of values to be yielded.
+
    Built-in functions
       .. index::
          object: built-in function

Doc/reference/expressions.rst

@@ -172,7 +172,7 @@ Common syntax elements for comprehensions are:
 
 .. productionlist::
    comprehension: `expression` `comp_for`
-   comp_for: "for" `target_list` "in" `or_test` [`comp_iter`]
+   comp_for: [ASYNC] "for" `target_list` "in" `or_test` [`comp_iter`]
    comp_iter: `comp_for` | `comp_if`
    comp_if: "if" `expression_nocond` [`comp_iter`]
 
@@ -186,6 +186,17 @@ each time the innermost block is reached.
 Note that the comprehension is executed in a separate scope, so names assigned
 to in the target list don't "leak" into the enclosing scope.
 
+Since Python 3.6, in an :keyword:`async def` function, an :keyword:`async for`
+clause may be used to iterate over a :term:`asynchronous iterator`.
+A comprehension in an :keyword:`async def` function may consist of either a
+:keyword:`for` or :keyword:`async for` clause following the leading
+expression, may contan additonal :keyword:`for` or :keyword:`async for`
+clauses, and may also use :keyword:`await` expressions.
+If a comprehension contains either :keyword:`async for` clauses
+or :keyword:`await` expressions it is called an
+:dfn:`asynchronous comprehension`.  An asynchronous comprehension may
+suspend the execution of the coroutine function in which it appears.
+See also :pep:`530`.
 
 .. _lists:
 
@@ -315,6 +326,14 @@ range(10) for y in bar(x))``.
 The parentheses can be omitted on calls with only one argument.  See section
 :ref:`calls` for details.
 
+Since Python 3.6, if the generator appears in an :keyword:`async def` function,
+then :keyword:`async for` clauses and :keyword:`await` expressions are permitted
+as with an asynchronous comprehension.  If a generator expression
+contains either :keyword:`async for` clauses or :keyword:`await` expressions
+it is called an :dfn:`asynchronous generator expression`.
+An asynchronous generator expression yields a new asynchronous
+generator object, which is an asynchronous iterator
+(see :ref:`async-iterators`).
 
 .. _yieldexpr:
 
@@ -330,9 +349,22 @@ Yield expressions
    yield_atom: "(" `yield_expression` ")"
    yield_expression: "yield" [`expression_list` | "from" `expression`]
 
-The yield expression is only used when defining a :term:`generator` function and
+The yield expression is used when defining a :term:`generator` function
+or an :term:`asynchronous generator` function and
 thus can only be used in the body of a function definition.  Using a yield
-expression in a function's body causes that function to be a generator.
+expression in a function's body causes that function to be a generator,
+and using it in an :keyword:`async def` function's body causes that
+coroutine function to be an asynchronous generator. For example::
+
+    def gen():  # defines a generator function
+        yield 123
+
+    async def agen(): # defines an asynchronous generator function (PEP 525)
+        yield 123
+
+Generator functions are described below, while asynchronous generator
+functions are described separately in section
+:ref:`asynchronous-generator-functions`.
 
 When a generator function is called, it returns an iterator known as a
 generator.  That generator then controls the execution of the generator function.
@@ -496,6 +528,134 @@ generator functions::
 For examples using ``yield from``, see :ref:`pep-380` in "What's New in
 Python."
 
+.. _asynchronous-generator-functions:
+
+Asynchronous generator functions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The presence of a yield expression in a function or method defined using
+:keyword:`async def` further defines the function as a
+:term:`asynchronous generator` function.
+
+When an asynchronous generator function is called, it returns an
+asynchronous iterator known as an asynchronous generator object.
+That object then controls the execution of the generator function.
+An asynchronous generator object is typically used in an
+:keyword:`async for` statement in a coroutine function analogously to
+how a generator object would be used in a :keyword:`for` statement.
+
+Calling one of the asynchronous generator's methods returns an
+:term:`awaitable` object, and the execution starts when this object
+is awaited on. At that time, the execution proceeds to the first yield
+expression, where it is suspended again, returning the value of
+:token:`expression_list` to the awaiting coroutine. As with a generator,
+suspension means that all local state is retained, including the
+current bindings of local variables, the instruction pointer, the internal
+evaluation stack, and the state of any exception handling.  When the execution
+is resumed by awaiting on the next object returned by the asynchronous
+generator's methods, the function can proceed exactly as if the yield
+expression were just another external call. The value of the yield expression
+after resuming depends on the method which resumed the execution.  If
+:meth:`~agen.__anext__` is used then the result is :const:`None`. Otherwise, if
+:meth:`~agen.asend` is used, then the result will be the value passed in to
+that method.
+
+In an asynchronous generator function, yield expressions are allowed anywhere
+in a :keyword:`try` construct. However, if an asynchronous generator is not
+resumed before it is finalized (by reaching a zero reference count or by
+being garbage collected), then a yield expression within a :keyword:`try`
+construct could result in a failure to execute pending :keyword:`finally`
+clauses.  In this case, it is the responsibility of the event loop or
+scheduler running the asynchronous generator to call the asynchronous
+generator-iterator's :meth:`~agen.aclose` method and run the resulting
+coroutine object, thus allowing any pending :keyword:`finally` clauses
+to execute.
+
+To take care of finalization, an event loop should define
+a *finalizer* function which takes an asynchronous generator-iterator
+and presumably calls :meth:`~agen.aclose` and executes the coroutine.
+This  *finalizer* may be registered by calling :func:`sys.set_asyncgen_hooks`.
+When first iterated over, an asynchronous generator-iterator will store the
+registered *finalizer* to be called upon finalization. For a reference example
+of a *finalizer* method see the implementation of
+``asyncio.Loop.shutdown_asyncgens`` in :source:`Lib/asyncio/base_events.py`.
+
+The expression ``yield from <expr>`` is a syntax error when used in an
+asynchronous generator function.
+
+.. index:: object: asynchronous-generator
+.. _asynchronous-generator-methods:
+
+Asynchronous generator-iterator methods
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This subsection describes the methods of an asynchronous generator iterator,
+which are used to control the execution of a generator function.
+
+
+.. index:: exception: StopAsyncIteration
+
+.. coroutinemethod:: agen.__anext__()
+
+   Returns an awaitable which when run starts to execute the asynchronous
+   generator or resumes it at the last executed yield expression.  When an
+   asynchronous generator function is resumed with a :meth:`~agen.__anext__`
+   method, the current yield expression always evaluates to :const:`None` in
+   the returned awaitable, which when run will continue to the next yield
+   expression. The value of the :token:`expression_list` of the yield
+   expression is the value of the :exc:`StopIteration` exception raised by
+   the completing coroutine.  If the asynchronous generator exits without
+   yielding another value, the awaitable instead raises an
+   :exc:`StopAsyncIteration` exception, signalling that the asynchronous
+   iteration has completed.
+
+   This method is normally called implicitly by a :keyword:`async for` loop.
+
+
+.. coroutinemethod:: agen.asend(value)
+
+   Returns an awaitable which when run resumes the execution of the
+   asynchronous generator. As with the :meth:`~generator.send()` method for a
+   generator, this "sends" a value into the asynchronous generator function,
+   and the *value* argument becomes the result of the current yield expression.
+   The awaitable returned by the :meth:`asend` method will return the next
+   value yielded by the generator as the value of the raised
+   :exc:`StopIteration`, or raises :exc:`StopAsyncIteration` if the
+   asynchronous generator exits without yielding another value.  When
+   :meth:`asend` is called to start the asynchronous
+   generator, it must be called with :const:`None` as the argument,
+   because there is no yield expression that could receive the value.
+
+
+.. coroutinemethod:: agen.athrow(type[, value[, traceback]])
+
+   Returns an awaitable that raises an exception of type ``type`` at the point
+   where the asynchronous generator was paused, and returns the next value
+   yielded by the generator function as the value of the raised
+   :exc:`StopIteration` exception.  If the asynchronous generator exits
+   without yielding another value, an :exc:`StopAsyncIteration` exception is
+   raised by the awaitable.
+   If the generator function does not catch the passed-in exception, or
+   raises a different exception, then when the awaitalbe is run that exception
+   propagates to the caller of the awaitable.
+
+.. index:: exception: GeneratorExit
+
+
+.. coroutinemethod:: agen.aclose()
+
+   Returns an awaitable that when run will throw a :exc:`GeneratorExit` into
+   the asynchronous generator function at the point where it was paused.
+   If the asynchronous generator function then exits gracefully, is already
+   closed, or raises :exc:`GeneratorExit` (by not catching the exception),
+   then the returned awaitable will raise a :exc:`StopIteration` exception.
+   Any further awaitables returned by subsequent calls to the asynchronous
+   generator will raise a :exc:`StopAsyncIteration` exception.  If the
+   asynchronous generator yields a value, a :exc:`RuntimeError` is raised
+   by the awaitable.  If the asynchronous generator raises any other exception,
+   it is propagated to the caller of the awaitable.  If the asynchronous
+   generator has already exited due to an exception or normal exit, then
+   further calls to :meth:`aclose` will return an awaitable that does nothing.
 
 .. _primaries: