bpo-30406: Make async and await proper keywords #1669

JelleZijlstra · 2017-05-19T21:29:22Z

https://bugs.python.org/issue30406

AraHaan

Looks alright to me.

JelleZijlstra · 2017-05-19T22:01:08Z

I might need some time to fix the test failures.

AraHaan · 2017-05-19T22:04:02Z

Alright, But in general it looks like a nice start.

vstinner · 2017-05-20T05:36:56Z

Lib/lib2to3/tests/test_parser.py

-        self.invalid_syntax("""def foo():
-                                   async for a in b: pass""")
+        self.validate("""def foo():
+                             async for a in b: pass""")


wait... this one looks like a bug.

I think lib2to3's parser in general doesn't do semantic checking of the sort that would be required to reject this. For example, it also doesn't fail on putting nonlocal or yield from outside a function, or yield from inside an async generator.

vstinner · 2017-05-20T05:37:09Z

Lib/lib2to3/tests/test_parser.py

-        self.invalid_syntax("""def foo():
-                                   async with a: pass""")
+        self.validate("""def foo():
+                             async with a: pass""")


ditto (also a bug)

Same here, this is also lib2to3.

1st1 · 2017-05-20T05:39:49Z

Overall looks good. I'll make a detailed review in a few days.

vstinner

https://docs.python.org/dev/reference/grammar.html should also be updated, no?

JelleZijlstra · 2017-05-20T14:03:18Z

grammar.html just directly includes Grammar/Grammar, so it shouldn't require further changes.

AraHaan · 2017-05-20T16:38:43Z

@JelleZijlstra you mind also removing 'asyncio.async' as well as that thing might cause an SyntaxError.

vstinner · 2017-05-20T16:45:36Z

Thanks for the clarification :-) I let Yury review the change. He knows the parser much better than me!

JelleZijlstra · 2017-05-21T15:01:48Z

@AraHaan it's currently implemented as something like globals()['async'] = ensure_future, so the implementation won't cause a SyntaxError (as you can tell from the test suite passing). I suppose I could remove the alias, but I also don't want to make this patch bigger than necessary, so I'll wait for Yury to review it.

AraHaan · 2017-05-21T23:37:11Z

Ok

1st1 · 2017-06-10T18:51:09Z

I'll merge this when I return from my vacation in 11 days. Thank you for keeping the pr up to date!

JelleZijlstra · 2017-07-20T15:25:47Z

@1st1 ping – have you had a chance to look at this PR?

1st1

Overall looks good. Could you please rename it once again and add a news entry with blurb?

1st1 · 2017-07-20T15:54:04Z

s/rename/rebase

JelleZijlstra · 2017-07-21T05:04:49Z

Just rebased from current master and added a news entry using blurb.

JelleZijlstra · 2017-07-21T15:46:11Z

The AppVeyor failure is in test_idle and looks unrelated to this PR.

JelleZijlstra · 2017-07-21T15:47:55Z

Oh, probably related to 7c5798e.

vstinner · 2017-07-21T15:51:31Z

The AppVeyor failure is in test_idle and looks unrelated to this PR.

Rebase again and it should be fine.

1st1

Playing with the code right now. One of the things you also need to do is to cleanup forbidden_name() function in Python/ast.c.

1st1 · 2017-07-21T21:29:51Z

Grammar/Grammar

+comp_for_or_async: comp_async_for | comp_for
+comp_iter: comp_async_for | comp_for | comp_if
+comp_async_for: 'async' 'for' exprlist 'in' or_test [comp_iter]
+comp_for: 'for' exprlist 'in' or_test [comp_iter]
 comp_if: 'if' test_nocond [comp_iter]


Why did you do all these renames and added atom_expr? Why don't we just replace ASYNC with 'async' (ditto for await)? I'd avoid making any changes to the grammar besides token adjustments.

I don't remember exactly (it's been a few months), but I just tried to implement your suggestions and I'm getting failures in test_parser that I don't currently understand. I'll look into it some more. The errors are all on comprehensions and are similar to

File "/Users/jzijlstra-mpbt/py/cpython/Lib/test/test_parser.py", line 24, in roundtrip self.fail("could not roundtrip %r: %s" % (s, why)) AssertionError: could not roundtrip '{x for x in seq}': Illegal terminal: expected NAME, got 330.

1st1 · 2017-07-21T21:33:00Z

Lib/lib2to3/Grammar.txt

+comp_for_or_async: comp_async_for | comp_for
+comp_iter: comp_async_for | comp_for | comp_if
+comp_async_for: 'async' 'for' exprlist 'in' testlist_safe [comp_iter]
+comp_for: 'for' exprlist 'in' testlist_safe [comp_iter]
 comp_if: 'if' old_test [comp_iter]


Adding to the point raised for changes in Grammar: when you change/rename/add new productions here, you might be breaking some code that uses lib2to3 to define custom fixters. Again, I'd avoid any changes.

1st1 · 2017-07-21T21:35:55Z

Lib/test/test_coroutines.py


-    def test_goodsyntax_1(self):
+    def test_badsyntax_4(self):
        # Tests for issue 24619


You can drop this comment now.

1st1 · 2017-07-21T21:36:44Z

Misc/NEWS.d/next/Core and Builtins/2017-07-20-22-03-44.bpo-30406._kr47t.rst

@@ -0,0 +1 @@
+``async`` and ``await`` are now proper keywords, as specified in PEP 492.


Change to:

Make ``async`` and ``await`` proper keywords, as specified in PEP 492.

1st1 · 2017-09-26T19:25:01Z

Ping? I plan to work on this myself if this isn't resolved soon.

JelleZijlstra · 2017-09-26T19:26:44Z

I was unable to resolve the issue I mentioned in #1669 (comment). I can push out the other fixes though and take another look within the next few days to see if I can get it working.

1st1 · 2017-09-26T19:29:53Z

I was unable to resolve the issue I mentioned in #1669 (comment). [..]

Interesting. I'll take a look at that too.

JelleZijlstra · 2017-10-06T02:11:53Z

Just rebased and adjusted the lib2to3 grammar changes to make local tests pass.

@1st1

@1st1 requested this in python#1669 (comment).

ambv · 2018-03-12T20:53:32Z

@JelleZijlstra This change is incorrect in lib2to3. That library's grammar is explicitly kept in a way that is able to parse all code valid for Python 2 - Python 3.7. With your change we lost the ability to read valid Python 2 - Python 3.6 code that uses async and await as names.

1st1 · 2018-03-12T20:58:16Z

Hm, I'm not sure what options we have here. Any suggestions? About the only one that I have is to undo the change for lib2to3 entirely (but keeping the rest of the changes).

ambv · 2018-03-12T21:01:12Z

Yes, this is what I'm inclined to do. Revert the lib2to3 part.

JelleZijlstra · 2018-03-12T21:03:14Z

If I recall correctly I made changes to lib2to3 because some test was failing otherwise, so it might not be as simple as just reverting parts of this PR. I can spend some time looking into this later today.

This reverts commit ac31770. (Reverts only the lib2to3 part.)

JelleZijlstra · 2018-03-18T16:36:00Z

OK, I made that up—tests pass cleanly if I undo my lib2to3 changes. @ambv @1st1 I submitted #6143 to fix the regression.

…" (pythonGH-6143) This reverts commit ac31770. (Reverts only the lib2to3 part.) (cherry picked from commit f64aae4) Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

…6143) This reverts commit ac31770. (Reverts only the lib2to3 part.)

…H-6143) This reverts commit ac31770. (Reverts only the lib2to3 part.) (cherry picked from commit f64aae4) Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

pythonGH-6143) This reverts commit ac31770. (Reverts only the lib2to3 part.)

the-knights-who-say-ni added the CLA signed label May 19, 2017

AraHaan approved these changes May 19, 2017

View reviewed changes

JelleZijlstra changed the title ~~bpo-30406: Make async and await proper keywords~~ bpo-30406: [WIP] Make async and await proper keywords May 19, 2017

JelleZijlstra changed the title ~~bpo-30406: [WIP] Make async and await proper keywords~~ bpo-30406: Make async and await proper keywords May 19, 2017

1st1 self-assigned this May 20, 2017

vstinner requested changes May 20, 2017

View reviewed changes

vstinner requested a review from 1st1 May 20, 2017 05:38

vstinner reviewed May 20, 2017

View reviewed changes

1st1 reviewed Jul 20, 2017

View reviewed changes

JelleZijlstra force-pushed the asyncsyntax branch from f2838a0 to 6562616 Compare July 21, 2017 05:04

JelleZijlstra force-pushed the asyncsyntax branch from 16949e3 to 36ea452 Compare July 21, 2017 15:52

1st1 requested changes Jul 21, 2017

View reviewed changes

JelleZijlstra added 7 commits October 5, 2017 18:54

fix whitespace (make patchcheck)

499aa40

fix test_symbol

7c664e3

use a separate sync_comp_for (suggested by Yury)

5659c8d

regenerate symbol.py

2413e3e

simplify lib2to3 changes

cdbc7d5

remove "again" from wording intoken.rst

38ba810

fix lib2to3

78ddbde

JelleZijlstra force-pushed the asyncsyntax branch from 2ed73d1 to 78ddbde Compare October 6, 2017 02:11

1st1 merged commit ac31770 into python:master Oct 6, 2017

bedevere-bot removed the awaiting merge label Oct 6, 2017

JelleZijlstra added a commit to JelleZijlstra/cpython that referenced this pull request Oct 9, 2017

bpo-31698: add REQ_NAME macro

669d7a0

@1st1 requested this in python#1669 (comment).

JelleZijlstra mentioned this pull request Oct 9, 2017

bpo-31698: add REQ_NAME macro #3926

Closed

pablogsal mentioned this pull request Oct 25, 2017

[3.6] bpo-31852: Fix segfault caused by using the async soft keyword #4122

Merged

llawall mentioned this pull request Feb 19, 2018

Add read the docs configuration wintoncode/winton-kafka-streams#46

Merged

JelleZijlstra deleted the asyncsyntax branch March 12, 2018 21:03

ambv mentioned this pull request Mar 15, 2018

error: cannot format hoge.py: Cannot parse: 1:4: def async(): psf/black#20

Closed

JelleZijlstra added a commit to JelleZijlstra/cpython that referenced this pull request Mar 18, 2018

Revert "bpo-30406: Make async and await proper keywords (python#1669)"

5c138cf

This reverts commit ac31770. (Reverts only the lib2to3 part.)

JelleZijlstra added a commit to JelleZijlstra/cpython that referenced this pull request Mar 18, 2018

Revert "bpo-30406: Make async and await proper keywords (python#1669)"

1f9ebe4

This reverts commit ac31770. (Reverts only the lib2to3 part.)

JelleZijlstra mentioned this pull request Mar 18, 2018

bpo-30406: allow async/await in lib2to3 #6143

Merged

ambv pushed a commit that referenced this pull request Mar 18, 2018

Revert "bpo-30406: Make async and await proper keywords (#1669)" (GH-…

f64aae4

…6143) This reverts commit ac31770. (Reverts only the lib2to3 part.)

jo2y pushed a commit to jo2y/cpython that referenced this pull request Mar 23, 2018

Revert "bpo-30406: Make async and await proper keywords (python#1669)" (

314ac54

pythonGH-6143) This reverts commit ac31770. (Reverts only the lib2to3 part.)

ddfisher mentioned this pull request Jun 30, 2018

Support 3.7 python/typed_ast#60

Closed

		@@ -0,0 +1 @@
		``async`` and ``await`` are now proper keywords, as specified in PEP 492.

bpo-30406: Make async and await proper keywords #1669

bpo-30406: Make async and await proper keywords #1669

Conversation

JelleZijlstra commented May 19, 2017 • edited by bedevere-bot Loading

AraHaan left a comment • edited Loading

Choose a reason for hiding this comment

JelleZijlstra commented May 19, 2017

AraHaan commented May 19, 2017

Choose a reason for hiding this comment

JelleZijlstra May 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

1st1 commented May 20, 2017

vstinner left a comment

Choose a reason for hiding this comment

JelleZijlstra commented May 20, 2017

AraHaan commented May 20, 2017

vstinner commented May 20, 2017 via email

JelleZijlstra commented May 21, 2017

AraHaan commented May 21, 2017

1st1 commented Jun 10, 2017

JelleZijlstra commented Jul 20, 2017

1st1 left a comment

Choose a reason for hiding this comment

1st1 commented Jul 20, 2017

JelleZijlstra commented Jul 21, 2017

JelleZijlstra commented Jul 21, 2017

JelleZijlstra commented Jul 21, 2017

vstinner commented Jul 21, 2017

1st1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

1st1 commented Sep 26, 2017

JelleZijlstra commented Sep 26, 2017

1st1 commented Sep 26, 2017 • edited Loading

JelleZijlstra commented Oct 6, 2017

ambv commented Mar 12, 2018

1st1 commented Mar 12, 2018 • edited Loading

ambv commented Mar 12, 2018

JelleZijlstra commented Mar 12, 2018

JelleZijlstra commented Mar 18, 2018

JelleZijlstra commented May 19, 2017 •

edited by bedevere-bot

Loading

AraHaan left a comment •

edited

Loading

JelleZijlstra May 20, 2017 •

edited

Loading

1st1 commented Sep 26, 2017 •

edited

Loading

1st1 commented Mar 12, 2018 •

edited

Loading