Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run SSL tests on each CI run #97923

Closed
sobolevn opened this issue Oct 5, 2022 · 6 comments
Closed

Run SSL tests on each CI run #97923

sobolevn opened this issue Oct 5, 2022 · 6 comments
Labels
tests Tests in the Lib/test dir topic-SSL type-bug An unexpected behavior, bug, or error

Comments

@sobolevn
Copy link
Member

sobolevn commented Oct 5, 2022

Idea: #93369 (comment)

Recent PR broke SSL tests: ce8fc18

It is not the first when this happens.
The idea is to run SSL tests on each pull request, instead of the current scheme, when tests are run for each push event on main, but not in PRs.

I will send a PR with the fix today :)
@sobolevn sobolevn added type-bug An unexpected behavior, bug, or error topic-SSL labels Oct 5, 2022
@AlexWaygood AlexWaygood added the tests Tests in the Lib/test dir label Oct 5, 2022
sobolevn added a commit to sobolevn/cpython that referenced this issue Oct 5, 2022
@sobolevn
pythongh-97923: Always run Ubuntu SSL tests with others in CI
37d4f5e
@bedevere-bot bedevere-bot mentioned this issue Oct 5, 2022
Merged
@vstinner
Copy link
Member

vstinner commented Oct 6, 2022

If I recall correctly, skipping SSL tests was decided to make the workflow faster, and to not waste CI resources.

@ambv
Copy link
Contributor

ambv commented Oct 7, 2022

Let's try it. Missing SSL test failures in PRs is a bigger problem than wasted resources. If the queues get too long we can always revert to what we have now.

ambv pushed a commit that referenced this issue Oct 7, 2022
@sobolevn
gh-97923: Always run Ubuntu SSL tests with others in CI (#97940)
3108fc1
@kumaraditya303
Copy link
Contributor

What's the reasoning behind running the tests with warnings turned into exceptions? What is the benefit of it? It causes a lot of issue when adding any warning in asyncio. No other tests are run with -Werror, IMO it should be removed.

cc @gvanrossum

@sobolevn
Copy link
Member Author

sobolevn commented Oct 8, 2022

СС @tiran

@arhadthedev
Copy link
Member

It causes a lot of issue when adding any warning in asyncio. No other tests are run with -Werror, IMO it should be removed.

In my case, a deprecation warning revealed that I accidentally broke a test in such a way that it gave no exception:

DeprecationWarning: It is deprecated to return a value!=None from a test case (<bound method StreamTests2.test_eof_feed_when_closing_writer of <test.test_asyncio.test_streams.StreamTests2 testMethod=test_eof_feed_when_closing_writer>>)

So we probably should do the opposite: investigate and fix all flukes then enable -Werror for all tests.

carljm added a commit to carljm/cpython that referenced this issue Oct 8, 2022
@carljm
Merge branch 'main' into typewatch
27a4980 
* main: (38 commits)
  pythongh-92886: make test_ast pass with -O (assertions off) (pythonGH-98058)
  pythongh-92886: make test_coroutines pass with -O (assertions off) (pythonGH-98060)
  pythongh-57179: Add note on symlinks for os.walk (python#94799)
  pythongh-94808: Fix regex on exotic platforms (python#98036)
  pythongh-90085: Remove vestigial -t and -c timeit options (python#94941)
  pythonGH-83901: Improve Signature.bind error message for missing keyword-only params (python#95347)
  pythongh-61105: Add default param, note on using cookiejar subclass (python#95427)
  pythongh-96288: Add a sentence to `os.mkdir`'s docstring. (python#96271)
  pythongh-96073: fix backticks in NEWS entry (pythonGH-98056)
  pythongh-92886: [clinic.py] raise exception on invalid input instead of assertion (pythonGH-98051)
  pythongh-97997: Add col_offset field to tokenizer and use that for AST nodes (python#98000)
  pythonGH-88968: Reject socket that is already used as a transport (python#98010)
  pythongh-96346: Use double caching for re._compile() (python#96347)
  pythongh-91708: Revert params note in urllib.parse.urlparse table (python#96699)
  pythongh-96265: Fix some formatting in faq/design.rst (python#96924)
  pythongh-73196: Add namespace/scope clarification for inheritance section (python#92840)
  pythongh-97646: Change `.js` and `.mjs` files mimetype to conform to RFC 9239 (python#97934)
  pythongh-97923: Always run Ubuntu SSL tests with others in CI (python#97940)
  pythongh-97956: Mention `generate_global_objects.py` in `AC How-To` (python#97957)
  pythongh-96959: Update HTTP links which are redirected to HTTPS (python#98039)
  ...
mpage pushed a commit to mpage/cpython that referenced this issue Oct 11, 2022
@sobolevn @mpage
pythongh-97923: Always run Ubuntu SSL tests with others in CI (python…
dc2c10e 
…#97940)
@kumaraditya303
Copy link
Contributor

Fixed by #97940

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
tests Tests in the Lib/test dir topic-SSL type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@ambv @vstinner @sobolevn @arhadthedev @kumaraditya303 @AlexWaygood