Replaced deprecated pkg_resources.packaging with packaging module #113023

vfdev-5 · 2023-11-06T13:06:19Z

Usage of from pkg_resources import packaging leads to a deprecation warning:

DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html

and in strict tests where warnings are errors, this leads to CI breaks, e.g.: pytorch/vision#8092

Replacing pkg_resources.package with package as it is now a pytorch dependency:

pytorch/requirements.txt

Line 19 in fa9045a

packaging

pytorch-bot · 2023-11-06T13:06:22Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/113023

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit c7f71ce with merge base fa9045a ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

vfdev-5 · 2023-11-06T17:54:14Z

@pytorchbot merge

pytorchmergebot · 2023-11-06T17:56:11Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…torch#113023) Usage of `from pkg_resources import packaging` leads to a deprecation warning: ``` DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html ``` and in strict tests where warnings are errors, this leads to CI breaks, e.g.: pytorch/vision#8092 Replacing `pkg_resources.package` with `package` as it is now a pytorch dependency: https://github.com/pytorch/pytorch/blob/fa9045a8725214c05ae4dcec5a855820b861155e/requirements.txt#L19 Pull Request resolved: pytorch#113023 Approved by: https://github.com/Skylion007

vfdev-5 · 2023-11-07T10:55:36Z

This PR leads to a break as installing nightly pytorch does not install somehow packaging:
https://github.com/pytorch/vision/actions/runs/6783326793/job/18437544348?pr=8092

pip install --progress-bar=off --pre torch --index-url=https://download.pytorch.org/whl/nightly/cpu
...
Installing collected packages: mpmath, typing-extensions, sympy, networkx, MarkupSafe, fsspec, filelock, jinja2, torch
  Successfully installed MarkupSafe-2.1.2 filelock-3.9.0 fsspec-2023.4.0 jinja2-3.1.2 mpmath-1.2.1 networkx-3.0rc1 sympy-1.11.1 torch-2.2.0.dev20231107+cpu typing-extensions-4.4.0

...
Install third party dependencies prior to TorchVision install
  + python setup.py egg_info
  /opt/conda/envs/ci/lib/python3.8/site-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: numpy.core.multiarray failed to import (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
    device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),
  Traceback (most recent call last):
  setup.py:10: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import DistributionNotFound, get_distribution, parse_version
  Traceback (most recent call last):
    File "setup.py", line 12, in <module>
      from torch.utils.cpp_extension import BuildExtension, CppExtension, CUDA_HOME, CUDAExtension
    File "/opt/conda/envs/ci/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 28, in <module>
      import packaging.version
  ModuleNotFoundError: No module named 'packaging.version'

pmeier · 2023-11-07T11:34:06Z

packaging is not installed, since it is not a dependency of torch.

pmeier · 2023-11-07T11:37:06Z

Forward fix in #113154.

mthrok · 2023-11-08T17:09:50Z

@vfdev-5 Can we fallback to the old import with try expcet ?

vfdev-5 · 2023-11-08T17:13:19Z

@mthrok OK, let's try that as another workaround

atalman · 2023-11-08T21:21:27Z

@vfdev-5 should we revert the PR and land it with all forward fixes when there is a workaround ?

vfdev-5 · 2023-11-08T21:27:55Z

@atalman how about #113154 ? If this is a way to go, then yes, let's revert this PR and reland with suggested workarounds

atalman · 2023-11-08T21:34:17Z

@vfdev-5 Nightlies have been broken already for 2 days. The forward fix is not landed yet, it may or may not fix the nightly issue. I suggest we revert this PR and then land it together with forward fixes and some domain build log demonstrating that it does not breaks the domains.

vfdev-5 · 2023-11-08T21:35:24Z

OK, sounds good

atalman · 2023-11-08T21:38:09Z

@pytorchbot revert -m "breaks nightlies" -c nosignal

pytorchmergebot · 2023-11-08T21:39:54Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2023-11-08T21:40:03Z

@vfdev-5 your PR has been successfully reverted.

…dule (#113023)" This reverts commit 81ea7a4. Reverted #113023 on behalf of https://github.com/atalman due to breaks nightlies ([comment](#113023 (comment)))

vfdev-5 · 2023-11-09T12:55:38Z

torch/utils/cpp_extension.py

-    extra_cflags_str = listToString(extra_cflags)
-    extra_include_paths_str = " ".join([f'-I{include}' for include in extra_include_paths])
+    extra_cflags_str = listToString(extra_cflags)    
+    extra_include_paths_str = (


This is a drive-by fix of the failing test:

test/test_cpp_extensions_jit.py::TestCppExtensionJIT::test_gen_extension_h_pch

when extra_include_paths = None. Error message:

File "/pytorch/torch/utils/cpp_extension.py", line 1595, in load_inline _check_and_build_extension_h_precompiler_headers(extra_cflags, extra_include_paths) File "/pytorch/torch/utils/cpp_extension.py", line 1452, in _check_and_build_extension_h_precompiler_headers extra_include_paths_str = " ".join([f'-I{include}' for include in extra_include_paths]) TypeError: 'NoneType' object is not iterable

malfet · 2023-11-10T15:01:28Z

torch/utils/cpp_extension.py

+    extra_include_paths_str = (
+        "" if extra_include_paths is None else " ".join([f"-I{include}" for include in extra_include_paths])
+    )


Nit (this looks more compact, as " ".join([]) returns an empty string)

Suggested change

extra_include_paths_str = (

"" if extra_include_paths is None else " ".join([f"-I{include}" for include in extra_include_paths])

)

extra_include_paths_str = " ".join(

[f"-I{include}" for include in extra_include_paths] if extra_include_paths else []

)

Ok, as CI is green, let's land it as is, will submit tiny BE fixes later

malfet · 2023-11-10T15:02:22Z

torch/utils/cpp_extension.py

+        required_cuda_version = '11.0'
+        if torch.version.cuda is not None and TorchVersion(torch.version.cuda) >= required_cuda_version:


TODO (just remove this one, perhaps in followup PR)

malfet · 2023-11-10T15:03:33Z

@pytorchbot merge

pytorchmergebot · 2023-11-10T15:05:27Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…torch#113023) Usage of `from pkg_resources import packaging` leads to a deprecation warning: ``` DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html ``` and in strict tests where warnings are errors, this leads to CI breaks, e.g.: pytorch/vision#8092 Replacing `pkg_resources.package` with `package` as it is now a pytorch dependency: https://github.com/pytorch/pytorch/blob/fa9045a8725214c05ae4dcec5a855820b861155e/requirements.txt#L19 Pull Request resolved: pytorch#113023 Approved by: https://github.com/Skylion007, https://github.com/malfet

vfdev-5 · 2023-11-13T10:06:22Z

@pytorchbot revert -m "breaks cuda ext build" -c nosignal

pytorchmergebot · 2023-11-13T10:08:25Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2023-11-13T10:08:30Z

Reverting PR 113023 failed

Reason: Command git -C /home/runner/work/pytorch/pytorch revert --no-edit 71ca42787f8a18e0e49714d45cab45e961700b8e returned non-zero exit code 1

Auto-merging torch/utils/cpp_extension.py
CONFLICT (content): Merge conflict in torch/utils/cpp_extension.py
error: could not revert 71ca42787f8... Replaced deprecated pkg_resources.packaging with packaging module (#113023)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git revert --continue".
hint: You can instead skip this commit with "git revert --skip".
hint: To abort and get back to the state before "git revert",
hint: run "git revert --abort".

Details for Dev Infra team

Raised by workflow job

…torch#113023) Usage of `from pkg_resources import packaging` leads to a deprecation warning: ``` DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html ``` and in strict tests where warnings are errors, this leads to CI breaks, e.g.: pytorch/vision#8092 Replacing `pkg_resources.package` with `package` as it is now a pytorch dependency: https://github.com/pytorch/pytorch/blob/fa9045a8725214c05ae4dcec5a855820b861155e/requirements.txt#L19 Pull Request resolved: pytorch#113023 Approved by: https://github.com/Skylion007

…dule (pytorch#113023)" This reverts commit 81ea7a4. Reverted pytorch#113023 on behalf of https://github.com/atalman due to breaks nightlies ([comment](pytorch#113023 (comment)))

…torch#113023) Usage of `from pkg_resources import packaging` leads to a deprecation warning: ``` DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html ``` and in strict tests where warnings are errors, this leads to CI breaks, e.g.: pytorch/vision#8092 Replacing `pkg_resources.package` with `package` as it is now a pytorch dependency: https://github.com/pytorch/pytorch/blob/fa9045a8725214c05ae4dcec5a855820b861155e/requirements.txt#L19 Pull Request resolved: pytorch#113023 Approved by: https://github.com/Skylion007, https://github.com/malfet

Replaced pkg_resources.packaging with packaging module

7469e38

vfdev-5 requested review from fmassa, soumith and ezyang as code owners November 6, 2023 13:06

vfdev-5 added the topic: not user facing topic category label Nov 6, 2023

pytorchbot added the open source label Nov 6, 2023

Fixed failing version import

ea4185c

Skylion007 approved these changes Nov 6, 2023

View reviewed changes

Fixed mypy errors

5137673

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 6, 2023

pytorchmergebot added the merging label Nov 6, 2023

pytorchmergebot added Merged and removed merging labels Nov 6, 2023

pytorchmergebot closed this in 81ea7a4 Nov 6, 2023

pmeier mentioned this pull request Nov 7, 2023

add packaging to the runtime requirements #113154

Closed

pytorchmergebot added the Reverted label Nov 8, 2023

pytorchmergebot reopened this Nov 8, 2023

vfdev-5 added 2 commits November 9, 2023 12:14

Use torch_version.Version instead of packaging

eac9429

Use TorchVersion instead of Version

c95820c

vfdev-5 commented Nov 9, 2023

View reviewed changes

Update cpp_extension.py

c7f71ce

vfdev-5 requested review from malfet and atalman November 10, 2023 09:16

malfet approved these changes Nov 10, 2023

View reviewed changes

pytorchmergebot added the merging label Nov 10, 2023

pytorchmergebot closed this in 71ca427 Nov 10, 2023

pytorchmergebot removed the merging label Nov 10, 2023

vfdev-5 mentioned this pull request Nov 13, 2023

Fixed error with cuda_ver in cpp_extension.py #113555

Closed

vfdev-5 deleted the replace_pkg_resources branch November 13, 2023 12:26

pmeier mentioned this pull request Nov 14, 2023

install packaging as part of the regular dev requirements pytorch/vision#8115

Closed

pmeier mentioned this pull request Nov 17, 2023

torch.compile introduces runtime dependency on setuptools or packaging #113940

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replaced deprecated pkg_resources.packaging with packaging module #113023

Replaced deprecated pkg_resources.packaging with packaging module #113023

vfdev-5 commented Nov 6, 2023

pytorch-bot bot commented Nov 6, 2023 •

edited

Loading

vfdev-5 commented Nov 6, 2023

pytorchmergebot commented Nov 6, 2023

vfdev-5 commented Nov 7, 2023 •

edited

Loading

pmeier commented Nov 7, 2023

pmeier commented Nov 7, 2023

mthrok commented Nov 8, 2023

vfdev-5 commented Nov 8, 2023 •

edited

Loading

atalman commented Nov 8, 2023

vfdev-5 commented Nov 8, 2023

atalman commented Nov 8, 2023

vfdev-5 commented Nov 8, 2023

atalman commented Nov 8, 2023

pytorchmergebot commented Nov 8, 2023

pytorchmergebot commented Nov 8, 2023

vfdev-5 Nov 9, 2023

malfet Nov 10, 2023

malfet Nov 10, 2023

malfet Nov 10, 2023

malfet commented Nov 10, 2023

pytorchmergebot commented Nov 10, 2023

vfdev-5 commented Nov 13, 2023

pytorchmergebot commented Nov 13, 2023

pytorchmergebot commented Nov 13, 2023

		required_cuda_version = '11.0'
		if torch.version.cuda is not None and TorchVersion(torch.version.cuda) >= required_cuda_version:

Replaced deprecated pkg_resources.packaging with packaging module #113023

Replaced deprecated pkg_resources.packaging with packaging module #113023

Conversation

vfdev-5 commented Nov 6, 2023

pytorch-bot bot commented Nov 6, 2023 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/113023

✅ No Failures

vfdev-5 commented Nov 6, 2023

pytorchmergebot commented Nov 6, 2023

Merge started

vfdev-5 commented Nov 7, 2023 • edited Loading

pmeier commented Nov 7, 2023

pmeier commented Nov 7, 2023

mthrok commented Nov 8, 2023

vfdev-5 commented Nov 8, 2023 • edited Loading

atalman commented Nov 8, 2023

vfdev-5 commented Nov 8, 2023

atalman commented Nov 8, 2023

vfdev-5 commented Nov 8, 2023

atalman commented Nov 8, 2023

pytorchmergebot commented Nov 8, 2023

pytorchmergebot commented Nov 8, 2023

vfdev-5 Nov 9, 2023

Choose a reason for hiding this comment

malfet Nov 10, 2023

Choose a reason for hiding this comment

malfet Nov 10, 2023

Choose a reason for hiding this comment

malfet Nov 10, 2023

Choose a reason for hiding this comment

malfet commented Nov 10, 2023

pytorchmergebot commented Nov 10, 2023

Merge started

vfdev-5 commented Nov 13, 2023

pytorchmergebot commented Nov 13, 2023

pytorchmergebot commented Nov 13, 2023

Reverting PR 113023 failed

pytorch-bot bot commented Nov 6, 2023 •

edited

Loading

vfdev-5 commented Nov 7, 2023 •

edited

Loading

vfdev-5 commented Nov 8, 2023 •

edited

Loading