Unable to build and compile v1.8.0 which is required for the Visual Quality Inspection AI reference kit

[tarekradi]

Version 1.8.0 is needed as per https://github.com/oneapi-src/visual-quality-inspection

However, i am unable to compile this version. So i used the dockerfile and tried to build the docker image, but also that failed.

Here is a dump of the last part of docker image creation failure. Please advise. Thanks.

[
#14 334.8 -- ******** Summary ********
#14 334.8 -- General:
#14 334.8 -- CMake version : 3.24.2
#14 334.8 -- CMake command : /opt/conda/lib/python3.8/site-packages/cmake/data/bin/cmake
#14 334.8 -- System : Linux
#14 334.8 -- Target name : torch_ccl
#14 334.8 -- Install path : /intel-extension-for-pytorch/build/lib.linux-x86_64-cpython-38
#14 334.8 -- Build type : Release
#14 334.8 -- C++ compiler : /usr/bin/c++
#14 334.8 -- C++ compiler id : GNU
#14 334.8 -- C++ compiler version : 9.4.0
#14 334.8 -- CXX flags : -D_GLIBCXX_USE_CXX11_ABI=0 -Wformat -Werror=cpp -Wformat-security -fstack-protector
#14 334.8 -- Compile flags :
#14 334.8 -- Compile definitions :
#14 334.8 -- Linker options :
#14 334.8 -- Linker directors : ccl;fabric;mpi;libtorch_python.so
#14 334.8 -- Linker libraries : /intel-extension-for-pytorch/third_party/torch_ccl/third_party/oneCCL/mpi/lib;/intel-extension-for-pytorch/third_party/torch_ccl/third_party/oneCCL/ofi/lib;/opt/conda/lib/python3.8/site-packages/torch/lib
#14 334.8 --
#14 334.8 -- Performing Test C_HAS_AVX512_1
#14 335.1 -- Performing Test C_HAS_AVX512_1 - Failed
#14 335.1 -- Performing Test C_HAS_AVX512_2
#14 335.3 -- Performing Test C_HAS_AVX512_2 - Failed
#14 335.3 -- Performing Test CXX_HAS_AVX512_1
#14 335.5 -- Performing Test CXX_HAS_AVX512_1 - Failed
#14 335.5 -- Performing Test CXX_HAS_AVX512_2
#14 335.7 -- Performing Test CXX_HAS_AVX512_2 - Failed
#14 335.7 -- Performing Test C_HAS_AVX512_BF16_1
#14 336.0 -- Performing Test C_HAS_AVX512_BF16_1 - Failed
#14 336.0 -- Performing Test C_HAS_AVX512_BF16_2
#14 336.0 -- Performing Test C_HAS_AVX512_BF16_2 - Failed
#14 336.0 -- Performing Test CXX_HAS_AVX512_BF16_1
#14 336.2 -- Performing Test CXX_HAS_AVX512_BF16_1 - Failed
#14 336.2 -- Performing Test CXX_HAS_AVX512_BF16_2
#14 336.3 -- Performing Test CXX_HAS_AVX512_BF16_2 - Failed
#14 336.3 CMake Error at cmake/CPU.cmake:21 (message):
#14 336.3 Please build IPEX on Machines that support AVX512.
#14 336.3 Call Stack (most recent call first):
#14 336.3 CMakeLists.txt:25 (include)
#14 336.3
#14 336.3
#14 336.3 -- Configuring incomplete, errors occurred!
#14 336.3 See also "/intel-extension-for-pytorch/build/CMakeFiles/CMakeOutput.log".
#14 336.3 See also "/intel-extension-for-pytorch/build/CMakeFiles/CMakeError.log".
#14 336.3 Traceback (most recent call last):
#14 336.3 File "", line 2, in
#14 336.3 File "", line 34, in
#14 336.3 File "/intel-extension-for-pytorch/setup.py", line 370, in
#14 336.3 setup(
#14 336.3 File "/opt/conda/lib/python3.8/site-packages/setuptools/init.py", line 87, in setup
#14 336.3 return distutils.core.setup(**attrs)
#14 336.3 File "/opt/conda/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
#14 336.3 return run_commands(dist)
#14 336.3 File "/opt/conda/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
#14 336.3 dist.run_commands()
#14 336.3 File "/opt/conda/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
#14 336.3 self.run_command(cmd)
#14 336.3 File "/opt/conda/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
#14 336.3 super().run_command(command)
#14 336.3 File "/opt/conda/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
#14 336.3 cmd_obj.run()
#14 336.3 File "/opt/conda/lib/python3.8/site-packages/setuptools/command/install.py", line 68, in run
#14 336.3 return orig.install.run(self)
#14 336.3 File "/opt/conda/lib/python3.8/site-packages/setuptools/_distutils/command/install.py", line 698, in run
#14 336.3 self.run_command('build')
#14 336.3 File "/opt/conda/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
#14 336.3 self.distribution.run_command(command)
#14 336.3 File "/opt/conda/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
#14 336.3 super().run_command(command)
#14 336.3 File "/opt/conda/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
#14 336.3 cmd_obj.run()
#14 336.3 File "/opt/conda/lib/python3.8/site-packages/setuptools/_distutils/command/build.py", line 132, in run
#14 336.3 self.run_command(cmd_name)
#14 336.3 File "/opt/conda/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
#14 336.3 self.distribution.run_command(command)
#14 336.3 File "/opt/conda/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
#14 336.3 super().run_command(command)
#14 336.3 File "/opt/conda/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
#14 336.3 cmd_obj.run()
#14 336.3 File "/intel-extension-for-pytorch/setup.py", line 286, in run
#14 336.3 self.build_extension(ext)
#14 336.3 File "/intel-extension-for-pytorch/setup.py", line 339, in build_extension
#14 336.3 check_call([self.cmake, ext.project_dir] + cmake_args, cwd=ext.build_dir, env=env)
#14 336.3 File "/opt/conda/lib/python3.8/subprocess.py", line 364, in check_call
#14 336.3 raise CalledProcessError(retcode, cmd)
#14 336.3 subprocess.CalledProcessError: Command '['cmake', '/intel-extension-for-pytorch', '-DCMAKE_BUILD_TYPE=Release', '-DPYTORCH_INSTALL_DIR=/opt/conda/lib/python3.8/site-packages/torch', '-DPYTHON_EXECUTABLE=/opt/conda/bin/python', '-DCMAKE_INSTALL_PREFIX=/intel-extension-for-pytorch/build/lib.linux-x86_64-cpython-38', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/intel-extension-for-pytorch/build/lib.linux-x86_64-cpython-38', '-DCMAKE_ARCHIVE_OUTPUT_DIRECTORY=/intel-extension-for-pytorch/build/lib.linux-x86_64-cpython-38', '-DPYTHON_INCLUDE_DIR=/opt/conda/include/python3.8', '-DPYTORCH_INCLUDE_DIRS=/opt/conda/lib/python3.8/site-packages/torch/include', '-DPYTORCH_LIBRARY_DIRS=/opt/conda/lib/python3.8/site-packages/torch/lib', '-DCMAKE_CXX_FLAGS=-D_GLIBCXX_USE_CXX11_ABI=0']' returned non-zero exit status 1.
#14 336.4 error: subprocess-exited-with-error
#14 336.4
#14 336.4 × Running setup.py install for torch-ipex did not run successfully.
#14 336.4 │ exit code: 1
#14 336.4 ╰─> See above for output.
#14 336.4
#14 336.4 note: This error originates from a subprocess, and is likely not a problem with pip.
#14 336.4 full command: /opt/conda/bin/python -u -c '
#14 336.4 exec(compile('"'"''"'"''"'"'
#14 336.4 # This is -- a caller that pip uses to run setup.py
#14 336.4 #
#14 336.4 # - It imports setuptools before invoking setup.py, to enable projects that directly
#14 336.4 # import from distutils.core to work with newer packaging standards.
#14 336.4 # - It provides a clear error message when setuptools is not installed.
#14 336.4 # - It sets sys.argv[0] to the underlying setup.py, when invoking setup.py so
#14 336.4 # setuptools doesn'"'"'t think the script is -c. This avoids the following warning:
#14 336.4 # manifest_maker: standard file '"'"'-c'"'"' not found".
#14 336.4 # - It generates a shim setup.py, for handling setup.cfg-only projects.
#14 336.4 import os, sys, tokenize
#14 336.4
#14 336.4 try:
#14 336.4 import setuptools
#14 336.4 except ImportError as error:
#14 336.4 print(
#14 336.4 "ERROR: Can not execute setup.py since setuptools is not available in "
#14 336.4 "the build environment.",
#14 336.4 file=sys.stderr,
#14 336.4 )
#14 336.4 sys.exit(1)
#14 336.4
#14 336.4 file = %r
#14 336.4 sys.argv[0] = file
#14 336.4
#14 336.4 if os.path.exists(file):
#14 336.4 filename = file
#14 336.4 with tokenize.open(file) as f:
#14 336.4 setup_py_code = f.read()
#14 336.4 else:
#14 336.4 filename = ""
#14 336.4 setup_py_code = "from setuptools import setup; setup()"
#14 336.4
#14 336.4 exec(compile(setup_py_code, filename, "exec"))
#14 336.4 '"'"''"'"''"'"' % ('"'"'/intel-extension-for-pytorch/setup.py'"'"',), "", "exec"))' install --record /tmp/pip-record-2r8g_0k1/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/include/python3.8/torch-ipex
#14 336.4 cwd: /intel-extension-for-pytorch/
#14 336.4 Running setup.py install for torch-ipex: finished with status 'error'
#14 336.4 error: legacy-install-failure
#14 336.4
#14 336.4 × Encountered error while trying to install package.
#14 336.4 ╰─> torch-ipex
#14 336.4
#14 336.4 note: This is an issue with the package mentioned above, not pip.
#14 336.4 hint: See above for output from the failure.

executor failed running [/bin/sh -c pip3 install torch==${PYTORCH_VERSION}+cpu torchvision -f https://download.pytorch.org/whl/torch_stable.html && git clone -b ${IPEX_VERSION} --single-branch https://github.com/intel/intel-extension-for-pytorch && cd intel-extension-for-pytorch && git submodule sync && git submodule update --init --recursive && pip3 install -r requirements.txt && pip3 install -v . && rm -rf *]: exit code: 1
tradi@tradi-desk3:/mnt/c/users/tradi/Downloads$

[jingxu10]

Hi, we will update that oneAPI sample to use the latest IPEX release. As a temporary workaround, would you try manually apply the patch at https://github.com/oneapi-src/visual-quality-inspection/pull/2/files? To overwrite the existing installed old version of ipex, please make a separate installation of the ipex with instructions at https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/installation.html.

[tarekradi]

Thanks. So are these the right versions to use? Please help. Thanks.

root@cnvrg-job-notebooksession-nmcfo5gveuxhor8ijks8-4-6fcd85dd94zm44:/cnvrg# pip list | grep torch
intel-extension-for-pytorch 1.12.300
torch 1.12.0
torchaudio 0.13.0+cpu
torchvision 0.14.0+cpu

[tarekradi]

I am getting the following now when training. Also, shouldn't i also make similar changes to pytorch_evaluation.py ? That currently ALSO does an import intel_pytorch_extension as ipex and sets DEVICE = ipex.DEVICE

/usr/local/lib/python3.8/dist-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: /usr/local/lib/python3.8/dist-packages/torchvision/image.so: undefined symbol: _ZN2at4_ops19empty_memory_format4callEN3c108ArrayRefINS2_6SymIntEEENS2_8optionalINS2_10ScalarTypeEEENS6_INS2_6LayoutEEENS6_INS2_6DeviceEEENS6_IbEENS6_INS2_12MemoryFormatEEE
warn(f"Failed to load image Python extension: {e}")

[jingxu10]

Yes, please do the same code change for training as well.
Please try torchvision 0.13.0+cpu. If that version gives this error, you need to compile torchvision from source.

[tarekradi]

Thanks. I updated training.py as well. Can you please make these fixes in the main branch and update the documentation. It would be nice not to have other people experience this issue.

I am not getting any errors with torchvision 0.14.0+cpu, but in case i find issues, can you tell me what steps i should take to install 0.13.0+cpu?

Thank you very much for your help and support.

[jingxu10]

Yeah, sure. We are in process of updating that sample code. Thanks for the suggestion.
To install torchvision 0.13.0+cpu, you can either install it via the prebuilt wheel or compile it from source.

• Prebuilt wheel
  python -m pip install torchvision==0.13.1+cpu --no-deps --extra-index-url https://download.pytorch.org/whl/cpu
• Compile from source

git clone https://github.com/pytorch/vision.git
cd vision
git checkout v0.13.1
git submodule sync
git submodule update --init --recursive
python setup.py install

[tarekradi]

Thank you. This was helpful. Much appreciated.