Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"Error: No CUDA Runtime library found" for ≥v4.0.0 #1808

Closed
wsshin opened this issue Mar 15, 2023 · 13 comments
Closed

"Error: No CUDA Runtime library found" for ≥v4.0.0 #1808

wsshin opened this issue Mar 15, 2023 · 13 comments

Comments

@wsshin
Copy link

wsshin commented Mar 15, 2023 •
edited
Loading

I'm using Julia 1.9.0-rc1:

julia> versioninfo()
Julia Version 1.9.0-rc1
Commit 3b2e0d8fbc* (2023-03-07 07:51 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 80 × Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, cascadelake)
  Threads: 1 on 80 virtual cores
Environment:
  LD_LIBRARY_PATH = /usr/lib64:/usr/lib64/atlas:/usr/lib64:/usr/lib64/atlas:
  JULIA_DEBUG = CUDA_Runtime_Discovery
  JULIA_PKG_DEVDIR = /home/gridsan/<MY ID>/code

CUDA.jl v3.13.1 works fine on my system:

(@v1.9) pkg> st CUDA
Status `~/.julia/environments/v1.9/Project.toml`
⌅ [052768ef] CUDA v3.13.1

julia> using CUDA

julia> CUDA.functional()
true

julia> CUDA.versioninfo()
CUDA toolkit 11.7, artifact installation
NVIDIA driver 470.57.2, for CUDA 11.4
CUDA driver 11.7

Libraries:
- CUBLAS: 11.10.1
- CURAND: 10.2.10
- CUFFT: 10.7.2
- CUSOLVER: 11.3.5
- CUSPARSE: 11.7.3
- CUPTI: 17.0.0
- NVML: 11.0.0+470.57.2
- CUDNN: 8.30.2 (for CUDA 11.5.0)
- CUTENSOR: 1.4.0 (for CUDA 11.5.0)

Toolchain:
- Julia: 1.9.0-rc1
- LLVM: 14.0.6
- PTX ISA support: 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3, 6.4, 6.5, 7.0, 7.1, 7.2, 7.3, 7.4, 7.5
- Device capability support: sm_35, sm_37, sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86

2 devices:
  0: Tesla V100-PCIE-32GB (sm_70, 31.745 GiB / 31.749 GiB available)
  1: Tesla V100-PCIE-32GB (sm_70, 31.745 GiB / 31.749 GiB available)

Now, if I remove CUDA.jl v3.13.1 and add v4.0.0, I get an error:

(@v1.9) pkg> rm CUDA
[...]

(@v1.9) pkg> add CUDA@4.0.0
    Updating registry at `~/.julia/registries/MITLL`
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating WARNING: redefinition of constant cudnnActivationDescriptorCache. This may fail, cause incorrect answers, or produce other errors.
[...]

julia> using CUDA
┌ Error: No CUDA Runtime library found. This can have several reasons:
│ * you are using an unsupported platform: CUDA.jl only supports Linux (x86_64, aarch64, ppc64le), and Windows (x86_64).
│   refer to the documentation for instructions on how to use a custom CUDA runtime.
│ * you precompiled CUDA.jl in an environment where the CUDA driver was not available.
│   in that case, you need to specify (during pre compilation) which version of CUDA to use.
│   refer to the documentation for instructions on how to use `CUDA.set_runtime_version!`.
│ * you requested use of a local CUDA toolkit, but not all components were discovered.
│   try running with JULIA_DEBUG=CUDA_Runtime_Discovery for more information.
└ @ CUDA ~/.julia/packages/CUDA/ZX8mg/src/initialization.jl:77

If I downgrade to v3.13.1 again, CUDA.jl starts working again.

I tried v4.0.1; the same error as v4.0.0.

Any insights on this issue?
@wsshin wsshin added the bug Something isn't working label Mar 15, 2023
@maleadt
Copy link
Member

maleadt commented Mar 16, 2023

Can you run with JULIA_DEBUG=all and post the output?

@wsshin
Copy link
Author

wsshin commented Mar 16, 2023 •
edited
Loading

I'll post the debug messages in two parts: the ones while adding CUDA and the ones while importing CUDA. First, I removed CUDA@3.13.1, restarted Julia with JULIA_DEBUG=all, and here are the messages I got during adding CUDA@4.0.0:

$ export JULIA_DEBUG=all; julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.0-rc1 (2023-03-07)
 _/ |\__'_|_|_|\__'_|  |
|__/                   |

(@v1.9) pkg> add CUDA@4.0.0
    Updating registry at `~/.julia/registries/MITLL`
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
   Installed LazyArrays ─── v0.22.18
   Installed PrettyTables ─ v2.2.3
    Updating `~/.julia/environments/v1.9/Project.toml`
⌃ [052768ef] + CUDA v4.0.0
    Updating `~/.julia/environments/v1.9/Manifest.toml`
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Preferences/pWSk8_ju0tv.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/JLLWrappers/7Zgw7_ju0tv.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CUDA_Driver_jll/QJyk7_ju0tv.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:1010
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 11.4.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:76
┌ Debug: Forward-compatible CUDA driver found at /home/gridsan/<MY ID>/.julia/artifacts/b5e755e06f4d49a5ab1a638eea5d75bf20c66e3d/lib/libcuda.so; reported as version 11.8.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:127
┌ Debug: Successfully loaded forwards-compatible CUDA driver
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:156
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Preferences/pWSk8_ju0tv.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/JLLWrappers/7Zgw7_ju0tv.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CUDA_Driver_jll/QJyk7_ju0tv.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:1010
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 11.4.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:76
┌ Debug: Forward-compatible CUDA driver found at /home/gridsan/<MY ID>/.julia/artifacts/b5e755e06f4d49a5ab1a638eea5d75bf20c66e3d/lib/libcuda.so; reported as version 11.8.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:127
┌ Debug: Successfully loaded forwards-compatible CUDA driver
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:156
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CUDA_Runtime_jll/Hs50y_ju0tv.so for CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2]
└ @ Base loading.jl:1010
  [ab4f0b2a] ↑ BFloat16s v0.2.0 ⇒ v0.4.2
⌃ [052768ef] ↑ CUDA v3.13.1 ⇒ v4.0.0
  [1af6417a] + CUDA_Runtime_Discovery v0.1.1
  [5078a376] ↑ LazyArrays v0.22.17 ⇒ v0.22.18
  [a00861dc] ↑ NNlibCUDA v0.2.6 ⇒ v0.2.7
  [08abe8d2] ↑ PrettyTables v2.2.2 ⇒ v2.2.3
  [02a925ec] + cuDNN v1.0.1
⌅ [4ee394cb] + CUDA_Driver_jll v0.2.0+0
⌅ [76a88914] + CUDA_Runtime_jll v0.2.3+2
⌅ [62b44479] + CUDNN_jll v8.6.0+5
        Info Packages marked with ⌃ and ⌅ have new versions available, but those with ⌅ are restricted by compatibility constraints from upgrading. To see why use `status --outdated -m`
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Preferences/pWSk8_ju0tv.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/JLLWrappers/7Zgw7_ju0tv.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CUDA_Driver_jll/QJyk7_ju0tv.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:1010
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 11.4.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:76
┌ Debug: Forward-compatible CUDA driver found at /home/gridsan/<MY ID>/.julia/artifacts/b5e755e06f4d49a5ab1a638eea5d75bf20c66e3d/lib/libcuda.so; reported as version 11.8.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:127
┌ Debug: Successfully loaded forwards-compatible CUDA driver
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:156
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Preferences/pWSk8_ju0tv.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/JLLWrappers/7Zgw7_ju0tv.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CUDA_Driver_jll/QJyk7_ju0tv.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:1010
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 11.4.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:76
┌ Debug: Forward-compatible CUDA driver found at /home/gridsan/<MY ID>/.julia/artifacts/b5e755e06f4d49a5ab1a638eea5d75bf20c66e3d/lib/libcuda.so; reported as version 11.8.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:127
┌ Debug: Successfully loaded forwards-compatible CUDA driver
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:156
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CUDA_Runtime_jll/Hs50y_ju0tv.so for CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2]
└ @ Base loading.jl:1010
┌ Debug: Rejecting cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/BFloat16s/iiZ8G_ju0tv.ji because it is for file /home/gridsan/<MY ID>/.julia/packages/BFloat16s/TNkTq/src/BFloat16s.jl not file /home/gridsan/<MY ID>/.julia/packages/BFloat16s/uUmkF/src/BFloat16s.jl
└ @ Base loading.jl:2773
┌ Debug: Rejecting cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/BFloat16s/iiZ8G_Fah0t.ji due to it containing an invalid cache header
└ @ Base loading.jl:2676
┌ Debug: Rejecting cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/PrettyTables/kRdcL_ju0tv.ji because it is for file /home/gridsan/<MY ID>/.julia/packages/PrettyTables/fFLkf/src/PrettyTables.jl not file /home/gridsan/<MY ID>/.julia/packages/PrettyTables/pZvTh/src/PrettyTables.jl
└ @ Base loading.jl:2773
┌ Debug: Rejecting cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/BFloat16s/iiZ8G_LKVXm.ji due to it containing an invalid cache header
└ @ Base loading.jl:2676
┌ Debug: Rejecting cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/LazyArrays/h0pkJ_ju0tv.ji because it is for file /home/gridsan/<MY ID>/.julia/packages/LazyArrays/2FXg4/src/LazyArrays.jl not file /home/gridsan/<MY ID>/.julia/packages/LazyArrays/VJHIG/src/LazyArrays.jl
└ @ Base loading.jl:2773
Precompiling environment...
  Progress [==================================>      ]  16/19
  ✗ cuDNN
  ✗ ArrayInterface → ArrayInterfaceCUDAExt
  ✗ NNlibCUDA
  Progress [====================================>    ]  17/19
  ✗ cuDNN
  ✗ ArrayInterface → ArrayInterfaceCUDAExt
  ✗ NNlibCUDA
  ✗ Flux
  ✗ ParameterSchedulers
  ✗ cuDNN
  ✗ ArrayInterface → ArrayInterfaceCUDAExt
  ✗ NNlibCUDA
  ✗ Flux
  ✗ ParameterSchedulers
  ✗ PointNet
  ✗ FluxTraining
  ✗ FastAI
  ✗ FastVision
  9 dependencies successfully precompiled in 206 seconds. 381 already precompiled. 1 skipped during auto due to previous errors.
  9 dependencies errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the packages
  8 dependencies had warnings during precompilation:
┌ CUDA [052768ef-5323-5732-b1bb-66c8b64840ba]
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CEnum/0gyUJ_ju0tv.so for CEnum [fa961155-64e5-5f13-b03f-caf6b980ea82]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Preferences/pWSk8_ju0tv.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/JLLWrappers/7Zgw7_ju0tv.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/LLVMExtra_jll/R9OeX_ju0tv.so for LLVMExtra_jll [dad2f222-ce93-54a1-a47d-0025e8a3acab]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/LLVM/e8NBy_ju0tv.so for LLVM [929cbde3-209d-540e-8aea-75f648917ca0]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Using LLVM 14.0.6 at /home/gridsan/<MY ID>/pkg/julia/julia-1.9/usr/bin/../lib/libLLVM-14jl.so
│  └ @ LLVM ~/.julia/packages/LLVM/s3bxG/src/LLVM.jl:86
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/ExprTools/eM8wu_ju0tv.so for ExprTools [e2ba6199-217a-4e67-a87a-7c52f15ade04]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/TimerOutputs/hd2yD_ju0tv.so for TimerOutputs [a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/GPUCompiler/yPwef_ju0tv.so for GPUCompiler [61eb1bfa-7361-4325-ad38-22787b887f55]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Adapt/rUIgN_ju0tv.so for Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Reexport/bTpYr_ju0tv.so for Reexport [189a3867-3050-52da-a836-e630ba90ab69]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/GPUArraysCore/qiYUe_ju0tv.so for GPUArraysCore [46192b85-c4d5-4398-a991-12ede77f4527]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Statistics/ERcPL_ju0tv.so for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/GPUArrays/v5u0T_ju0tv.so for GPUArrays [0c68f7d7-f131-5f86-a1c3-88cf8149b2d7]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Requires/IyxeS_ju0tv.so for Requires [ae029012-a4dd-5104-9daa-d747884805df]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/BFloat16s/iiZ8G_ju0tv.so for BFloat16s [ab4f0b2a-ad5b-11e8-123f-65d77653426b]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CUDA_Driver_jll/QJyk7_ju0tv.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
│  └ @ Base loading.jl:1010
│  ┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 11.4.0
│  └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:76
│  ┌ Debug: Forward-compatible CUDA driver found at /home/gridsan/<MY ID>/.julia/artifacts/b5e755e06f4d49a5ab1a638eea5d75bf20c66e3d/lib/libcuda.so; reported as version 11.8.0
│  └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:127
│  ┌ Debug: Successfully loaded forwards-compatible CUDA driver
│  └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:156
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CUDA_Runtime_jll/Hs50y_ju0tv.so for CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/RandomNumbers/pgCpR_ju0tv.so for RandomNumbers [e6cf234a-135c-5ec9-84dd-332b85af5143]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Random123/1imiM_ju0tv.so for Random123 [74087812-796a-5b5d-8853-05524746bad3]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/AbstractFFTs/Di3HZ_ju0tv.so for AbstractFFTs [621f4979-c628-5d54-868e-fcf4e3e8185c]
│  └ @ Base loading.jl:1010
└
┌ PrettyTables [08abe8d2-0d0c-5749-adfa-8a2ac140af0d]
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Formatting/vCa4N_ju0tv.so for Formatting [59287772-0a20-5a39-b81b-1366585eb4c0]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Reexport/bTpYr_ju0tv.so for Reexport [189a3867-3050-52da-a836-e630ba90ab69]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DataValueInterfaces/9Lpkp_ju0tv.so for DataValueInterfaces [e2d170a0-9d28-54be-80f0-106bbe20a464]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DataAPI/3a8mN_ju0tv.so for DataAPI [9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/IteratorInterfaceExtensions/N0h8q_ju0tv.so for IteratorInterfaceExtensions [82899510-4779-5014-852e-03e436cf321d]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/TableTraits/I6SaN_ju0tv.so for TableTraits [3783bdb8-4a98-5b6b-af9a-565f29a5fe9c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/OrderedCollections/LtT3J_ju0tv.so for OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Tables/Z804B_ju0tv.so for Tables [bd369af6-aec1-5ad0-b16a-f7cc5008161c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/StringManipulation/4nJQd_ju0tv.so for StringManipulation [892a3eda-7b42-436c-8928-eab12a02cf0e]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Crayons/TXPcU_ju0tv.so for Crayons [a8cc5b0e-0ffa-5ad4-8c14-923d3ee1735f]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/LaTeXStrings/H4HGh_ju0tv.so for LaTeXStrings [b964fa9f-0449-5b57-a5c2-d3ea65f4040f]
│  └ @ Base loading.jl:1010
└
┌ POPSSceneGenUtils [9f337a4a-a41b-4491-946b-bf42a175d4fa]
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Preferences/pWSk8_ju0tv.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/SnoopPrecompile/TcfDT_ju0tv.so for SnoopPrecompile [66db9d55-30c0-4569-8b51-7e840670fc0c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Parsers/833b9_ju0tv.so for Parsers [69de0a69-1ddd-5017-9359-2bf0b02dc9f0]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DataValueInterfaces/9Lpkp_ju0tv.so for DataValueInterfaces [e2d170a0-9d28-54be-80f0-106bbe20a464]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DataAPI/3a8mN_ju0tv.so for DataAPI [9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/IteratorInterfaceExtensions/N0h8q_ju0tv.so for IteratorInterfaceExtensions [82899510-4779-5014-852e-03e436cf321d]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/TableTraits/I6SaN_ju0tv.so for TableTraits [3783bdb8-4a98-5b6b-af9a-565f29a5fe9c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/OrderedCollections/LtT3J_ju0tv.so for OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Tables/Z804B_ju0tv.so for Tables [bd369af6-aec1-5ad0-b16a-f7cc5008161c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/PooledArrays/vi11X_ju0tv.so for PooledArrays [2dfb63ee-cc39-5dd5-95bd-886bf059d720]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/SentinelArrays/uMYVe_ju0tv.so for SentinelArrays [91c51154-3ec4-41a3-a24f-3f23e20d615c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/InlineStrings/prmY3_ju0tv.so for InlineStrings [842dd82b-1e85-43dc-bf29-5d0ee9dffc48]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/WeakRefStrings/muGbw_ju0tv.so for WeakRefStrings [ea10d353-3f73-51f8-a26c-33c1cb351aa5]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/TranscodingStreams/eJQ7D_ju0tv.so for TranscodingStreams [3bb67fe8-82b1-5028-8e26-92a6c54297fa]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Zlib_jll/xjq3Q_ju0tv.so for Zlib_jll [83775a58-1f1d-513f-b197-d71354ab007a]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CodecZlib/1TI30_ju0tv.so for CodecZlib [944b1d66-785c-5afd-91f1-9de20f533193]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Compat/GSFWK_ju0tv.so for Compat [34da2185-b29b-5c13-b0c7-acf172513d20]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CompatLinearAlgebraExt/Zxpzq_ju0tv.so for CompatLinearAlgebraExt [dbe5ba0b-aecc-598a-a867-79051b540f49]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Extension CompatLinearAlgebraExt of Compat loaded
│  └ @ Base loading.jl:1196
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/FilePathsBase/Efv1v_ju0tv.so for FilePathsBase [48062228-2e41-5def-b9a4-89aafe57970f]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/WorkerUtilities/8EFUg_ju0tv.so for WorkerUtilities [76eceee3-57b5-4d4a-8e66-0e911cebbf60]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CSV/HHBkp_ju0tv.so for CSV [336ed68f-0bac-5ca0-87d4-7b16caf5d00b]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Statistics/ERcPL_ju0tv.so for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Reexport/bTpYr_ju0tv.so for Reexport [189a3867-3050-52da-a836-e630ba90ab69]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DataStructures/xKiwJ_ju0tv.so for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/SortingAlgorithms/6dCmw_ju0tv.so for SortingAlgorithms [a2af1166-a08f-5f64-846c-94a0d3cef48c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Missings/wLfgT_ju0tv.so for Missings [e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/InvertedIndices/0P8C6_ju0tv.so for InvertedIndices [41ab1584-1d38-5bbf-9106-f11c6c58b48f]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Formatting/vCa4N_ju0tv.so for Formatting [59287772-0a20-5a39-b81b-1366585eb4c0]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/StringManipulation/4nJQd_ju0tv.so for StringManipulation [892a3eda-7b42-436c-8928-eab12a02cf0e]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Crayons/TXPcU_ju0tv.so for Crayons [a8cc5b0e-0ffa-5ad4-8c14-923d3ee1735f]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/LaTeXStrings/H4HGh_ju0tv.so for LaTeXStrings [b964fa9f-0449-5b57-a5c2-d3ea65f4040f]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/PrettyTables/kRdcL_ju0tv.so for PrettyTables [08abe8d2-0d0c-5749-adfa-8a2ac140af0d]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DataFrames/AR9oZ_ju0tv.so for DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/StaticArraysCore/Tzw28_ju0tv.so for StaticArraysCore [1e83bf80-4336-4d27-bf5d-d5a4f845583c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/StaticArrays/yY9vm_ju0tv.so for StaticArrays [90137ffa-7385-5640-81b9-e52037218182]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Adapt/rUIgN_ju0tv.so for Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/AdaptStaticArraysExt/9bCdf_ju0tv.so for AdaptStaticArraysExt [e1699a77-9e31-5da8-bb3e-0a796f95f0a0]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Extension AdaptStaticArraysExt of Adapt loaded
│  └ @ Base loading.jl:1196
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/GPUArraysCore/qiYUe_ju0tv.so for GPUArraysCore [46192b85-c4d5-4398-a991-12ede77f4527]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/StructArrays/jRMFC_ju0tv.so for StructArrays [09ab397b-f2b6-538f-b94a-2f83cf4a842a]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/RealDot/dCVlS_ju0tv.so for RealDot [c1ae055f-0cd5-4b69-90a6-9a35b1a98df9]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Quaternions/hbPLI_ju0tv.so for Quaternions [94ee1d12-ae83-5a48-8b1c-48b8ff168ae0]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Rotations/SJ1Dh_ju0tv.so for Rotations [6038ab10-8711-5258-84ad-4b1120ba62dc]
│  └ @ Base loading.jl:1010
└
┌ DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0]
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Statistics/ERcPL_ju0tv.so for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Reexport/bTpYr_ju0tv.so for Reexport [189a3867-3050-52da-a836-e630ba90ab69]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Compat/GSFWK_ju0tv.so for Compat [34da2185-b29b-5c13-b0c7-acf172513d20]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CompatLinearAlgebraExt/Zxpzq_ju0tv.so for CompatLinearAlgebraExt [dbe5ba0b-aecc-598a-a867-79051b540f49]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Extension CompatLinearAlgebraExt of Compat loaded
│  └ @ Base loading.jl:1196
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/OrderedCollections/LtT3J_ju0tv.so for OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DataStructures/xKiwJ_ju0tv.so for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/SortingAlgorithms/6dCmw_ju0tv.so for SortingAlgorithms [a2af1166-a08f-5f64-846c-94a0d3cef48c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DataAPI/3a8mN_ju0tv.so for DataAPI [9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/PooledArrays/vi11X_ju0tv.so for PooledArrays [2dfb63ee-cc39-5dd5-95bd-886bf059d720]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Missings/wLfgT_ju0tv.so for Missings [e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/InvertedIndices/0P8C6_ju0tv.so for InvertedIndices [41ab1584-1d38-5bbf-9106-f11c6c58b48f]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/IteratorInterfaceExtensions/N0h8q_ju0tv.so for IteratorInterfaceExtensions [82899510-4779-5014-852e-03e436cf321d]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/TableTraits/I6SaN_ju0tv.so for TableTraits [3783bdb8-4a98-5b6b-af9a-565f29a5fe9c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Formatting/vCa4N_ju0tv.so for Formatting [59287772-0a20-5a39-b81b-1366585eb4c0]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DataValueInterfaces/9Lpkp_ju0tv.so for DataValueInterfaces [e2d170a0-9d28-54be-80f0-106bbe20a464]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Tables/Z804B_ju0tv.so for Tables [bd369af6-aec1-5ad0-b16a-f7cc5008161c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/StringManipulation/4nJQd_ju0tv.so for StringManipulation [892a3eda-7b42-436c-8928-eab12a02cf0e]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Crayons/TXPcU_ju0tv.so for Crayons [a8cc5b0e-0ffa-5ad4-8c14-923d3ee1735f]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/LaTeXStrings/H4HGh_ju0tv.so for LaTeXStrings [b964fa9f-0449-5b57-a5c2-d3ea65f4040f]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/PrettyTables/kRdcL_ju0tv.so for PrettyTables [08abe8d2-0d0c-5749-adfa-8a2ac140af0d]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Preferences/pWSk8_ju0tv.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/SnoopPrecompile/TcfDT_ju0tv.so for SnoopPrecompile [66db9d55-30c0-4569-8b51-7e840670fc0c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/SentinelArrays/uMYVe_ju0tv.so for SentinelArrays [91c51154-3ec4-41a3-a24f-3f23e20d615c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Parsers/833b9_ju0tv.so for Parsers [69de0a69-1ddd-5017-9359-2bf0b02dc9f0]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/InlineStrings/prmY3_ju0tv.so for InlineStrings [842dd82b-1e85-43dc-bf29-5d0ee9dffc48]
│  └ @ Base loading.jl:1010
└
┌ MLDatasets [eb30cadb-4394-5ae3-aed4-317e484a6458]
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Statistics/ERcPL_ju0tv.so for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/FixedPointNumbers/Ple2W_ju0tv.so for FixedPointNumbers [53c48c17-4a7d-5ca2-90c5-79b7896eea93]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DataValueInterfaces/9Lpkp_ju0tv.so for DataValueInterfaces [e2d170a0-9d28-54be-80f0-106bbe20a464]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DataAPI/3a8mN_ju0tv.so for DataAPI [9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/IteratorInterfaceExtensions/N0h8q_ju0tv.so for IteratorInterfaceExtensions [82899510-4779-5014-852e-03e436cf321d]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/TableTraits/I6SaN_ju0tv.so for TableTraits [3783bdb8-4a98-5b6b-af9a-565f29a5fe9c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/OrderedCollections/LtT3J_ju0tv.so for OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Tables/Z804B_ju0tv.so for Tables [bd369af6-aec1-5ad0-b16a-f7cc5008161c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/URIs/eec2u_ju0tv.so for URIs [5c2747f8-b7ea-4ff2-ba2e-563bfd36b1d4]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/LoggingExtras/9fRHf_ju0tv.so for LoggingExtras [e6f89c97-d47a-5376-807f-9c37f3926c36]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/MbedTLS_jll/u5NEn_ju0tv.so for MbedTLS_jll [c8ffd9c3-330d-5841-b78e-0817d7145fa1]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/MbedTLS/bf9T0_ju0tv.so for MbedTLS [739be429-bea8-5141-9913-cc70e7f3736d]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/BitFlags/sbYUO_ju0tv.so for BitFlags [d1d4a3ce-64b1-5f1a-9ba4-7e7e69966f35]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Preferences/pWSk8_ju0tv.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/JLLWrappers/7Zgw7_ju0tv.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/OpenSSL_jll/M3X35_ju0tv.so for OpenSSL_jll [458c3c95-2e84-50aa-8efc-19380b2a3a95]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/OpenSSL/B4VRP_ju0tv.so for OpenSSL [4d8831e6-92b7-49fb-bdf8-b643e874388c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/TranscodingStreams/eJQ7D_ju0tv.so for TranscodingStreams [3bb67fe8-82b1-5028-8e26-92a6c54297fa]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Zlib_jll/xjq3Q_ju0tv.so for Zlib_jll [83775a58-1f1d-513f-b197-d71354ab007a]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CodecZlib/1TI30_ju0tv.so for CodecZlib [944b1d66-785c-5afd-91f1-9de20f533193]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/SimpleBufferStream/QbHsM_ju0tv.so for SimpleBufferStream [777ac1f9-54b0-4bf8-805c-2214025038e7]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/HTTP/zXWya_ju0tv.so for HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Reexport/bTpYr_ju0tv.so for Reexport [189a3867-3050-52da-a836-e630ba90ab69]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DataDeps/cCpxD_ju0tv.so for DataDeps [124859b0-ceae-595e-8997-d05f6a7a8dfe]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/ShowCases/Q8uTD_ju0tv.so for ShowCases [605ecd9f-84a6-4c9e-81e2-4798472b76a3]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Compat/GSFWK_ju0tv.so for Compat [34da2185-b29b-5c13-b0c7-acf172513d20]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CompatLinearAlgebraExt/Zxpzq_ju0tv.so for CompatLinearAlgebraExt [dbe5ba0b-aecc-598a-a867-79051b540f49]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Extension CompatLinearAlgebraExt of Compat loaded
│  └ @ Base loading.jl:1196
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/ConstructionBase/sBbW6_ju0tv.so for ConstructionBase [187b0558-2788-49d3-abe0-74a17ed4e7c9]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/InitialValues/djxcV_ju0tv.so for InitialValues [22cec73e-a1b8-11e9-2c92-598750a2cf9c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Requires/IyxeS_ju0tv.so for Requires [ae029012-a4dd-5104-9daa-d747884805df]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/MacroTools/38lnR_ju0tv.so for MacroTools [1914dd2f-81c6-5fcd-8719-6d5c9610ff09]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/ZygoteRules/LmjSI_ju0tv.so for ZygoteRules [700de1a5-db45-46bc-99cf-38207098b444]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/StaticArraysCore/Tzw28_ju0tv.so for StaticArraysCore [1e83bf80-4336-4d27-bf5d-d5a4f845583c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Setfield/dVM1R_ju0tv.so for Setfield [efcf1570-3423-57d1-acb7-fd33fddbac46]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/BangBang/Ovsha_ju0tv.so for BangBang [198e06fe-97b7-11e9-32a5-e1d131e6ad66]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/ContextVariablesX/xQyR3_ju0tv.so for ContextVariablesX [6add18c4-b38d-439d-96f6-d6bc489c04c5]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/FLoopsBase/CNUyQ_ju0tv.so for FLoopsBase [b9860ae5-e623-471e-878b-f6a53c775ea6]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/PrettyPrint/OWUdY_ju0tv.so for PrettyPrint [8162dcfd-2161-5ef2-ae6c-7681170c5f98]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/NameResolution/PJklF_ju0tv.so for NameResolution [71a1bf82-56d0-4bbc-8a3c-48b961074391]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/MLStyle/v4qmW_ju0tv.so for MLStyle [d8e11817-5142-5d16-987a-aa16d5891078]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/JuliaVariables/ViR92_ju0tv.so for JuliaVariables [b14d175d-62b4-44ba-8fb7-3064adc8c3ec]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Adapt/rUIgN_ju0tv.so for Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/ArgCheck/P66Js_ju0tv.so for ArgCheck [dce04be8-c92d-5529-be00-80e4d2c0e197]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Baselet/J6WYP_ju0tv.so for Baselet [9718e550-a3fa-408a-8086-8db961cd8217]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CompositionsBase/KqDTx_ju0tv.so for CompositionsBase [a33af91c-f02d-484b-be07-31d278c5ca2b]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DefineSingletons/YTrQa_ju0tv.so for DefineSingletons [244e2a9f-e319-4986-a169-4d1fe445cd52]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/MicroCollections/UymYM_ju0tv.so for MicroCollections [128add7d-3638-4c79-886c-908ea0c25c34]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/SplittablesBase/htRqy_ju0tv.so for SplittablesBase [171d559e-b47b-412a-8079-5efa626c420e]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Transducers/cATK6_ju0tv.so for Transducers [28d57a85-8fef-5791-bfe6-a80928e7c999]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/FLoops/99zMn_ju0tv.so for FLoops [cc61a311-1640-44b5-9fba-1b764f453329]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/InverseFunctions/PkVmn_ju0tv.so for InverseFunctions [3587e190-3f89-42d0-90ee-14403ec27112]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Accessors/XelUh_ju0tv.so for Accessors [7d9f7c33-5ae7-4f3b-8dc6-eff91059b697]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/FunctionWrappers/wbVUX_ju0tv.so for FunctionWrappers [069b7b12-0de2-55c6-9aab-29f3d0a68a2e]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/FoldsThreads/p45Dr_ju0tv.so for FoldsThreads [9c68100b-dfe1-47cf-94c8-95104e173443]
│  └ @ Base loading.jl:1010
│  ┌ Debug: `init_primary_task`: Initializing `PRIMARY_TASK` channel...
│  └ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/root_spawners.jl:24
│  ┌ Debug: `init_primary_task`: Initializing `PRIMARY_TASK` channel... DONE
│  └ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/root_spawners.jl:36
│  ┌ Debug: `init_each_thread`: Initializing `EACH_THREAD` channels...
│  └ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/root_spawners.jl:57
│  ┌ Debug: `init_each_thread`: Initializing `EACH_THREAD` channels... DONE
│  └ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/root_spawners.jl:68
│  ┌ Debug: `init_taskpool`: Initializing `TASKPOOL_REQUEST` channels...
│  └ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/taskpool.jl:85
│  ┌ Debug: Processing request run_future
│  └ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/root_spawners.jl:7
│  ┌ Debug: `init_taskpool`: Initializing `TASKPOOL_REQUEST` channels... DONE
│  └ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/taskpool.jl:88
│  ┌ Debug: `init_background_taskpool`: Initializing `BACKGROUND_TASKPOOL_REQUEST` channels...
│  └ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/taskpool.jl:92
│  ┌ Debug: Processing request run_future
│  └ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/root_spawners.jl:7
│  ┌ Debug: `init_background_taskpool`: Initializing `BACKGROUND_TASKPOOL_REQUEST` channels... DONE
│  └ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/taskpool.jl:95
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DataStructures/xKiwJ_ju0tv.so for DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/SortingAlgorithms/6dCmw_ju0tv.so for SortingAlgorithms [a2af1166-a08f-5f64-846c-94a0d3cef48c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Missings/wLfgT_ju0tv.so for Missings [e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DocStringExtensions/KRdZs_ju0tv.so for DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/IrrationalConstants/ukdUG_ju0tv.so for IrrationalConstants [92d709cd-6900-40b7-9082-c6be49f344b6]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/LogExpFunctions/cmCYR_ju0tv.so for LogExpFunctions [2ab3a3ac-af41-5b50-aa03-7779005ae688]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/LogExpFunctionsInverseFunctionsExt/IXKft_ju0tv.so for LogExpFunctionsInverseFunctionsExt [1e5f9c58-a15c-5ce5-87cf-a68a2eda25d9]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Extension LogExpFunctionsInverseFunctionsExt of LogExpFunctions loaded
│  └ @ Base loading.jl:1196
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/StatsAPI/Rf3YC_ju0tv.so for StatsAPI [82ae8749-77ed-4fe6-ae5f-f523153014b0]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/StatsBase/EZjIG_ju0tv.so for StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/ChainRulesCore/G6ax7_ju0tv.so for ChainRulesCore [d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/LogExpFunctionsChainRulesCoreExt/1yuhV_ju0tv.so for LogExpFunctionsChainRulesCoreExt [1bf5f11d-9a0a-5d25-85d0-d1d9a28a239c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Extension LogExpFunctionsChainRulesCoreExt of LogExpFunctions loaded
│  └ @ Base loading.jl:1196
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/SimpleTraits/pLp8z_ju0tv.so for SimpleTraits [699a6c99-e7fa-54fc-8d76-47d257e15c1d]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/NNlib/A7zdE_ju0tv.so for NNlib [872c559c-99b0-510c-b3b7-b6c96a88d5cd]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DelimitedFiles/dlKZm_ju0tv.so for DelimitedFiles [8bb1440f-4735-579b-a4ab-409b98df4dab]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/MLUtils/S1qG5_ju0tv.so for MLUtils [f1d291b0-491e-4a28-83b9-f70985020b54]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Glob/3FzEV_ju0tv.so for Glob [c27321d9-0574-5035-807b-f59d2c89b15c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/FileIO/6iKRU_ju0tv.so for FileIO [5789e2e9-d7fb-5bc7-8068-2c6fae9b9549]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/SnoopPrecompile/TcfDT_ju0tv.so for SnoopPrecompile [66db9d55-30c0-4569-8b51-7e840670fc0c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Parsers/833b9_ju0tv.so for Parsers [69de0a69-1ddd-5017-9359-2bf0b02dc9f0]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/PooledArrays/vi11X_ju0tv.so for PooledArrays [2dfb63ee-cc39-5dd5-95bd-886bf059d720]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/SentinelArrays/uMYVe_ju0tv.so for SentinelArrays [91c51154-3ec4-41a3-a24f-3f23e20d615c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/InlineStrings/prmY3_ju0tv.so for InlineStrings [842dd82b-1e85-43dc-bf29-5d0ee9dffc48]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/WeakRefStrings/muGbw_ju0tv.so for WeakRefStrings [ea10d353-3f73-51f8-a26c-33c1cb351aa5]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/FilePathsBase/Efv1v_ju0tv.so for FilePathsBase [48062228-2e41-5def-b9a4-89aafe57970f]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/WorkerUtilities/8EFUg_ju0tv.so for WorkerUtilities [76eceee3-57b5-4d4a-8e66-0e911cebbf60]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CSV/HHBkp_ju0tv.so for CSV [336ed68f-0bac-5ca0-87d4-7b16caf5d00b]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/LazyModules/K2iU0_ju0tv.so for LazyModules [8cdb02fc-e678-4876-92c5-9defec4f444e]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/GZip/s2LKY_ju0tv.so for GZip [92fee26a-97fe-5a0c-ad85-20a5f3185b63]
│  └ @ Base loading.jl:1010
└
┌ LazyArrays [5078a376-72f3-5289-bfd5-ec5146d43c02]
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Statistics/ERcPL_ju0tv.so for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/FillArrays/2Dg1l_ju0tv.so for FillArrays [1a297f60-69ca-5386-bcde-b61e274b549b]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/StaticArraysCore/Tzw28_ju0tv.so for StaticArraysCore [1e83bf80-4336-4d27-bf5d-d5a4f845583c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/StaticArrays/yY9vm_ju0tv.so for StaticArrays [90137ffa-7385-5640-81b9-e52037218182]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/ArrayLayouts/QloHc_ju0tv.so for ArrayLayouts [4c555306-a7a7-4459-81d9-ec55ddd5c99a]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/MatrixFactorizations/5wecE_ju0tv.so for MatrixFactorizations [a3b82374-2e81-5b9e-98ce-41277c0e4c87]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/MacroTools/38lnR_ju0tv.so for MacroTools [1914dd2f-81c6-5fcd-8719-6d5c9610ff09]
│  └ @ Base loading.jl:1010
└
┌ FeatureRegistries [c6aefb4f-3ac3-4095-8805-528476b02c02]
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/InlineTest/W03sq_ju0tv.so for InlineTest [bd334432-b1e7-49c7-a2dc-dd9149e4ebd6]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Formatting/vCa4N_ju0tv.so for Formatting [59287772-0a20-5a39-b81b-1366585eb4c0]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Reexport/bTpYr_ju0tv.so for Reexport [189a3867-3050-52da-a836-e630ba90ab69]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DataValueInterfaces/9Lpkp_ju0tv.so for DataValueInterfaces [e2d170a0-9d28-54be-80f0-106bbe20a464]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DataAPI/3a8mN_ju0tv.so for DataAPI [9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/IteratorInterfaceExtensions/N0h8q_ju0tv.so for IteratorInterfaceExtensions [82899510-4779-5014-852e-03e436cf321d]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/TableTraits/I6SaN_ju0tv.so for TableTraits [3783bdb8-4a98-5b6b-af9a-565f29a5fe9c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/OrderedCollections/LtT3J_ju0tv.so for OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Tables/Z804B_ju0tv.so for Tables [bd369af6-aec1-5ad0-b16a-f7cc5008161c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/StringManipulation/4nJQd_ju0tv.so for StringManipulation [892a3eda-7b42-436c-8928-eab12a02cf0e]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Crayons/TXPcU_ju0tv.so for Crayons [a8cc5b0e-0ffa-5ad4-8c14-923d3ee1735f]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/LaTeXStrings/H4HGh_ju0tv.so for LaTeXStrings [b964fa9f-0449-5b57-a5c2-d3ea65f4040f]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/PrettyTables/kRdcL_ju0tv.so for PrettyTables [08abe8d2-0d0c-5749-adfa-8a2ac140af0d]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/StaticArraysCore/Tzw28_ju0tv.so for StaticArraysCore [1e83bf80-4336-4d27-bf5d-d5a4f845583c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Adapt/rUIgN_ju0tv.so for Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/GPUArraysCore/qiYUe_ju0tv.so for GPUArraysCore [46192b85-c4d5-4398-a991-12ede77f4527]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/StructArrays/jRMFC_ju0tv.so for StructArrays [09ab397b-f2b6-538f-b94a-2f83cf4a842a]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Requires/IyxeS_ju0tv.so for Requires [ae029012-a4dd-5104-9daa-d747884805df]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/FileIO/6iKRU_ju0tv.so for FileIO [5789e2e9-d7fb-5bc7-8068-2c6fae9b9549]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Statistics/ERcPL_ju0tv.so for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/FixedPointNumbers/Ple2W_ju0tv.so for FixedPointNumbers [53c48c17-4a7d-5ca2-90c5-79b7896eea93]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/ColorTypes/db21U_ju0tv.so for ColorTypes [3da002f7-5984-5a60-b8a6-cbb66c0b333f]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Colors/NKjaT_ju0tv.so for Colors [5ae59095-9a9b-59fe-a467-6f913c188581]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/IrrationalConstants/ukdUG_ju0tv.so for IrrationalConstants [92d709cd-6900-40b7-9082-c6be49f344b6]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/DocStringExtensions/KRdZs_ju0tv.so for DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/LogExpFunctions/cmCYR_ju0tv.so for LogExpFunctions [2ab3a3ac-af41-5b50-aa03-7779005ae688]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/OpenLibm_jll/ToVO1_ju0tv.so for OpenLibm_jll [05823500-19ac-5b8b-9628-191a04bc5112]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Preferences/pWSk8_ju0tv.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/JLLWrappers/7Zgw7_ju0tv.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/OpenSpecFun_jll/TDl1L_ju0tv.so for OpenSpecFun_jll [efe28fd5-8261-553b-a9e1-b2916fc3738e]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/SpecialFunctions/78gOt_ju0tv.so for SpecialFunctions [276daf66-3868-5448-9aa4-cd146d93841b]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/TensorCore/wqzBb_ju0tv.so for TensorCore [62fd8b95-f654-4bbd-a8a5-9c27f68ccd50]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/ColorVectorSpace/7uC4N_ju0tv.so for ColorVectorSpace [c3611d14-8923-5661-9e6a-0046d554d3a4]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/OffsetArrays/UDEDl_ju0tv.so for OffsetArrays [6fe1bfb0-de20-5000-8ca7-80f57d26f881]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/PaddedViews/SrUyA_ju0tv.so for PaddedViews [5432bcbf-9aad-5242-b902-cca2824c8663]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/MappedArrays/xJB11_ju0tv.so for MappedArrays [dbb5928d-eab1-5f90-85c2-b9b0edb7c900]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/StackViews/VkAPD_ju0tv.so for StackViews [cae243ae-269e-4f55-b966-ac2d0dc13c15]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/MosaicViews/X3bJX_ju0tv.so for MosaicViews [e94cdb99-869f-56ef-bcf0-1ae2bcbe0389]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/NaNMath/k9Y1O_ju0tv.so for NaNMath [77ba4419-2d1f-58cd-9bb1-8ffee604a2e3]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Graphics/DwlUf_ju0tv.so for Graphics [a2bd30eb-e257-5431-a919-1863eab51364]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/AbstractFFTs/Di3HZ_ju0tv.so for AbstractFFTs [621f4979-c628-5d54-868e-fcf4e3e8185c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/ImageCore/cqNLm_ju0tv.so for ImageCore [a09fc81d-aa75-5fe9-8630-4744c3626534]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/ImageBase/kFbbd_ju0tv.so for ImageBase [c817782e-172a-44cc-b673-b171935fbb9e]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/ColorSchemes/EO5fj_ju0tv.so for ColorSchemes [35d6a980-a343-548e-a6ea-1d62b119f2f4]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/ImageShow/76qZM_ju0tv.so for ImageShow [4e3cecfd-b093-5904-9786-8bbb286a6a31]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/MacroTools/38lnR_ju0tv.so for MacroTools [1914dd2f-81c6-5fcd-8719-6d5c9610ff09]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/ConstructionBase/sBbW6_ju0tv.so for ConstructionBase [187b0558-2788-49d3-abe0-74a17ed4e7c9]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Setfield/dVM1R_ju0tv.so for Setfield [efcf1570-3423-57d1-acb7-fd33fddbac46]
│  └ @ Base loading.jl:1010
└
┌ InfiniteArrays [4858937d-0d70-526a-a4dd-2d5cb5dd786c]
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Statistics/ERcPL_ju0tv.so for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/FillArrays/2Dg1l_ju0tv.so for FillArrays [1a297f60-69ca-5386-bcde-b61e274b549b]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/ArrayLayouts/QloHc_ju0tv.so for ArrayLayouts [4c555306-a7a7-4459-81d9-ec55ddd5c99a]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Infinities/D8uzA_ju0tv.so for Infinities [e1ba4f0e-776d-440f-acd9-e1d2e9742647]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/StaticArraysCore/Tzw28_ju0tv.so for StaticArraysCore [1e83bf80-4336-4d27-bf5d-d5a4f845583c]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/StaticArrays/yY9vm_ju0tv.so for StaticArrays [90137ffa-7385-5640-81b9-e52037218182]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/MatrixFactorizations/5wecE_ju0tv.so for MatrixFactorizations [a3b82374-2e81-5b9e-98ce-41277c0e4c87]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/MacroTools/38lnR_ju0tv.so for MacroTools [1914dd2f-81c6-5fcd-8719-6d5c9610ff09]
│  └ @ Base loading.jl:1010
│  ┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/LazyArrays/h0pkJ_ju0tv.so for LazyArrays [5078a376-72f3-5289-bfd5-ec5146d43c02]
│  └ @ Base loading.jl:1010
└

@wsshin
Copy link
Author

wsshin commented Mar 16, 2023

Then, I restarted Julia with JULIA_DEBUG=all, and here are the messages I got while importing CUDA:

julia> using CUDA
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CEnum/0gyUJ_ju0tv.so for CEnum [fa961155-64e5-5f13-b03f-caf6b980ea82]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Preferences/pWSk8_ju0tv.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/JLLWrappers/7Zgw7_ju0tv.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/LLVMExtra_jll/R9OeX_ju0tv.so for LLVMExtra_jll [dad2f222-ce93-54a1-a47d-0025e8a3acab]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/LLVM/e8NBy_ju0tv.so for LLVM [929cbde3-209d-540e-8aea-75f648917ca0]
└ @ Base loading.jl:1010
┌ Debug: Using LLVM 14.0.6 at /home/gridsan/<MY ID>/pkg/julia/julia-1.9/usr/bin/../lib/libLLVM-14jl.so
└ @ LLVM ~/.julia/packages/LLVM/s3bxG/src/LLVM.jl:86
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/ExprTools/eM8wu_ju0tv.so for ExprTools [e2ba6199-217a-4e67-a87a-7c52f15ade04]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/TimerOutputs/hd2yD_ju0tv.so for TimerOutputs [a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/GPUCompiler/yPwef_ju0tv.so for GPUCompiler [61eb1bfa-7361-4325-ad38-22787b887f55]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Adapt/rUIgN_ju0tv.so for Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Reexport/bTpYr_ju0tv.so for Reexport [189a3867-3050-52da-a836-e630ba90ab69]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/GPUArraysCore/qiYUe_ju0tv.so for GPUArraysCore [46192b85-c4d5-4398-a991-12ede77f4527]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Statistics/ERcPL_ju0tv.so for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/GPUArrays/v5u0T_ju0tv.so for GPUArrays [0c68f7d7-f131-5f86-a1c3-88cf8149b2d7]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Requires/IyxeS_ju0tv.so for Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/BFloat16s/iiZ8G_ju0tv.so for BFloat16s [ab4f0b2a-ad5b-11e8-123f-65d77653426b]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CUDA_Driver_jll/QJyk7_ju0tv.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:1010
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 11.4.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:76
┌ Debug: Forward-compatible CUDA driver found at /home/gridsan/<MY ID>/.julia/artifacts/b5e755e06f4d49a5ab1a638eea5d75bf20c66e3d/lib/libcuda.so; reported as version 11.8.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:127
┌ Debug: Successfully loaded forwards-compatible CUDA driver
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:156
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CUDA_Runtime_jll/Hs50y_ju0tv.so for CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/RandomNumbers/pgCpR_ju0tv.so for RandomNumbers [e6cf234a-135c-5ec9-84dd-332b85af5143]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Random123/1imiM_ju0tv.so for Random123 [74087812-796a-5b5d-8853-05524746bad3]
└ @ Base loading.jl:1010
┌ Debug: Requires conditionally ran code in 0.003468152 seconds: `RandomNumbers` detected `Random123`
└ @ Requires ~/.julia/packages/RandomNumbers/3pD1N/src/RandomNumbers.jl:38
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/AbstractFFTs/Di3HZ_ju0tv.so for AbstractFFTs [621f4979-c628-5d54-868e-fcf4e3e8185c]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CUDA/oWw5k_ju0tv.so for CUDA [052768ef-5323-5732-b1bb-66c8b64840ba]
└ @ Base loading.jl:1010
ERROR: InitError: UndefVarError: `libcublas` not defined
Stacktrace:
  [1] macro expansion
    @ ~/.julia/packages/CUDA/ZX8mg/lib/cublas/libcublas.jl:1572 [inlined]
  [2] macro expansion
    @ ~/.julia/packages/CUDA/ZX8mg/src/pool.jl:312 [inlined]
  [3] macro expansion
    @ ~/.julia/packages/CUDA/ZX8mg/lib/cublas/libcublas.jl:22 [inlined]
  [4] cublasSetLoggerCallback(userCallback::Ptr{Nothing})
    @ CUDA.CUBLAS ~/.julia/packages/CUDA/ZX8mg/lib/utils/call.jl:26
  [5] __init__()
    @ CUDA.CUBLAS ~/.julia/packages/CUDA/ZX8mg/lib/cublas/CUBLAS.jl:254
  [6] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1074
  [7] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1020
  [8] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1465
  [9] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1742
 [10] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1619
 [11] macro expansion
    @ ./loading.jl:1607 [inlined]
 [12] macro expansion
    @ ./lock.jl:267 [inlined]
 [13] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1570
during initialization of module CUBLAS

@maleadt
Copy link
Member

maleadt commented Mar 16, 2023

That error is due to using CUDA.jl 4.0.0, please use 4.0.1 (or even better, try the release-4.0 branch).

It would also be useful if you could test the master branch.

@wsshin
Copy link
Author

wsshin commented Mar 16, 2023 •
edited
Loading

Adding CUDA@4.0.1 generated too long messages that I cannot post here. Subsequent using CUDA@4.0.1 generated the following messages (I will try CUDA#master later):

(@v1.9) pkg> st CUDA
Status `~/.julia/environments/v1.9/Project.toml`
  [052768ef] CUDA v4.0.1

julia> using CUDA
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CEnum/0gyUJ_ju0tv.so for CEnum [fa961155-64e5-5f13-b03f-caf6b980ea82]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Preferences/pWSk8_ju0tv.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/JLLWrappers/7Zgw7_ju0tv.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/LLVMExtra_jll/R9OeX_ju0tv.so for LLVMExtra_jll [dad2f222-ce93-54a1-a47d-0025e8a3acab]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/LLVM/e8NBy_ju0tv.so for LLVM [929cbde3-209d-540e-8aea-75f648917ca0]
└ @ Base loading.jl:1010
┌ Debug: Using LLVM 14.0.6 at /home/gridsan/<MY ID>/pkg/julia/julia-1.9/usr/bin/../lib/libLLVM-14jl.so
└ @ LLVM ~/.julia/packages/LLVM/s3bxG/src/LLVM.jl:86
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/ExprTools/eM8wu_ju0tv.so for ExprTools [e2ba6199-217a-4e67-a87a-7c52f15ade04]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/TimerOutputs/hd2yD_ju0tv.so for TimerOutputs [a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/GPUCompiler/yPwef_ju0tv.so for GPUCompiler [61eb1bfa-7361-4325-ad38-22787b887f55]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Adapt/rUIgN_ju0tv.so for Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Reexport/bTpYr_ju0tv.so for Reexport [189a3867-3050-52da-a836-e630ba90ab69]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/GPUArraysCore/qiYUe_ju0tv.so for GPUArraysCore [46192b85-c4d5-4398-a991-12ede77f4527]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Statistics/ERcPL_ju0tv.so for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/GPUArrays/v5u0T_ju0tv.so for GPUArrays [0c68f7d7-f131-5f86-a1c3-88cf8149b2d7]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Requires/IyxeS_ju0tv.so for Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/BFloat16s/iiZ8G_ju0tv.so for BFloat16s [ab4f0b2a-ad5b-11e8-123f-65d77653426b]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CUDA_Driver_jll/QJyk7_ju0tv.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:1010
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 11.4.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:76
┌ Debug: Forward-compatible CUDA driver found at /home/gridsan/<MY ID>/.julia/artifacts/b5e755e06f4d49a5ab1a638eea5d75bf20c66e3d/lib/libcuda.so; reported as version 11.8.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:127
┌ Debug: Successfully loaded forwards-compatible CUDA driver
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/9E4Mc/src/wrappers/x86_64-linux-gnu.jl:156
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CUDA_Runtime_jll/Hs50y_ju0tv.so for CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/RandomNumbers/pgCpR_ju0tv.so for RandomNumbers [e6cf234a-135c-5ec9-84dd-332b85af5143]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/Random123/1imiM_ju0tv.so for Random123 [74087812-796a-5b5d-8853-05524746bad3]
└ @ Base loading.jl:1010
┌ Debug: Requires conditionally ran code in 0.00303225 seconds: `RandomNumbers` detected `Random123`
└ @ Requires ~/.julia/packages/RandomNumbers/3pD1N/src/RandomNumbers.jl:38
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/AbstractFFTs/Di3HZ_ju0tv.so for AbstractFFTs [621f4979-c628-5d54-868e-fcf4e3e8185c]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/<MY ID>/.julia/compiled/v1.9/CUDA/oWw5k_ju0tv.so for CUDA [052768ef-5323-5732-b1bb-66c8b64840ba]
└ @ Base loading.jl:1010
┌ Error: No CUDA Runtime library found. This can have several reasons:
│ * you are using an unsupported platform: CUDA.jl only supports Linux (x86_64, aarch64, ppc64le), and Windows (x86_64).
│   refer to the documentation for instructions on how to use a custom CUDA runtime.
│ * you precompiled CUDA.jl in an environment where the CUDA driver was not available.
│   in that case, you need to specify (during pre compilation) which version of CUDA to use.
│   refer to the documentation for instructions on how to use `CUDA.set_runtime_version!`.
│ * you requested use of a local CUDA toolkit, but not all components were discovered.
│   try running with JULIA_DEBUG=CUDA_Runtime_Discovery for more information.
└ @ CUDA ~/.julia/packages/CUDA/ZdCxS/src/initialization.jl:77

@maleadt
Copy link
Member

maleadt commented Mar 16, 2023

Now that you can import CUDA, can you post Base.BinaryPlatforms.triplet(CUDA.CUDA_Runtime_jll.host_platform)?

@wsshin
Copy link
Author

wsshin commented Mar 16, 2023

The debug messages generated while adding CUDA#master were too long to post here. Here are the debug messages generated while importing CUDA#master:

(@v1.9) pkg> st CUDA
Status `~/.julia/environments/v1.9/Project.toml`
  [052768ef] CUDA v4.0.1 `https://github.com/JuliaGPU/CUDA.jl.git#master`

julia> using CUDA
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/CEnum/0gyUJ_ju0tv.so for CEnum [fa961155-64e5-5f13-b03f-caf6b980ea82]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/Preferences/pWSk8_ju0tv.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/JLLWrappers/7Zgw7_ju0tv.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/LLVMExtra_jll/R9OeX_ju0tv.so for LLVMExtra_jll [dad2f222-ce93-54a1-a47d-0025e8a3acab]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/LLVM/e8NBy_ju0tv.so for LLVM [929cbde3-209d-540e-8aea-75f648917ca0]
└ @ Base loading.jl:1010
┌ Debug: Using LLVM 14.0.6 at /home/gridsan/WO28768/pkg/julia/julia-1.9/usr/bin/../lib/libLLVM-14jl.so
└ @ LLVM ~/.julia/packages/LLVM/s3bxG/src/LLVM.jl:86
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/ExprTools/eM8wu_ju0tv.so for ExprTools [e2ba6199-217a-4e67-a87a-7c52f15ade04]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/TimerOutputs/hd2yD_ju0tv.so for TimerOutputs [a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/GPUCompiler/yPwef_ju0tv.so for GPUCompiler [61eb1bfa-7361-4325-ad38-22787b887f55]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/Adapt/rUIgN_ju0tv.so for Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/Reexport/bTpYr_ju0tv.so for Reexport [189a3867-3050-52da-a836-e630ba90ab69]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/GPUArraysCore/qiYUe_ju0tv.so for GPUArraysCore [46192b85-c4d5-4398-a991-12ede77f4527]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/Statistics/ERcPL_ju0tv.so for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/GPUArrays/v5u0T_ju0tv.so for GPUArrays [0c68f7d7-f131-5f86-a1c3-88cf8149b2d7]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/Requires/IyxeS_ju0tv.so for Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/BFloat16s/iiZ8G_ju0tv.so for BFloat16s [ab4f0b2a-ad5b-11e8-123f-65d77653426b]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/CUDA_Driver_jll/QJyk7_ju0tv.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:1010
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 11.4.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/2z0RQ/src/wrappers/x86_64-linux-gnu.jl:87
┌ Debug: Forward-compatible CUDA driver found at /home/gridsan/WO28768/.julia/artifacts/7a7fd08bbad6b42e7ed42fd2fc058e42039b075f/lib/libcuda.so; known to be version 12.1.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/2z0RQ/src/wrappers/x86_64-linux-gnu.jl:138
┌ Debug: Successfully loaded forwards-compatible CUDA driver
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/2z0RQ/src/wrappers/x86_64-linux-gnu.jl:167
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/CUDA_Runtime_jll/Hs50y_ju0tv.so for CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/RandomNumbers/pgCpR_ju0tv.so for RandomNumbers [e6cf234a-135c-5ec9-84dd-332b85af5143]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/Random123/1imiM_ju0tv.so for Random123 [74087812-796a-5b5d-8853-05524746bad3]
└ @ Base loading.jl:1010
┌ Debug: Requires conditionally ran code in 0.003785877 seconds: `RandomNumbers` detected `Random123`
└ @ Requires ~/.julia/packages/RandomNumbers/3pD1N/src/RandomNumbers.jl:38
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/AbstractFFTs/Di3HZ_ju0tv.so for AbstractFFTs [621f4979-c628-5d54-868e-fcf4e3e8185c]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/gridsan/WO28768/.julia/compiled/v1.9/CUDA/oWw5k_ju0tv.so for CUDA [052768ef-5323-5732-b1bb-66c8b64840ba]
└ @ Base loading.jl:1010

This seems working:

julia> CUDA.functional()
true

So CUDA#master works indeed, whereas CUDA@4.0.1 doesn't.

@wsshin
Copy link
Author

wsshin commented Mar 16, 2023

Now that you can import CUDA, can you post Base.BinaryPlatforms.triplet(CUDA.CUDA_Runtime_jll.host_platform)?

Not sure if you wanted this information for CUDA@4.0.1 or CUDA#master. I am currently at CUDA#master, and the requested information for it is as follows:

(@v1.9) pkg> st CUDA
Status `~/.julia/environments/v1.9/Project.toml`
  [052768ef] CUDA v4.0.1 `https://github.com/JuliaGPU/CUDA.jl.git#master`

julia> Base.BinaryPlatforms.triplet(CUDA.CUDA_Runtime_jll.host_platform)
"x86_64-linux-gnu-libgfortran4-cxx11-cuda+12.1-julia_version+1.9.0"

Do you also need the same information for CUDA@4.0.1? CUDA#master is working, but installing it took much longer than installing 4.0.1. If you don't need the information for 4.0.1, I would prefer keeping master...

@maleadt
Copy link
Member

maleadt commented Mar 16, 2023

You can just try to install 4.0.1 in a temporary environment, so ]activate --temp and add CUDA@4.0.1 there.
Also, the installation just took longer because of the initial download of CUDA 12, it won't download that again if you switch versions.

@wsshin
Copy link
Author

wsshin commented Mar 16, 2023

You can just try to install 4.0.1 in a temporary environment, so ]activate --temp and add CUDA@4.0.1 there.

That's a very useful trick! Thanks. Here is the result:

(jl_rEByw3) pkg> st CUDA
Status `/state/partition1/slurm_tmp/63373444.0.0/jl_rEByw3/Project.toml`
  [052768ef] CUDA v4.0.1

julia> using CUDA

julia> Base.BinaryPlatforms.triplet(CUDA.CUDA_Runtime_jll.host_platform)
"x86_64-linux-gnu-libgfortran4-cxx11-cuda+11.8-julia_version+1.9.0"

@wsshin
Copy link
Author

wsshin commented Mar 17, 2023 •
edited
Loading

Now even adding CUDA#master doesn't help. Interestingly, I get the following result:

julia> Base.BinaryPlatforms.triplet(CUDA.CUDA_Runtime_jll.host_platform)
"x86_64-linux-gnu-libgfortran4-cxx11-cuda+none-julia_version+1.9.0"

Notice that the version number is none rather than 11.8 or 12.1 reported earlier. What should I do?

[Update] After removing and adding CUDA#master a few times, the problem got resolved...

[Update 2] OK, something is weird. I am working on a Linux cluster managed by my organization. I connect to a login node, and from there I can request different types of compute nodes: CPU or GPU.

If I request a GPU node, start Julia, add CUDA#master, CUDA works fine on the GPU node. However, using CUDA on a CPU node generates an error. The CPU node does not have an nvidia GPU, so I cannot use the functionalities of CUDA there, but still, it should be possible to perform using CUDA on a non-GPU environment (and this was the case before) in order to run the same Julia script containing using CUDA on both the CPU and GPU nodes.

Conversely, if I request a CPU node, start Julia, add CUDA#master there, now using CUDA does not generate the error on the CPU node. However, this time using CUDA on a GPU node generates an error. It seems that CUDA installation depends on where it was added.

@maleadt
Copy link
Member

maleadt commented Mar 21, 2023

Ah, well, in that case the error is on your end. The error explains it:

│ * you precompiled CUDA.jl in an environment where the CUDA driver was not available.
│ in that case, you need to specify (during pre compilation) which version of CUDA to use.
│ refer to the documentation for instructions on how to use CUDA.set_runtime_version!.

@maleadt maleadt closed this as not planned Won't fix, can't repro, duplicate, stale Mar 21, 2023
@maleadt maleadt removed the bug Something isn't working label Mar 21, 2023
@wsshin
Copy link
Author

wsshin commented May 15, 2023

Because I was busy, I stopped solving this issue on Julia 1.9 release candidates. Since 1.9 was officially released, I revisited this issue on Julia 1.9.0, and the problem persisted.

Eventually I was able to resolve the issue by executing CUDA.set_runtime_version!() as recommended in the error message, but initially it was not clear what version number to use. I described the detailed steps in https://discourse.julialang.org/t/cuda-could-not-find-an-appropiate-cuda-runtime-to-use/97201/3.

@benz0li benz0li mentioned this issue Jun 19, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@maleadt @wsshin