Skip to content

[release/2.5] aten::copy optimization (revised) #2032

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 5 commits into
base: release/2.5
Choose a base branch
from
Open

[release/2.5] aten::copy optimization (revised) #2032

wants to merge 5 commits into from
+192 −42

Conversation

apakbin
Copy link

@apakbin apakbin commented Apr 15, 2025

This PR is the same as #1862 with an extra condition to make sure the new code path is not taken on an gfx1100 machine which failed during tests.

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Apr 15, 2025 •
edited
Loading

Jenkins build for 616d924f4f85aee911536bcc24184c774e190957 commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

Detected error during Pytorch building:

[7826/8684] Building HIPCC object caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/hip/torch_hip_generated_Sleep.hip.o
clang++: warning: argument unused during compilation: '--offload-compress' [-Wunused-command-line-argument]
[7827/8684] Building HIPCC object caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/hip/detail/torch_hip_generated_IndexUtils.hip.o
clang++: warning: argument unused during compilation: '--offload-compress' [-Wunused-command-line-argument]
[7828/8684] Building HIPCC object caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_Copy.hip.o
FAILED: caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_Copy.hip.o /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_Copy.hip.o 
cd /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip && /opt/conda/envs/py_3.10/bin/cmake -E make_directory /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/. && /opt/conda/envs/py_3.10/bin/cmake -D verbose:BOOL=OFF -D build_configuration:STRING=RELEASE -D generated_file:STRING=/var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/./torch_hip_generated_Copy.hip.o -P /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_Copy.hip.o.cmake
clang++: warning: argument unused during compilation: '--offload-compress' [-Wunused-command-line-argument]
/var/lib/jenkins/pytorch/aten/src/ATen/native/hip/Copy.hip:346:129: error: too few arguments to function call, expected 2, have 1
  346 |   } else if (is_permute_021(iter) && (dtype == kBFloat16 || dtype == kHalf) && !at::detail::getCUDAHooks().isGPUArch({"gfx1100"})) {
      |                                                                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~            ^

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Apr 15, 2025 •
edited
Loading

Jenkins build for d5407e4531d0137b2d02d1c4a9c1a51616e3b05b commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

Detected error during Pytorch building:

[7826/8684] Building HIPCC object caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/hip/torch_hip_generated_Sleep.hip.o
clang++: warning: argument unused during compilation: '--offload-compress' [-Wunused-command-line-argument]
[7827/8684] Building HIPCC object caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/hip/detail/torch_hip_generated_IndexUtils.hip.o
clang++: warning: argument unused during compilation: '--offload-compress' [-Wunused-command-line-argument]
[7828/8684] Building HIPCC object caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_CrossKernel.hip.o
FAILED: caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_CrossKernel.hip.o /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_CrossKernel.hip.o 
cd /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip && /opt/conda/envs/py_3.10/bin/cmake -E make_directory /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/. && /opt/conda/envs/py_3.10/bin/cmake -D verbose:BOOL=OFF -D build_configuration:STRING=RELEASE -D generated_file:STRING=/var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/./torch_hip_generated_CrossKernel.hip.o -P /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_CrossKernel.hip.o.cmake
clang++: warning: argument unused during compilation: '--offload-compress' [-Wunused-command-line-argument]
In file included from /var/lib/jenkins/pytorch/aten/src/ATen/native/hip/CrossKernel.hip:6:
In file included from /var/lib/jenkins/pytorch/aten/src/ATen/native/hip/Loops.cuh:14:
/var/lib/jenkins/pytorch/aten/src/ATen/native/hip/MemoryAccess.cuh:550:44: error: reference to type 'const std::vector<std::string>' (aka 'const vector<basic_string<char>>') could not bind to an lvalue of type 'c10::DeviceIndex' (aka 'signed char')

@apakbin
Copy link
Author

apakbin commented Apr 15, 2025

I just confirmed that this builds locally.

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Apr 16, 2025 •
edited
Loading

Jenkins build for d5407e4531d0137b2d02d1c4a9c1a51616e3b05b commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

Detected error during Pytorch building:

[7826/8684] Building HIPCC object caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/hip/torch_hip_generated_Sleep.hip.o
clang++: warning: argument unused during compilation: '--offload-compress' [-Wunused-command-line-argument]
[7827/8684] Building HIPCC object caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/hip/detail/torch_hip_generated_IndexUtils.hip.o
clang++: warning: argument unused during compilation: '--offload-compress' [-Wunused-command-line-argument]
[7828/8684] Building HIPCC object caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_CrossKernel.hip.o
FAILED: caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_CrossKernel.hip.o /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_CrossKernel.hip.o 
cd /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip && /opt/conda/envs/py_3.10/bin/cmake -E make_directory /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/. && /opt/conda/envs/py_3.10/bin/cmake -D verbose:BOOL=OFF -D build_configuration:STRING=RELEASE -D generated_file:STRING=/var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/./torch_hip_generated_CrossKernel.hip.o -P /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_CrossKernel.hip.o.cmake
clang++: warning: argument unused during compilation: '--offload-compress' [-Wunused-command-line-argument]
In file included from /var/lib/jenkins/pytorch/aten/src/ATen/native/hip/CrossKernel.hip:6:
In file included from /var/lib/jenkins/pytorch/aten/src/ATen/native/hip/Loops.cuh:14:
/var/lib/jenkins/pytorch/aten/src/ATen/native/hip/MemoryAccess.cuh:550:44: error: reference to type 'const std::vector<std::string>' (aka 'const vector<basic_string<char>>') could not bind to an lvalue of type 'c10::DeviceIndex' (aka 'signed char')

lcskrishna and others added 4 commits April 17, 2025 18:56
@lcskrishna @coderfeli
[release/2.5] aten::copy optimization. (#1862)
6eafbb3 
This PR cherry-picks all the aten::copy optimization from 2.5_perf_fix
branch into 2.5 release branch.
cc: @coderfeli

---------

Co-authored-by: feli <[email protected]>
Co-authored-by: coderfeli <[email protected]>
@apakbin
Added a guard to not run the optimization on gfx1100 as it is incompa…
622393d 
…tible.
@apakbin
[ROCm] code cleanup of architecture checks (pytorch#150473)
70146d5 
This PR replaces several calls to `at::cuda::getCurrentDeviceProperties()->gcnArchName` and `at::cuda::getDeviceProperties(device_index)->gcnArchName` when checking to see if the GPU architecture is in a certain list.

Pull Request resolved: pytorch#150473
Approved by: https://github.com/jeffdaily, https://github.com/cyyever
@apakbin
pulled latest changes, one PR made in between was using the old signa…
6b8f6cf 
…ture.
@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Apr 17, 2025 •
edited
Loading

Jenkins build for 6b8f6cf245754ac855302db81a2685ae0ecc737a commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

pruthvistony
pruthvistony requested changes Apr 17, 2025
View reviewed changes
@pruthvistony
Copy link
Collaborator

Upstream this change only for MI300 or only MI hardware.
Skip this for all Navi.

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Apr 21, 2025 •
edited
Loading

Jenkins build for 6b8f6cf245754ac855302db81a2685ae0ecc737a commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Apr 21, 2025 •
edited
Loading

Jenkins build for 6b8f6cf245754ac855302db81a2685ae0ecc737a commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Apr 22, 2025 •
edited
Loading

Jenkins build for 6b8f6cf245754ac855302db81a2685ae0ecc737a commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

Jenkins build for ae9b843fda825a78eedef6dca97bc1852a73e40e commit is in progress
Links: Blue Ocean view / Build artifacts

@apakbin apakbin marked this pull request as ready for review April 25, 2025 17:56
@rocm-repo-management-api
Copy link

Jenkins build for ae9b843fda825a78eedef6dca97bc1852a73e40e commit is in progress
Links: Blue Ocean view / Build artifacts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pruthvistony pruthvistony pruthvistony requested changes

@lcskrishna lcskrishna Awaiting requested review from lcskrishna

@jerrymannil jerrymannil Awaiting requested review from jerrymannil

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@apakbin @pruthvistony @lcskrishna