Skip to content

Shard several of the most costly targets. #2373

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 13 commits into
base: develop
Choose a base branch
from

Conversation

shumway
Copy link
Collaborator

@shumway shumway commented Jun 19, 2025

Proposed changes

Note: This is a roll forward of #2266 which was rolled back in #2361.

We want to reduce build time for the CK library that is consumed by MIOpen.

Several of the build units (source files) take ~20 minutes to compile. To reduce compilation time, we split those files
out into multiple copies (build shards). There are three complications:

  1. The kernel type instantiations to be sharded are in std::tuple types. We don't to break up those tuples this finely.
    The solution we propose is to create a utility metafunction filter_tuple_by_modulo_t that splits up the kernel
    type tuples using a stride and offset.
  2. To avoid duplicating a lot of the instantiation code, template the instantiation functions on a shard and offset.
  3. To call the code, wrap the templated instantiation functions in the original instantiation function, and use
    extern template to make sure we don't reinstantiate these templates from the header code.

We partially automated this processes with code generation, by writing a CMake function
generate_sharded_instantiations to generate the source files for the shared instatiation functions and the calling
function directly from CMake.

I was missing two if-blocks from two template files in #2266, which causes some kernels to be omitted:

  • library/src/tensor_operation_instance/gpu/grouped_conv3d_fwd/xdl/comp/device_grouped_conv3d_fwd_xdl_ndhwgc_gkzyxc_ndhwgk_f16_comp_instance.in
  • library/src/tensor_operation_instance/gpu/grouped_conv3d_fwd/xdl/comp/device_grouped_conv3d_fwd_xdl_ndhwgc_gkzyxc_ndhwgk_f16_comp_instance.in

The failing test for gfx94x was disabled in MIOpen, but now I've checked this all in gfx94x and it passes now.

shumway and others added 11 commits June 17, 2025 21:16
Introduces a filter_tuple_by_modulo to break up tuples.

Drops build time of target from 21 minutes to under 14 minutes with 64
build processes, or 11 minutes with 128 build processes.

time ninja -j 64 device_grouped_conv3d_fwd_instance
I wasn't sure how to test the header-only instantiation code on my
initial commit. From Jenkins CI test results, I see that there is a
test target that depends on these headers:

ninja -j 128 test_grouped_convnd_fwd

This allowed me to test the build locally. I found three mistakes I
made, mostly related to early experiments on I tried on the code.
This was hard to find earlier because this PR is really too large.

I also discovered that there are five 2D convolution targets that now
dominate the compilation time. I will likely address those in a later
PR, rather than adding even more changes to this PR.
Our pattern for instantiating MIOpen templates uses duplicate
declarations (instead of headers). This is fragile, and I didn't
notice that my last commit had a bunch of link errors. I fixed these
mistakes, and the bin/test_grouped_conv_fwd test target binary now links
correctly.
Use a CMake function with template files to generate the source files for the
intantiating the kerenels and to generate the calling function.
Now that we have automated the shard instantiation, we can shard the 2D
convolution targets that take the longest to build. The target
test_grouped_conv2d_fwd now compiles in 15 minutes.
I used CMAKE_SOURCE_DIR to refer to the top-level source directory in
the ShardInstantiation.cmake file, but this can cause issues with
git submodules.  Instead, we should use PROJECT_SOURCE_DIR to ensure
compatibility when this project is used as a submodule in another
project.
Use a CMake function with template files to generate the source files for the
intantiating the kerenels and to generate the calling function.
Use a CMake function with template files to generate the source files for the
intantiating the kerenels and to generate the calling function.
Copy link
Contributor

@spolifroni-amd spolifroni-amd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really good in-code comments! Very clear!

@shumway shumway changed the title Shumway/refactor targets Shard several of the most costly targets. Jun 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants