Shard several of the most costly targets.

[shumway]

Proposed changes

We want to reduce build time for the CK library that is consumed by MIOpen.

Several of the build units (source files) take ~20 minutes to compile. To reduce compilation time, we split those files
out into multiple copies (build shards). There are three complications:

1. The kernel type instantiations to be sharded are in std::tuple types. We don't to break up those tuples this finely.
   The solution we propose is to create a utility metafunction filter_tuple_by_modulo_t that splits up the kernel
   type tuples using a stride and offset.
2. To avoid duplicating a lot of the instantiation code, template the instantiation functions on a shard and offset.
3. To call the code, wrap the templated instantiation functions in the original instantiation function, and use
   extern template to make sure we don't reinstantiate these templates from the header code.

We partially automated this processes with code generation, by writing a CMake function
generate_sharded_instantiations to generate the source files for the shared instatiation functions and the calling
function directly from CMake.

Checklist

Please put an x into the boxes that apply. You can also fill these out after creating the PR. If you're not sure, please don't hesitate to ask.

• I have added tests relevant to the introduced functionality, and the unit tests are passing locally
• I have added inline documentation which enables the maintainers with understanding the motivation
• I have run clang-format on all changed files
• Any dependent changes have been merged

Discussion

I went with this solution because the strategy of breaking down tuples into smaller tuples wasn't going to
get us all the individual kernels. This was the best compromise I could find between vebosity and getting
build performance. Now we have about 165 targets for this target, which shard well on 128 shards:

time ninja device_grouped_conv3d_fwd_instance

This reduces build time from this target from about 22 minutes to 11 minutes.

[spolifroni-amd]

doesn't look like there's anything for docs.

[bartekxk]

Hi @shumway , great job. Is it possible to generate these c++ files using some python script? At now it is impossible to check/review if each instance has been added correctly. It will be much easier to handle such python generator instead of 150 files. It will also be very useful to port such python script to generate c++ files for bwd data and bwd weight

[shumway]

Hi @shumway , great job. Is it possible to generate these c++ files using some python script? At now it is impossible to check/review if each instance has been added correctly. It will be much easier to handle such python generator instead of 150 files. It will also be very useful to port such python script to generate c++ files for bwd data and bwd weight

@bartekxk and I chatted, and I'll look at existing codegen inside the CK codebase to see how to set this up so that it matches what's already here.

[shumway]

@bartekxk 's suggestion to use codegen looks like a good idea. I have started to implement this as a cmake function, using configure_file. This has the advantage that it's pure CMake and we'll get back to essentially no extra hand-written source files. The input template generalizes a lot, but it make take me another day to get all the codegen rules sorted out and replace everything.

[shumway]

This code is much simpler now, down to 35 files changed. I followed @bartekxk 's suggestion and converted to code
generation. WIth this simplification, I can now shard the 2D convolution targets that are limiting the build time.

[bartekxk]

lgtm

[bartekxk]

Could you just make sanity check and verify if test_grouped_convnd_fwd generates the same output on the develop and on your branch?