windows openCL compilation issue

[bachlaw]

hello, I discussed with Rok on Discourse that I was having similar problems compiling openCL models on Windows that he had earlier addressed in Linux. https://discourse.mc-stan.org/t/opencl-cmdstanr-windows-error/22375/4

I took his advice to apply these commands:

cmdstan_make_local(cpp_options = "CXXFLAGS += -fpermissive")
rebuild_cmdstan(cores = 4)

But unfortunately now there is a new error and according to the error message, I am supposed to file a bug report. So here goes. Continuing on with the steps described in the Discourse post, I now get this error:

In file included from stan/lib/stan_math/stan/math/opencl/prim.hpp:86,
                 from stan/lib/stan_math/stan/math/prim.hpp:7,
                 from stan/src/stan/io/dump.hpp:7,
                 from src/cmdstan/command.hpp:24,
                 from src/cmdstan/main.cpp:1:
stan/lib/stan_math/stan/math/opencl/scalar_type.hpp:14:8: warning: partial specialization of 'struct stan::scalar_type<T, typename std::enable_if<stan::math::conjunction<stan::is_kernel_expression_and_not_scalar<T, void> >::value, void>::type>' after instantiation of 'struct stan::scalar_type<stan::math::constant_<int>, void>' [-fpermissive]
 struct scalar_type<T, require_all_kernel_expressions_and_none_scalar_t<T>> {
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from stan/src/stan/model/model_header.hpp:16:
stan/src/stan/services/util/create_rng.hpp:35:1: internal compiler error: in gt_pch_note_object, at ggc-common.c:276
 }  // namespace stan
 ^
libbacktrace could not find executable to open
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://github.com/r-windows> for instructions.
mingw32-make.exe: *** [make/program:25: stan/src/stan/model/model_header_opencl.hpp.gch] Error 1
Error: An error occured during compilation! See the message above for more information.

[rok-cesnovar]

Thanks for the report, can you try by running:

cmdstan_make_local(cpp_options = list("CXXFLAGS += -fpermissive", "PRECOMPILED_HEADERS"=FALSE))
rebuild_cmdstan(cores=4)

[bachlaw]

Unfortunately, after following the instructions above, we are back to another error:

Compiling Stan program...
In file included from stan/lib/stan_math/stan/math/opencl/prim.hpp:86,
                 from stan/lib/stan_math/stan/math/prim.hpp:7,
                 from stan/src/stan/io/dump.hpp:7,
                 from src/cmdstan/command.hpp:24,
                 from src/cmdstan/main.cpp:1:
stan/lib/stan_math/stan/math/opencl/scalar_type.hpp:14:8: warning: partial specialization of 'struct stan::scalar_type<T, typename std::enable_if<stan::math::conjunction<stan::is_kernel_expression_and_not_scalar<T, void> >::value, void>::type>' after instantiation of 'struct stan::scalar_type<stan::math::constant_<int>, void>' [-fpermissive]
 struct scalar_type<T, require_all_kernel_expressions_and_none_scalar_t<T>> {
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
C:/rtools40/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lOpenCL
collect2.exe: error: ld returned 1 exit status
mingw32-make.exe: *** [make/program:62: C:\Users\bachl\AppData\Local\Temp\Rtmp2tjaKh\model-401466f33b64.exe] Error 1
Error: An error occured during compilation! See the message above for more information.

The script I am using now is this:

library(cmdstanr)
check_cmdstan_toolchain(fix = TRUE)

cmdstan_make_local(cpp_options = list("CXXFLAGS += -fpermissive", "PRECOMPILED_HEADERS"=FALSE))
rebuild_cmdstan(cores=4)

# Generate some fake data
n <- 250000
k <- 20
X <- matrix(rnorm(n * k), ncol = k)
y <- rbinom(n, size = 1, prob = plogis(3 * X[,1] - 2 * X[,2] + 1))
mdata <- list(k = k, n = n, y = y, X = X)

# Compile the model with STAN_OPENCL=TRUE
mod_cl <- cmdstan_model("bernoulli_logit_glm.stan"
                        ,
                        cpp_options = list(stan_opencl = TRUE)
                        )

[rok-cesnovar]

Ok, we are getting somewhere.

cannot find -lOpenCL

This one is Windows specific and this instruction is missing from the vignette for Windows. Sorry about that. The location of the OpenCL.lib file must be either in the PATH environemnt variable or the LDFLAGS

path_to_opencl_lib <- "C:/Program Files (x86)/OCL_SDK_Light/lib/x86_64"
cpp_options = list(
    "CXXFLAGS += -fpermissive",
    "PRECOMPILED_HEADERS"=FALSE,
    "LDFLAGS+= -L\"",path_to_opencl_lib,"\" -lOpenCL"
    )
cmdstan_make_local(cpp_options = cpp_options)
rebuild_cmdstan(cores=4)

and just replace path_to_opencl_lib with the location of the OpenCL.lib file. For OCL_SDK_Light its typically C:/Program Files (x86)/OCL_SDK_Light/lib/x86_64, for NVIDIA CUDA its something like C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.2/lib/x64 (depending on the cuda version). Will update the vignette.

[bachlaw]

Thanks Rok. To avoid having the LDFLAGS statement end up being three non-sensical lines that break the compilation, you need to either use a combination of single and double quotes or, even easier, put a paste0() around the line.

The good news is that these cpp_options seem to compile acceptably via both a fresh install of install_cmdstan or rebuild_cmdstan if you take that route instead. The bad news is that we are back to the original error. Here is what I get after running the following:

library(cmdstanr)
check_cmdstan_toolchain(fix = TRUE)

path_to_opencl_lib <- "C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.3/lib/x64"
cpp_options = list(
  "CXXFLAGS += -fpermissive",
  "PRECOMPILED_HEADERS"=FALSE,
  paste0("LDFLAGS+= -L\"",path_to_opencl_lib,"\" -lOpenCL")
)

install_cmdstan(cores=4, overwrite = TRUE, cpp_options = cpp_options)

# cmdstan_make_local(cpp_options = cpp_options)
# rebuild_cmdstan(cores=4)

# Generate some fake data
n <- 250000
k <- 20
X <- matrix(rnorm(n * k), ncol = k)
y <- rbinom(n, size = 1, prob = plogis(3 * X[,1] - 2 * X[,2] + 1))
mdata <- list(k = k, n = n, y = y, X = X)

# Compile the model with STAN_OPENCL=TRUE
mod_cl <- cmdstan_model("bernoulli_logit_glm.stan"
                        ,
                        cpp_options = list(stan_opencl = TRUE)
                        )

The readout from Stan to failure is:

Compiling Stan program...
In file included from stan/lib/stan_math/stan/math/opencl/prim.hpp:86,
                 from stan/lib/stan_math/stan/math/prim.hpp:7,
                 from stan/src/stan/io/dump.hpp:7,
                 from src/cmdstan/command.hpp:24,
                 from src/cmdstan/main.cpp:1:
stan/lib/stan_math/stan/math/opencl/scalar_type.hpp:14:8: warning: partial specialization of 'struct stan::scalar_type<T, typename std::enable_if<stan::math::conjunction<stan::is_kernel_expression_and_not_scalar<T, void> >::value, void>::type>' after instantiation of 'struct stan::scalar_type<stan::math::constant_<int>, void>' [-fpermissive]
 struct scalar_type<T, require_all_kernel_expressions_and_none_scalar_t<T>> {
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from stan/src/stan/model/model_header.hpp:16:
stan/src/stan/services/util/create_rng.hpp:35:1: internal compiler error: in gt_pch_note_object, at ggc-common.c:276
 }  // namespace stan
 ^
libbacktrace could not find executable to open
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://github.com/r-windows> for instructions.
mingw32-make.exe: *** [make/program:25: stan/src/stan/model/model_header_opencl.hpp.gch] Error 1
Error: An error occured during compilation! See the message above for more information.

[rok-cesnovar]

Unfortunately right now I think you need to run

install_cmdstan(cores=4, overwrite = TRUE)

path_to_opencl_lib <- "C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.3/lib/x64"
cpp_options = list(
  "CXXFLAGS += -fpermissive",
  "PRECOMPILED_HEADERS"=FALSE,
  paste0("LDFLAGS+= -L\"",path_to_opencl_lib,"\" -lOpenCL")
)

cmdstan_make_local(cpp_options = cpp_options)
rebuild_cmdstan(cores=4)

The problem is that install_cmdstan overwrite PRECOMPILED_HEADERS on windows, even if the user has it set in cpp_options (https://github.com/stan-dev/cmdstanr/blob/master/R/install.R#L157).

There is a fix for this that can be installed with

remotes::install_github("stan-dev/cmdstanr@install_precompiled_headers_fix")

[bachlaw]

It works! Using the vignette example I get a 6x speedup over single-thread chains, while TBB within-thread parallelization generates just under a 2x speedup versus single-thread chains. Particularly notable that this is using an ancient GTX 760, so in theory lots of room for improvement.

[rok-cesnovar]

Thanks @bachlaw for the patience and report!

Will close once we update the vignette and the fix for the cpp_options in install_cmdstan.

[rok-cesnovar]

Reopening this because we need to add windows instructions to the vignette.