Help Understanding Permission Denied Warning #793
Comments
It's not a warning, it's fatal. That's a bug in our output message. You can see that it's fatal from this report:
It's referencing the compilation of threading. You might be able to run just fine without threading. Did your eventual process work? The root cause appears to be a permission problem on the cluster when trying to compile with threading. Can you check that your process has permission to read that file and that it exists? Clusters are often much more locked down. They can also have very slow I/O. |
|
Thanks for the quick response, @bob-carpenter! Yes, I saw the I'm assuming that that path is relative to my
doesn't yield any results. Should it? I can find |
|
The path should be relative to wherever the command is running from. You don't need multi-threading to run Stan, which is why it's probably still working (and maybe that's why it was only listed as a WARNING? whatever the intent, that message was confusing). The I'm not sure about how the conda is going to be structured. @WardBrian is our conda expert and will probably know if this is a real error or red herring. |
|
This sounds like the kind of error that happens when cmdstan is being used from a read-only directory. We unfortunately do not support out-of-tree builds with our current build system. Are you installing cmdstan as part of the docker image build? That tends to leave things in unwritable folders by default at runtime. I am surprised at the fact that the model still builds -- when I try to mark such a file as not-writable, I get the same |
|
I saw a few similar threads regarding read-only directories. It does seem to have some parallels.
I can play around with the permissions in my Docker image to see if that helps resolve anything. I don't get the same error when running the container locally, however, so my guess would be it's something to do with permissions on the cluster. |
|
I was able to recreate locally and I can confirm a few things:
But!
Depending on how we fix this, it might mean a future version would really refuse to build (like I think you disabling |
|
All sounds good. Thank you both for your help! A quick fix I suppose could also be just installing cmdstanpy/cmdstan as part of my job rather than in the container. It would add a little bit of overhead but should ensure writability. I think I'll cross that bridge when I come to it though. If things are looking good for now, I'll just keep going as is. BTW, is there a recommended version of c++ and/or cmdstan that I could use that would not trigger all those deprecation warnings on compilation? |
|
Unfortunately those are in our dependency Boost and aren't avoidable at the moment. As a user, you are free to set I have been trying off-and-on-again to update Boost to make them go away -- my latest attempt is here |
|
Just an update: Adding the below line to my Dockerfile removed the 'fatal error' message.
So, confirming that it was indeed permissions on my end. My guess is that our cluster will not run as the root user by default in our images, meaning it does not have automatic write permissions in my environment. Adding the above grants those permissions to all users. |
|
Glad it is resolved! |
I'm trying to deploy a model on a cluster and running into some odd warnings. Specifically,
cmdstanpyreturns the below:I'm trying to understand why
compilation terminatedis just considered a warning. Even more confusing, the model seems to compile just fine--the code does not crash, but keeps going with sampling, which it certainly wouldn't do if compilation had failed.So, my questions are, what is this
compilation terminatedwarning talking about (I assume not my model?)? Should I be worried about it? How do I fix whatever is triggering it?I've copied the relevant stdout up to when sampling actually begins. I'm running with O1 optimizations and
STAN_THREADS=true; otherwise, no major deviations from the defaultsamplearguments.The text was updated successfully, but these errors were encountered: