-
Notifications
You must be signed in to change notification settings - Fork 632
Fix misdesign in unordered_map_concurrent that broke it for unordered_map #2454
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…_map Background: unordered_concurrent_map<> is a thread-safe map that is parameterized by the type of the underlying "shard" maps. The default underlying map type is std::unordered map. Though for a while, as used by ImageCache, we have been using robin_map as the underlying map type, having found it has superior performance. PR 2314 improved performance even more by taking advantage of the fact that robin_map has a find(key,hash) that lets you supply the precomputed hash value (in situations where you already had it), thus avoiding redundant hash computations. BUT... we failed to notice that unordered_map doesn't have this kind of find() variety, and so even though the u_c_m is templated on the map type, this meant that it would no longer work with std::unordered_map. That's especially painful since the underlying type defaults to this. The solution, in this patch, is to make a helper function `find_with_hash(map, key, hash)` that uses the hash if the map type has a find(key,hash) method, and otherwise falls back to the standard find(key). There is some template metaprogramming involved, and we add a couple of handy idioms to platform.h (it sort of belongs there because the way to express the idioms there depends on whether it's C++ 11 or 14+, and platform.h is where we sort that kind of thing out).
Works for me. Thanks |
Are you sure this was the original problem reported? Maybe @sambler was using a version of robin-map that didn't have the required extension to |
No, same version of robin-map. The problem was that in Cycles, they use our unordered_map_concurrent, except instead of specifying a robin map, they just let it default to std::unordered_map, and that lacks the 2-argument version of find. So when they were trying to update Debian's version of OIIO, it broke the Cycles build. |
I have yet to build 2.1.x for Fedora just yet but planning on doing it soon. Should I apply this as a patch? We do use Cycles with Blender. |
@hobbes1069 Yes, please wait until this is merged, then either apply it as a patch, or wait for me to tag a 2.1 release that incorporates it. Not just cycles, but anybody else who might use our u_m_c. It's not part of OIIO's public API per se, but we make the header public just because it's useful. It just didn't occur to us that we were breaking it when it used the default underlying map type, that was unintentional. |
Oh I see it now, didn't read the original mail carefully enough. LGTM - though its worth noting that cycles should probably consider using |
That may be true, but it's beyond my pay grade to suggest Cycles changes and also robin map is only used by OIIO at build time, not distributed with our public headers, so Cycles or other downstream users would need to make it a dependency, etc. I actually rather like the fact that now u_m_c can work seamlessly with map types that support find(key,prehash) as well as those that only have find(key), without hard coding to any one particular map. Though I wish C++ had a straightforward built-in way to ask if a particular method is present in a class, without having to set up multiple layers of enable_if nonsense. |
…_map (AcademySoftwareFoundation#2454) Background: unordered_concurrent_map<> is a thread-safe map that is parameterized by the type of the underlying "shard" maps. The default underlying map type is std::unordered map. Though for a while, as used by ImageCache, we have been using robin_map as the underlying map type, having found it has superior performance. PR 2314 improved performance even more by taking advantage of the fact that robin_map has a find(key,hash) that lets you supply the precomputed hash value (in situations where you already had it), thus avoiding redundant hash computations. BUT... we failed to notice that unordered_map doesn't have this kind of find() variety, and so even though the u_c_m is templated on the map type, this meant that it would no longer work with std::unordered_map. That's especially painful since the underlying type defaults to this. The solution, in this patch, is to make a helper function `find_with_hash(map, key, hash)` that uses the hash if the map type has a find(key,hash) method, and otherwise falls back to the standard find(key). There is some template metaprogramming involved, and we add a couple of handy idioms to platform.h (it sort of belongs there because the way to express the idioms there depends on whether it's C++ 11 or 14+, and platform.h is where we sort that kind of thing out).
…_map (AcademySoftwareFoundation#2454) Background: unordered_concurrent_map<> is a thread-safe map that is parameterized by the type of the underlying "shard" maps. The default underlying map type is std::unordered map. Though for a while, as used by ImageCache, we have been using robin_map as the underlying map type, having found it has superior performance. PR 2314 improved performance even more by taking advantage of the fact that robin_map has a find(key,hash) that lets you supply the precomputed hash value (in situations where you already had it), thus avoiding redundant hash computations. BUT... we failed to notice that unordered_map doesn't have this kind of find() variety, and so even though the u_c_m is templated on the map type, this meant that it would no longer work with std::unordered_map. That's especially painful since the underlying type defaults to this. The solution, in this patch, is to make a helper function `find_with_hash(map, key, hash)` that uses the hash if the map type has a find(key,hash) method, and otherwise falls back to the standard find(key). There is some template metaprogramming involved, and we add a couple of handy idioms to platform.h (it sort of belongs there because the way to express the idioms there depends on whether it's C++ 11 or 14+, and platform.h is where we sort that kind of thing out).
…_map (AcademySoftwareFoundation#2454) Background: unordered_concurrent_map<> is a thread-safe map that is parameterized by the type of the underlying "shard" maps. The default underlying map type is std::unordered map. Though for a while, as used by ImageCache, we have been using robin_map as the underlying map type, having found it has superior performance. PR 2314 improved performance even more by taking advantage of the fact that robin_map has a find(key,hash) that lets you supply the precomputed hash value (in situations where you already had it), thus avoiding redundant hash computations. BUT... we failed to notice that unordered_map doesn't have this kind of find() variety, and so even though the u_c_m is templated on the map type, this meant that it would no longer work with std::unordered_map. That's especially painful since the underlying type defaults to this. The solution, in this patch, is to make a helper function `find_with_hash(map, key, hash)` that uses the hash if the map type has a find(key,hash) method, and otherwise falls back to the standard find(key). There is some template metaprogramming involved, and we add a couple of handy idioms to platform.h (it sort of belongs there because the way to express the idioms there depends on whether it's C++ 11 or 14+, and platform.h is where we sort that kind of thing out).
…_map (AcademySoftwareFoundation#2454) Background: unordered_concurrent_map<> is a thread-safe map that is parameterized by the type of the underlying "shard" maps. The default underlying map type is std::unordered map. Though for a while, as used by ImageCache, we have been using robin_map as the underlying map type, having found it has superior performance. PR 2314 improved performance even more by taking advantage of the fact that robin_map has a find(key,hash) that lets you supply the precomputed hash value (in situations where you already had it), thus avoiding redundant hash computations. BUT... we failed to notice that unordered_map doesn't have this kind of find() variety, and so even though the u_c_m is templated on the map type, this meant that it would no longer work with std::unordered_map. That's especially painful since the underlying type defaults to this. The solution, in this patch, is to make a helper function `find_with_hash(map, key, hash)` that uses the hash if the map type has a find(key,hash) method, and otherwise falls back to the standard find(key). There is some template metaprogramming involved, and we add a couple of handy idioms to platform.h (it sort of belongs there because the way to express the idioms there depends on whether it's C++ 11 or 14+, and platform.h is where we sort that kind of thing out).
Background: unordered_concurrent_map<> is a thread-safe map that is
parameterized by the type of the underlying "shard" maps. The default
underlying map type is std::unordered map. Though for a while, as used
by ImageCache, we have been using robin_map as the underlying map
type, having found it has superior performance.
PR #2314 improved performance even more by taking advantage of the fact
that robin_map has a find(key,hash) that lets you supply the
precomputed hash value (in situations where you already had it), thus
avoiding redundant hash computations.
BUT... we failed to notice that unordered_map doesn't have this kind
of find() variety, and so even though the u_c_m is templated on the
map type, this meant that it would no longer work with
std::unordered_map. That's especially painful since the underlying
type defaults to this.
The solution, in this patch, is to make a helper function
find_with_hash(map, key, hash)
that uses the hash if the map typehas a find(key,hash) method, and otherwise falls back to the standard
find(key).
There is some template metaprogramming involved, and we add a couple
of handy idioms to platform.h (it sort of belongs there because the way
to express the idioms there depends on whether it's C++ 11 or 14+, and
platform.h is where we sort that kind of thing out).