Serialized result of java Class & Record is different with RedisTemplate

[doljae]

Describe the bug
Serialized result of java Class & Record is different with RedisTemplate.

Version information
2.13.3

To Reproduce

====== RedisConfiguration ======
@Configuration
@AutoConfigureAfter(RedisAutoConfiguration.class)
@ConditionalOnClass(RedisOperations.class)
public class RedisConfiguration {

    @Bean
    public RedisTemplate<String, ?> jacksonRedisTemplate(RedisConnectionFactory redisConnectionFactory) {
        final RedisSerializer<String> stringRedisSerializer = new StringRedisSerializer();

        final RedisTemplate<String, ?> template = new RedisTemplate<>();
        template.setConnectionFactory(redisConnectionFactory);

        final GenericJackson2JsonRedisSerializer jackson2JsonRedisSerializer = genericJackson2JsonRedisSerializer();

        template.setKeySerializer(stringRedisSerializer);
        template.setValueSerializer(jackson2JsonRedisSerializer);
        template.setHashKeySerializer(stringRedisSerializer);
        template.setHashValueSerializer(jackson2JsonRedisSerializer);

        return template;
    }

    @Bean
    public GenericJackson2JsonRedisSerializer genericJackson2JsonRedisSerializer() {
        final ObjectMapper mapper = new ObjectMapper()
            .registerModule(new ParameterNamesModule(JsonCreator.Mode.DEFAULT))
            .registerModule(new Jdk8Module())
            .registerModule(new JavaTimeModule())
            .configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false)
            .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);

        mapper.activateDefaultTyping(mapper.getPolymorphicTypeValidator(),
                                     DefaultTyping.NON_FINAL,
                                     As.PROPERTY);

        return new GenericJackson2JsonRedisSerializer(mapper);
    }
}

====== RequestDto ======
@Data
@NoArgsConstructor
@AllArgsConstructor
public class RequestDto {

    private Long id;
    private String name;
    private OffsetDateTime createdAt;
}

====== RequestRecord ======
public record RequestRecord(Long id,
                            String name,
                            OffsetDateTime createdAt) {
}


====== Test Code ======
@SpringBootTest
class RedisRecordTest {

    @Autowired
    private RedisTemplate<String, RequestRecord> redisRecordTemplate;
    @Autowired
    private RedisTemplate<String, RequestDto> redisDtoTemplate;
    @Autowired
    private RedisTemplate<String, String> redisStringTemplate;
    @Autowired
    private RedisTemplate<String, Object> redisObjectTemplate;
    @Autowired
    private ObjectMapper mapper;

    @DisplayName("record -> String -> set() -> get() -> String -> record -> O")
    @Test
    void test() throws JsonProcessingException {
        final RequestRecord record = new RequestRecord(1L, "doljae", OffsetDateTime.now());
        final String serialized = mapper.writeValueAsString(record);
        System.out.println(serialized);
        redisStringTemplate.opsForValue().set("key", serialized);

        final String fromRedis = redisStringTemplate.opsForValue().get("key");
        System.out.println(fromRedis);
        final RequestRecord deserialized = mapper.readValue(fromRedis, RequestRecord.class);
        System.out.println(deserialized);
    }

    @DisplayName("Class -> set() -> get() -> Class -> O")
    @Test
    void test2() {
        final RequestDto dto = new RequestDto(1L, "doljae", OffsetDateTime.now());
        redisDtoTemplate.opsForValue().set("key", dto);
        /*
         redis-cli
         get key
         -> "{\"@class\":\"com.example.redis.RequestDto\",\"id\":1,\"name\":\"doljae\",\"createdA\":\"2022-06-08T21:39:01.166705+09:00\"}"
         */

        final RequestDto fromRedis = redisDtoTemplate.opsForValue().get("key");
        System.out.println(fromRedis);
    }

    @DisplayName("record -> set() -> get() -> record -> X")
    @Test
    void test3() {
        final RequestRecord record = new RequestRecord(1L, "seokjae", OffsetDateTime.now());
        redisRecordTemplate.opsForValue().set("key", record);
        /*
         redis-cli
         get key
         -> "{\"id\":1,\"name\":\"doljae\",\"createdAt\":\"2022-06-08T21:40:36.057138+09:00\"}"
         */

        // Caused by: com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve subtype of [simple type, class java.lang.Object]: missing type id property '@class'
        // at [Source: (byte[])"{"id":1,"name":"doljae","createdAt":"2022-06-08T21:40:36.057138+09:00"}"; line: 1, column: 72]
        final RequestRecord deserialized = redisRecordTemplate.opsForValue().get("key");
    }
}

Expected behavior
Both class and record objects should be stored and deserialized normally in RedisTemplate.

Additional context

• The issue occurred while using Spring Data Redis class, but since ObjectMapper does serialization and deserialization, I reported it to Jackson issue borad.

• You can reproduce this issue with this repository's test class (use redis docker image in repo/docker/docker-compose.yml)

  • https://github.com/doljae/toy
  • https://github.com/doljae/toy/blob/master/japring/src/test/java/com/example/redis/RedisRecordTest.java

• I found that if i adjust ObjectMapper Configuration, both Class & Record serialize & deserialize normally.

  mapper.activateDefaultTyping(mapper.getPolymorphicTypeValidator(),
                                     DefaultTyping.EVERYTHING,
                                     As.PROPERTY);

• but this way exposes every meta data of object's fields...

[yawkat]

Can you give a test case that does not use spring?

But my guess is that DefaultTyping.NON_FINAL is the issue – records are final.

[doljae]

@yawkat

I write a test case without using Spring, same ObjectMapper configuration.

class MapperTest {

    @Test
    void test() throws JsonProcessingException {
        final ObjectMapper mapper = new ObjectMapper()
            .registerModule(new ParameterNamesModule(JsonCreator.Mode.DEFAULT))
            .registerModule(new Jdk8Module())
            .registerModule(new JavaTimeModule())
            .configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false)
            .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);

        mapper.activateDefaultTyping(mapper.getPolymorphicTypeValidator(),
                                     DefaultTyping.NON_FINAL,
                                     As.PROPERTY);

        final RequestDto dto = new RequestDto(1L, "doljae", OffsetDateTime.now());
        final RequestRecord record = new RequestRecord(1L, "doljae", OffsetDateTime.now());

        final byte[] dtoBytes = mapper.writeValueAsBytes(dto);
        final byte[] recordBytes = mapper.writeValueAsBytes(record);
        
        // how to print the result?...
        System.out.println(Arrays.toString(dtoBytes));
        System.out.println(Arrays.toString(recordBytes));
    }

}

[doljae]

@yawkat

Could you please explain in more detail about ParameterNamesModule and activateDefaultTyping?

The ObjectMapper setting I am using is written with reference to the reference, so I do not fully understand the operation.

As you said, these settings seem to be the clues to the problem, and I googled and looked at the documentation, but I didn't quite understand.

Could you please explain in more detail? Or if you have a reference link that explains it in detail, please share.

[yawkat]

i can reproduce your issue with the new test case, and yes it is because of DefaultTyping.NON_FINAL. Changing it to DefaultTyping.EVERYTHING resolves this. However it is somewhat problematic because it also adds default typing to unrelated places like the Long field.

A better alternative is to use mapper.writerFor(Object.class).writeValueAsBytes(record). This adds the type information even in the NON_FINAL case. However I can't say how to make spring use this approach.

[doljae]

@yawkat

Spring Data Redis uses ObjectMapper in GenericJackson2JsonRedisSerializer.

RedisTemplate just uses configured Serializer & Deserializer. Inside it, it use ObjectMapper's serialization & deserialization methods.

Could you please explain in more detail about ParameterNamesModule and activateDefaultTyping?
Plus, I am hoping to use it regardless of class or record under a common setting. In this situation, please suggest me how to get the ObjectMapper configuration.

[cowtowncoder]

@doljae getting the ObjectMapper configuration is something that Spring Data Redis users or maintainers can help with.

I think your problem can be divided in two parts:

1. How to make Jackson (without considering Redis template) to work the way you want -- if possible, keep in mind that Polymorphic Deserialization by definition may expose internal aspects that differ between differ object types
2. How to configure framework around to use settings.

But one other thing I would STRONGLY recommend: instead of attempting to serialize any given "Root value" directly, you really should instead use a wrapper if possible: something like:

public class Wrapper {
   @JsonTypeInfo(....)
   public Object wrapped; // or whatever name

   // and/or getters, setters for accessing wrapped value 
}

This will avoid many pitfalls, including problem of Java Type Erasure for the root value.
It may also remove the need to active default typing completely (although this depends a little bit on kind of data you serialize).

In your case, for example, one problem is that Record types are final and non-polymorphic, and as such inclusion of type id may not occur as you'd expect, if such value is serialized directly.

[doljae]

@cowtowncoder , @yawkat

I improved my understanding by looking for settings of ObjectMapper and writing test codes.

• doljae/toy@136e853

I still don't quite understand the principle. In other words, I did not fully understand exactly what the role of the ObjectMapper settings used in this comment.

However, as @yawkat said, I understood that root class information was not saved when saving a Java Record as a RedisTemplate and then immediately taking it out as a Record. (Because Java Record is treated as a final class and due to the setting of ObjectMapper, final class information is saved in redis without remaining when serialized.)

  mapper.activateDefaultTyping(mapper.getPolymorphicTypeValidator(),
                                     DefaultTyping.NON_FINAL,
                                     As.PROPERTY); //  <- this is the point

Here's what I can do in this situation:

1. Using DefaultTyping.EVERYTHING

Of course, this method is not recommended by @cowtowncoder. However, it is also the simplest way to solve it. And I'm not sure if it's good to use the Wrapper class just for this problem.

2. Fetch the value as a String and deserialize it when retrieving it via RedisTemplate.

I think this method is the most flexible. If you take it out as a String and deserialize it through ObjectMapper, there is no problem because the target class is explicitly declared.

final String fromRedis = redisStringTemplate.opsForValue().get("key");
final RequestRecord deserialized = mapper.readValue(fromRedis, RequestRecord.class);

3, Using a regular class instead of a Java Record

In this way, you do not need to modify ObjectMapper related settings. I just can't use Java Record...

---

For reference, I chose method 3 in my current project (because I had to make a quick fix).

And on the Spring Data Redis side, from the latest version, it was changed to use DefaultTyping.EVERYTHING as the ObjectMapper setting that is used by default in GenericJackson2JsonRedisSerializer.

• Add support for final/Kotlin classes in GenericJackson2JsonRedisSerializer [DATAREDIS-995] spring-projects/spring-data-redis#1566

[doljae]

By the way, although I'm not a good English speaker, I couldn't find any really good material about detailed settings related to ObjectMapper and Jackson. (But there is a possibility that I did not understand it well)

If you have a reference that you can recommend, please share it. The more detailed and the more examples, the better.

[cowtowncoder]

Sorry, I do not think there is much good documentation regarding this particular configuration (default typing).

But one quick comment. This:

  mapper.activateDefaultTyping(mapper.getPolymorphicTypeValidator(),
                                     DefaultTyping.NON_FINAL,
                                     As.PROPERTY); //  <- this is the point

is something I strongly suggest you change, you should usually not use As.PROPERTY with Default Typing.
Use As.WRAPPER_ARRAY (or even As.WRAPPER_OBJECT). Those will work with all types, are more efficient and simpler. As.PROPERTY is ok with @JsonTypeInfo, for types serialized as JSON Objects.

This may not be the main issue you are facing but I would avoid this setting anyway.

However, I do have one question: why do you not try the approach I suggested as the best solution:

public class Wrapper {
   @JsonTypeInfo(include=JsonTypeInfo.As.WRAPPER_ARRAY,  use = JsonTypeInfo.Id.CLASS)
   public Object wrapped; // or whatever name

   // and/or getters, setters for accessing wrapped value 
}```

Then you would wrap value yourself on read/write: you would not need default typing; it would be guaranteed that Polymorphic Type Id was always written and so on.
This is what I would try myself.

[doljae]

@cowtowncoder
Thank you for your answer. I answer your question.

To sum it up

As, the more general setting you recommended.It is better to use AS.WRAPPER_OBJECT. However, in many projects, it seems that AS.PROPERTY is being used a lot.

I think it is more convenient to create and use a Wrapper class and a DTO class without additional annotation setting after setting up a common ObjectMapper (limited to RedisTemplate).

---

Use As.WRAPPER_ARRAY (or even As.WRAPPER_OBJECT). Those will work with all types, are more efficient and simpler. As.PROPERTY is ok with @JsonTypeInfo, for types serialized as JSON Objects.

Thank you for your advice. I understood that using WRAPPER_OBJECT is more universally available than using PROPERTY.
I don't think this change will cause a side effects. Because the ObjectMapper does serialization and de-serialization, so they will do serialization and de-serialization according to the setting. However, I think it needs to be checked because it was used as a common setting for the project.

However, the ObjectMapper setting that I wrote in the comment seems to be the setting that I had been using before I participated in the project. Not only that, but it seems to be used in many places. Even the Spring Data Redis project's GenericJackson2JsonRedisSerializer uses this setting as its default.

This setting that I applied to ObjectMapper is not the global ObjectMapper setting of the project, but the ObjectMapper setting of Serializer, Deserializer used by RedisTemplate of Spring Data Redis. I'll post a PR on the Spring Data Redis side about this.

However, I do have one question: why do you not try the approach I suggested as the best solution:

This is just because it is convenient not to use the Wrapper class.
For example, suppose have a Target class and a Wrapper class, I can use it like the code snippet you suggested.

    @Test
    void test11() throws JsonProcessingException {
        final RequestRecord targetClass = new RequestRecord(1L, "doljae", OffsetDateTime.now());
        final RequestWrapper wrapperClass = new RequestWrapper(targetClass);

        final ObjectMapper mapper = new ObjectMapper();
        mapper.registerModule(new ParameterNamesModule(JsonCreator.Mode.DEFAULT))
              .registerModule(new Jdk8Module())
              .registerModule(new JavaTimeModule())
              .configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false)
              .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
              .configure(DeserializationFeature.ADJUST_DATES_TO_CONTEXT_TIME_ZONE, false);
        
        final String serialized = mapper.writeValueAsString(wrapperClass);
        System.out.println(serialized);

        final RequestWrapper requestWrapper = mapper.readValue(serialized, RequestWrapper.class);
        final RequestRecord wrapped = requestWrapper.getWrapped();
        System.out.println(wrapped);
    }

As in the sample code above, serialization and de-serialization using ObjectMapper directly work well without any problems.

But the first direction I was going to do was to set and get objects directly to RedisTemplate without any preprocessing. And to use this way, eventually, I need to provide information so that the ObjectMapper can be deserialized when serializing to the Wrapper class as well. That is, the identifier information should be serialized together.

    @DisplayName("record -> set() -> get() -> record -> X")
    @Test
    void test3() {
        final RequestRecord targetClass = new RequestRecord(1L, "doljae", OffsetDateTime.now());
        final RequestWrapper wrapperClass = new RequestWrapper(targetClass);
        redisWrapperTemplate.opsForValue().set("key", wrapperClass);

        // error!, cause RedisTemplate's deserializer does not know the wrapper class information
        final RequestWrapper deserialized = redisWrapperTemplate.opsForValue().get("key");
        
        final ObjectMapper mapper = new ObjectMapper();
        .......
        // works, cause we gives class information to ObjectMapper
        final RequestWrapper requestWrapper = mapper.readValue(serialized, RequestWrapper.class);
        .......
    }

To get and set objects directly to RedisTemplate, information about the Wrapper class must also be provided when serializing. In other words, @JsonTypeInfo should be used for the Wrapper class.

I think this part is a matter of style. If the type I use as a target class is an abstract class and I need to apply polymorphism, this approach would be fine. However, the method you suggested should only use the Wrapper class for DTO to deal with DTO, and even add additional annotations to DTO.

On the other hand, the way to take the common ObjectMapper setting is to write code to the convention of our commonly used DTO class without separate Wrapper class and annotation.

[cowtowncoder]

Not quite sure what would be needed, and since Record-type handling has been improved a lot since 2.13, closing as there's no action planned for now.

May be re-filed with more specific request for change.