SIGSEGV in java application while attempting to c2 compile ApiCallAttemptTimeoutTrackingStage

[roysti10]

Describe the bug

We are running java application with java version 1.8.0_72 b15 and sdk version of

<dependency>
  <groupId>software.amazon.awssdk</groupId>
  <artifactId>dynamodb</artifactId>
  <version>2.27.18</version>
</dependency>

SIGSEGV stack trace:
Stack: [0x00007f8bbc5dd000,0x00007f8bbc6dd000], sp=0x00007f8bbc6d9ca0, free space=1011k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xa22fc1] Symbol::increment_refcount()+0x1
V [libjvm.so+0x440c4d] ciBytecodeStream::get_method(bool&, ciSignature**)+0x44d
V [libjvm.so+0x5699f8] Parse::do_call()+0x58
V [libjvm.so+0x95184a] Parse::do_one_bytecode()+0x330a
V [libjvm.so+0x943ab8] Parse::do_one_block()+0x178
V [libjvm.so+0x943d37] Parse::do_all_blocks()+0x127
V [libjvm.so+0x948259] Parse::Parse(JVMState*, ciMethod*, float)+0x7f9
V [libjvm.so+0x3f1e38] ParseGenerator::generate(JVMState*)+0x88
V [libjvm.so+0x4a2d3a] Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool)+0x127a
V [libjvm.so+0x3f0808] C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0x198
V [libjvm.so+0x4ad37a] CompileBroker::invoke_compiler_on_method(CompileTask*)+0xc9a
V [libjvm.so+0x4ae326] CompileBroker::compiler_thread_loop()+0x5d6
V [libjvm.so+0xa69dcf] JavaThread::thread_main_inner()+0xdf
V [libjvm.so+0xa69efc] JavaThread::run()+0x11c
V [libjvm.so+0x91d9d8] java_start(Thread*)+0x108
C [libpthread.so.0+0x744b] start_thread+0xdb

Current CompileTask:
C2:149433579 31924 ! 4 software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage::execute (104 bytes)

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Current Behavior

Reproduction Steps

This happens randomly and has no fixed pattern

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.27.18

JDK version used

1.8.0_72 b15

Operating System and version

Linux 4.14.326-245.539.amzn2.x86_64

[debora-ito]

Hi @roysti10 thank you for raising this.

• How often do you see the error, can you estimate a failures/request rate?
• Can you share a code sample showing how you use the DynamoDB client, and any other logs/stacktrace at the time of the error?
• Which API is called when the error happens?

I understand this happens at random and there's no fixed pattern, but in order to investigate this further we need more information, a repro code would be ideal.

[debora-ito]

Another question: do you have the same code running in a different JDK version ? Does it throw SIGSEGV too? We're trying to narrow down if this can be a problem with JDK 8.

[roysti10]

Hi @debora-ito , Apologies for the delay

1. This happens once or twice per week on multiple nodes

2 & 3. I have shared the thread dump on the issue, this is all we get, we have tried analyzing the core dump but it doesnt provide much input on where the issue occured, It occurs on ApiCallAttemptTimeoutTrackingStage which when debugged gets called on any dynamo db operation

We are currently running a POC with coretto 1.8_382 , will update you whether we can reproduce it there , we believe it to be a issue with 1.8_72 as C2 compiler errors seem to be common with this build but we cant find any issue which links it to the aws sdk package

[debora-ito]

Yes, let us know what you find.

[debora-ito]

@roysti10 have you had the same SIGSEGV error with corretto 1.8_382?

[github-actions]

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.