Increasing the timeout for InvokeEndpoint

[velociraptor111]

The current timeout for InvokeEndpoint is 60 seconds as specified here: https://docs.aws.amazon.com/en_pv/sagemaker/latest/dg/API_runtime_InvokeEndpoint.html

Is there any way we can increase this limit, to say 120 seconds?

***EDIT

Just to be clear, I was able to keep the process on the server running by passing an environment variable in the Model definition like so

 model = MXNetModel(..., env = {'SAGEMAKER_MODEL_SERVER_TIMEOUT' : '300' })

Through CloudWatch, I was able to confirm that the task is still running even after 60 seconds. (For my usecase, I am processing a video frame by frame) My question is however, on the client side I am receiving this kind of error due to the timeout

An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (0) from model with message "Your invocation timed out while waiting for a response from container model. Review the latency metrics for each container in Amazon CloudWatch, resolve the issue, and try again.". See https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logEventViewer:group=/aws/sagemaker/Endpoints/nightingale-pose-estimation in account 552571371228 for more information.: ModelError

[wenzhaoAtAws]

Sagemaker model hosting engineer here. Thanks for your interest in our product!

With respect to your question -- currently, it is not possible to increase the the 60 seconds timeout.

[velociraptor111]

@wenzhaoAtAws Are there any plans in the future to allow customers to increase inference timeout?

Also, I notice that in the response object of

response = sagemaker_client.invoke_endpoint(EndpointName='pose-estimation',Body=request_body)
print(response)

This is the log

{'ResponseMetadata': {'RequestId': 'a0343e2a-5390-4af0-a7fd-ef63d576ca45', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'a0343e2a-5390-4af0-a7fd-ef63d576ca45', 'x-amzn-invoked-production-variant': 'AllTraffic', 'date': 'Tue, 12 Nov 2019 01:30:32 GMT', 'content-type': 'application/json', 'content-length': '18'}, 'RetryAttempts': 2}, 'ContentType': 'application/json', 'InvokedProductionVariant': 'AllTraffic', 'Body': <botocore.response.StreamingBody object at 0x10c60e358>}

Is it possible for me to make the RetryAttempts to 0?

[sanjams2]

For disabling retries, you should be able to do something like the following (please note I havent tested this code myself, it serves as a reference):

import boto3
from botocore.config import Config
from sagemaker.session import Session

config = Config(
    read_timeout=80,
    retries={
        'max_attempts': 0
    }
)
sagemaker_runtime_client = boto3.client('sagemaker-runtime', config=config)
sagemaker_client = Session(sagemaker_runtime_client=sagemaker_runtime_client)

See:

• https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

• sagemaker-python-sdk/src/sagemaker/session.py

  Lines 119 to 125 in 04a2e75

  	if sagemaker_runtime_client is not None:
  	self.sagemaker_runtime_client = sagemaker_runtime_client
  	else:
  	config = botocore.config.Config(read_timeout=80)
  	self.sagemaker_runtime_client = self.boto_session.client(
  	"runtime.sagemaker", config=config
  	)

In regards to the feature request, one option is to use SageMaker's batch transform option (https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html). May not fit your use case though...

[hlzl]

@ajaykarpur Any update on this? Switching to batch transforms doesn‘t seem to be interesting for video input.

[tomaszdudek7]

I agree 60s limit is quite low and I hope they will slightly bump it.

If you need more than 1 minute (and less than 15) you might be interested in the newest SageMaker offering, namely Asynchronous Inference. The client sends the payload to the endpoint and the result will eventually appear in specified S3 bucket. You can set up SNS or Lambda to inform the client that it is ready to consume (and for example generate S3 presigned URL in the process).

More on that here: https://aws.amazon.com/about-aws/whats-new/2021/08/amazon-sagemaker-asynchronous-new-inference-option/

[lminer]

FYI Even using the async endpoint, I'm still getting timed out and the env variable mentioned at the top of this thread doesn't exist any more. This issue really needs to be fixed.

If anyone wants to know how to fix this, I create by own docker image from the default inference. Then I slightly modified serve.py and nginx.conf.template so that they had higher timeout values for gunicorn, tensorflow-serving, and nginx. Then I copied them over the default files.

[whh14]

hi @lminer, could you elaborate how to modify the nginx.conf.template please? many thanks

[lminer]

@whh14 I changed it to this. Probably overkill!

load_module modules/ngx_http_js_module.so;

worker_processes auto;
daemon off;
pid /tmp/nginx.pid;
error_log  /dev/stderr %NGINX_LOG_LEVEL%;

worker_rlimit_nofile 4096;

events {
  worker_connections 2048;
}

http {
  include /etc/nginx/mime.types;
  default_type application/json;
  access_log /dev/stdout combined;
  js_include tensorflow-serving.js;
  fastcgi_read_timeout %MODEL_TIMEOUT%;
  proxy_read_timeout %MODEL_TIMEOUT%;

  upstream tfs_upstream {
    %TFS_UPSTREAM%;
  }

  upstream gunicorn_upstream {
    server unix:/tmp/gunicorn.sock fail_timeout=1;
  }

  server {
    listen %NGINX_HTTP_PORT% deferred;
    client_max_body_size 0;
    client_body_buffer_size 100m;
    subrequest_output_buffer_size 100m;

    set $tfs_version %TFS_VERSION%;
    set $default_tfs_model %TFS_DEFAULT_MODEL_NAME%;

    location /tfs {
        rewrite ^/tfs/(.*) /$1  break;
        proxy_redirect off;
        proxy_pass_request_headers off;
        proxy_set_header Content-Type 'application/json';
        proxy_set_header Accept 'application/json';
        proxy_pass http://tfs_upstream;
        proxy_read_timeout %MODEL_TIMEOUT%;
        proxy_connect_timeout %MODEL_TIMEOUT%;
        proxy_send_timeout %MODEL_TIMEOUT%;
        send_timeout %MODEL_TIMEOUT%;
    }

    location /ping {
        %FORWARD_PING_REQUESTS%;
    }

    location /invocations {
        %FORWARD_INVOCATION_REQUESTS%;
    }

    location /models {
        proxy_pass http://gunicorn_upstream/models;
        proxy_read_timeout %MODEL_TIMEOUT%;
        proxy_connect_timeout %MODEL_TIMEOUT%;
        proxy_send_timeout %MODEL_TIMEOUT%;
        send_timeout %MODEL_TIMEOUT%;
    }

    location / {
        return 404 '{"error": "Not Found"}';
    }

    keepalive_timeout 3;
  }
}

[whh14]

Hi @lminer , Thanks a lot for your help. I have made those changes, and I also made some changes to the serve.py, but I am still having the timeout issue, (it timed out after 30 second while uploading data from inference.py to the model. Could you show me what modification you did to the serve.py file? Many thanks.

[lminer]

@whh14 I set the --timeout flag to gunicorn and the --rest_api_timeout_in_ms to tensorflow-serving. That being said, if it's timing out while uploading data to the model, then it seems like you have other problems you need to solve!

[n0thing233]

any plan to work on this in your 2022 roadmap?

[mullermp]

Please add configuration for this!