botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL

I am using boto3 to read many text files in S3 through a Lambda Python function.

My codes for the connection to S3 below

config = Config(
            read_timeout=900,
            connect_timeout=900,
            retries={"max_attempts": 0}
        )
        session = boto3.Session(region_name='ap-southeast-2')

        s3 = session.resource('s3', config=config)
        bucket = s3.Bucket(bucket)
        for obj in bucket.objects.all():

About 30 text files are successfully ok. But after, it gets below error message

[ERROR] 2021-09-01T09:35:08.628Z    6a5bf649-2057-43e4-a3eb-1b4438e582fb    Exception on /process [POST]
Traceback (most recent call last):
  File "/opt/python/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/opt/python/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/opt/python/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/runtime/botocore/httpsession.py", line 323, in send
    urllib_response = conn.urlopen(
  File "/var/task/serverless_sdk/vendor/wrapt/wrappers.py", line 602, in __call__
    return self._self_wrapper(self.__wrapped__, self._self_instance,
  File "/opt/python/urllib3/connection.py", line 353, in connect
    conn = self._new_conn()
  File "/opt/python/urllib3/connection.py", line 174, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPSConnection object at 0x7fe7b7a3e130>, 'Connection to xxxxxxxxxxxxxxxxxxxxxx.amazonaws.com timed out. (connect timeout=5)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/sls-py-req/flask/app.py", line 2070, in wsgi_app
    response = self.full_dispatch_request()
  File "/var/runtime/botocore/httpsession.py", line 356, in send raise ConnectTimeoutError(endpoint_url=request.url, error=e) botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://xxxxxxxxxxxxxxxxx.s3.ap-southeast-2.amazonaws.com/?encoding-type=url"

Any way I can resolve this?

Answer

Lambda in a VPC does not have public IP and therefor can’t access internet from public subnets. Thus you can create NAT gateway in a public subnet, and place your lambda in private subnet. Once you setup route tables for any 0.0.0.0/0 connections in the private subnet to go to the NAT, your lambda will get internet access:

• How do I give internet access to a Lambda function that’s connected to an Amazon VPC?

Instead, you can avoid using NAT and the internet with VPC interface endpoint for AWS Config.