I am using boto3 to read many text files in S3 through a Lambda Python function.
My codes for the connection to S3 below
config = Config( read_timeout=900, connect_timeout=900, retries={"max_attempts": 0} ) session = boto3.Session(region_name='ap-southeast-2') s3 = session.resource('s3', config=config) bucket = s3.Bucket(bucket) for obj in bucket.objects.all():
About 30 text files are successfully ok. But after, it gets below error message
[ERROR] 2021-09-01T09:35:08.628Z 6a5bf649-2057-43e4-a3eb-1b4438e582fb Exception on /process [POST] Traceback (most recent call last): File "/opt/python/urllib3/connection.py", line 169, in _new_conn conn = connection.create_connection( File "/opt/python/urllib3/util/connection.py", line 96, in create_connection raise err File "/opt/python/urllib3/util/connection.py", line 86, in create_connection sock.connect(sa) socket.timeout: timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/runtime/botocore/httpsession.py", line 323, in send urllib_response = conn.urlopen( File "/var/task/serverless_sdk/vendor/wrapt/wrappers.py", line 602, in __call__ return self._self_wrapper(self.__wrapped__, self._self_instance, File "/opt/python/urllib3/connection.py", line 353, in connect conn = self._new_conn() File "/opt/python/urllib3/connection.py", line 174, in _new_conn raise ConnectTimeoutError( urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPSConnection object at 0x7fe7b7a3e130>, 'Connection to xxxxxxxxxxxxxxxxxxxxxx.amazonaws.com timed out. (connect timeout=5)') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/tmp/sls-py-req/flask/app.py", line 2070, in wsgi_app response = self.full_dispatch_request() File "/var/runtime/botocore/httpsession.py", line 356, in send raise ConnectTimeoutError(endpoint_url=request.url, error=e) botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://xxxxxxxxxxxxxxxxx.s3.ap-southeast-2.amazonaws.com/?encoding-type=url"
Any way I can resolve this?
Advertisement
Answer
Lambda in a VPC does not have public IP and therefor can’t access internet from public subnets. Thus you can create NAT gateway in a public subnet, and place your lambda in private subnet. Once you setup route tables for any 0.0.0.0/0
connections in the private subnet to go to the NAT, your lambda will get internet access:
Instead, you can avoid using NAT and the internet with VPC interface endpoint for AWS Config.