I was using AWS glue python shell. The program uses multiple python libraries which not natively available for AWS. Glue can take .egg or .whl files for external library reference. All we need to do is put these .egg or .whl file in some S3 location and point to it using it’s full path. I tried with one external library [for instance openpyxl] and it worked. Now the problem is since I have multiple external libraries like pandas, numpy, openpyxl and pytz to be referred, I cant give full path of all these packages as only path can be specified as external python library reference. I tried giving the s3 folder name where I placed all these packages, it does not work.
How can I specify these multile .egg or .whl files so that my glue job can use them.
Advertisement
Answer
This question is already answered by gbeaven, but for some reasons I am unable mark it as answer. This was fixed by comma separating the file paths in the additional python modules.