Python generate_blob_sas doesn’t create right SAS token (failed on copy operation in azcopy)

It’s my first question at such cool resource.

Description of a goal: I need to copy from one Azure storage account to another. … Not only this, but this part doesn’t work for me now.

Problem: If generate token with generate_blob_sas you will have in azcopy operation:

INFO: Authentication failed, it is either not correct, or expired, or does not have the correct permission -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go@v0.13.1-0.20210914164749-2d6cd3e07548/azblob/zc_storage_error.go:42 ===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) ===== Description=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

If insert manually created token on Azure portal – works fine.

Why I use azcopy inside of Python, because I have structure like this:

.
|
|└── Folder1
|    ├── file1.json
|    ├── file2.json
|    ├── file3.json
|    ├── file4.json
|    ├── file5.son
|    └── Folder2
|        ├── fileswithsomenames1.html
|        ├── fileswithsomenames2.html
|        ├── fileswithsomenames3.html
|        ├── fileswithsomenames4.html
|        ├── fileswithsomenames5.html
|        ├── fileswithsomenames6.html
|        ├── fileswithsomenames7.html
|        ├── fileswithsomenames8.html
|        ├── fileswithsomenames9.html
|        └── fileswithsomenames10.html
└──receipt.json (this file will be eachtime but with different content)

A number of files will be different in each situation. And I need to work ahead with files from target blob. I didn’t find understandable description how to copy from store1/conteiner1/{var.name}-blob to var.name-store2/conteainer1/ all content including many folders. Almost all descriptions and manuals for 1 blob file.

My code:

code to authorise and get storage account name and string 
...
customer_name = 'abc-qa'
...
container_name_source = "container1"
blob_name_source = customer_name+"-blobfolder1/blobfolder2"

container_name_target = customer_name+"-bl"
blob_name_target = "/"

# Function to generate sas for source blob
def get_blob_sas_source(account_name_source,account_key_1_source, container_name_source, blob_name_source):
    sas_blob_source = generate_blob_sas(account_name=account_name_source, 
                                container_name=container_name_source,
                                blob_name=blob_name_source,
                                account_key=account_key_1_source,
                                permission=BlobSasPermissions(read=True),
                                expiry=datetime.utcnow() + timedelta(hours=1))
    return sas_blob_source

sas_blob_source = get_blob_sas_source(account_name_source,account_key_1_source, container_name_source, blob_name_source)
url_blob_source = 'https://'+account_name_source+'.blob.core.windows.net/'+container_name_source+'/'+blob_name_source+'?'+sas_blob_source

# Function to generate sas for target blob
def get_blob_sas_target(account_name_target,account_key_1_target, container_name_target, blob_name_target):
    sas_blob_target = generate_blob_sas(account_name=account_name_target, 
                                container_name=container_name_target,
                                blob_name=blob_name_target,
                                account_key=account_key_1_target,
                                permission=BlobSasPermissions(add=True, create=True, write=True, tag=False, delete_previous_version=True),
                                expiry=datetime.utcnow() + timedelta(hours=1))
    return sas_blob_target

sas_blob_target = get_blob_sas_target(account_name_target,account_key_1_target, container_name_target, blob_name_target)
print(sas_blob_target)
url_blob_target = 'https://'+account_name_target+'.blob.core.windows.net/'+container_name_target+'/?'+sas_blob_target




#copy seed-data folder from account_name_source/container1/customer_name+"-blobfolder1/blobfolder2" to account_name_target/customer_name+"-bl"
cmd = f"azcopy copy '{url_blob_source}' '{url_blob_target}' --recursive=True"
print(str(cmd))
results = subprocess.run(
    str(cmd), shell=True, universal_newlines=True, check=True)
print(results.stdout)

Result if gen manually ‘sp=rl&st=2022-04-14T14:29:48Z&se=2022-04-14T22:29:48Z&spr=https&sv=2020-08-04&sr=c&sig={token}’

Rusult if gen over generate_blob_sas

‘se=2022-04-14T17%3A18%3A48Z&sp=r&sv=2020-10-02&sr=b&sig={token}’

Answer

The reason why your code is failing with the SAS token generated programmatically is because it is missing List permission. If you look at the working SAS token, it has both Read and List permissions (sp=rl) whereas the non-working SAS token only has Read permission (sp=r).

Another issue is that in your code you are generating SAS token for the blob (sr=b) whereas the working SAS token is for the container (sr=c). This is for both source and target SAS token.

To fix this:

Create the SAS token with both Read and List permissions.
Create the SAS token for the container instead of blob. For this, you will need to use generate_container_sas.

Once you do these two things, you should not get the error.

Advertisement

Answer