502 Error for Large files

I am trying to load a very large file using a library partially built by Jesse Wilson.

 

I am using the attached code, but am receiving the following error after about 40 minutes.  For smaller files (files that take less than 30 minutes, everything works fine)  

 

502 Server Error: Bad Gateway for url: https://api.anaplan.com/2/0/workspaces/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/models/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/files/xxxxxxxxxxxx/chunks/3

 

This is after receiving these messages:

 

xxxxxxxxxxxxxxxxxxxxxxx.pem
<class 'str'>
Token validated
Token validated
None
Uploading chunk 1, Status: 204
Uploading chunk 1, Status: 204
Token validated
Token validated
None
Uploading chunk 2, Status: 204
Uploading chunk 2, Status: 204
Token validated
Token validated
None
Uploading chunk 3, Status: 204
Uploading chunk 3, Status: 204
Token validated
Token validated

 

I don't understand why it is giving me a 502 error in the same spot.  I can manually upload the file in anaplan.  I am re-authenticating, and then testing my authentication. It should work.  

 

Because the connection times out after 30 minutes, this makes using APIs for large files almost impossible.   

 

 

 

Answers

  • Was able to fix the main prloblem.  

    I am still receiving a 500 error, however, with really large files.

     

    Below is the fix I implemented.

     

    def flat_file_upload(conn, fileId, chunkSize, file):
    '''
    :param conn: AnaplanConnection object which contains authorization string, workspace ID, and model ID
    :param fileId: ID of the file in the Anaplan model
    :param chunkSize: Desired size of the chunk, in megabytes
    :param file: Path to the local file to be uploaded to Anaplan
    '''

    #Setting local variables for connection details
    authorization = conn.authorization
    workspaceGuid = conn.workspaceGuid
    modelGuid = conn.modelGuid

    #Restrict users from entering a value for chunkSize greater than 50mb to prevent issues with API server
    if chunkSize > 50:
    return "Chunk size must be 50mb or less."
    else:

    #Assigning the size of the local file in MB
    file_size = os.stat(file).st_size / __BYTES__
    file_data = ""
    post_header = {
    "Authorization": authorization,
    "Content-Type":"application/json"
    }
    put_header = {
    "Authorization": authorization,
    "Content-Type":"application/octet-stream"
    }
    file_metadata_start = {
    "id":fileId,
    "chunkCount":-1
    }
    file_metadata_complete = {
    "id":fileId,
    "chunkCount": file_size / (__BYTES__ * chunkSize)
    }
    url = __base_url__ + "/" +workspaceGuid + "/models/" + modelGuid + "/files/" + fileId

    try:
    start_upload_post = requests.post(url, headers=post_header, json=file_metadata_start)
    start_upload_post.raise_for_status()
    except HTTPError as e:
    raise HTTPError(e)
    #Confirm that the metadata update for the requested file was OK before proceeding with file upload
    if start_upload_post.ok:
    complete = True #Variable to track whether file has finished uploaded
    with open(file, "rt") as f: #Loop through the file, reading a user-defined number of bytes until complete
    chunkNum = 0
    while True:
    buf=f.readlines(__BYTES__ * chunkSize)
    if not buf:
    break
    file_data = "" ##Added to create new file_data for each new load
    for item in buf:
    file_data += item
    try:
    authorization_verify = anaplan_auth.verify_auth_flat_file_upload(authorization)
    print(authorization_verify)
    authorization = anaplan_auth.refresh_token(authorization) #adding re-authorization for larger files
    authorization_verify = anaplan_auth.verify_auth_flat_file_upload(authorization)
    print(authorization_verify)
    put_header = {
    "Authorization": authorization,
    "Content-Type":"application/octet-stream"
    }
    file_upload = requests.put(url + "/chunks/" + str(chunkNum), headers=put_header, data=file_data)
    print(file_upload.raise_for_status())
    file_upload.raise_for_status()
    print("Uploading chunk " + str(chunkNum + 1) +", Status: " + str(file_upload.status_code))
    except HTTPError as e:
    raise HTTPError(e)
    print("Uploading chunk " + str(chunkNum + 1) +", Status: " + str(file_upload.status_code))
    if not file_upload.ok:
    complete = False #If the upload fails, break the loop and prevent subsequent requests. Print error to screen
    print("Error " + str(file_upload.status_code) + '\n' + file_upload.text)
    break
    else:
    chunkNum += 1
    if complete:
    complete_upload = requests.post(url + "/complete", headers=post_header, json=file_metadata_complete)
    if complete_upload.ok:
    return "File upload complete, " + str(chunkNum) + " chunk(s) uploaded to the server."
    else:
    return "There was an error with your request: " + str(complete_upload.status_code) + " " + complete_upload.text
    else:
    return "There was an error with your request: " + str(start_upload_post.status_code) + " " + start_upload_post.text

  • Hello,

     

    Did you have a fix on this issue? Facing same issue but no response from anyone and couldn't find this topic troubleshooting anywhere.

  • @Hawkes06 

     

    I found a solution.  I posted my updated code, although I am still struggling with a 500 error for larger files.