Highlighted
Contributor

Uploading file Via REST API and Triggering Imports

Hi All -

 

I am working on a small POC to do the following:

 

  1. Create a simple list in anaplan
  2. Import data into the list via file upload.
  3. Permission on import/upload file will be sent to "Admin Only".
  4. Import/upload will be done initially with my user account (ie damian.shameer@xcompany.com)
  5. Create a systematic/batched process within my org to periodically do the following:
    1. Upload data to Anaplan into file created in #2
    2. Trigger the import process created in #2 to load the data from the upload file
    3. Upload and Import will be done via the REST APIs and not Anaplan Connect.
  6. A special data integration account, ie my_aplan_integ@xcompany.com, will be used for #5(I cannot used my login account for security reasons). The data integration account will be added as a Work Space Admin to the applicable Workspace/Model.

 

I have completed #1-4 with out any issues.

 

clipboard_image_2.png

 

clipboard_image_3.png

 

clipboard_image_4.png

 

#5 is also complete. I can upload a file via the REST API and trigger the Import Task created in #2 with my integration account.

 

How the POC was tested:

*Special integration account  used for all REST API calls

 

1. Perform import with 100K rows via Anaplan UI with my login id. Verify list contains 100K rows

2. Create an Action to truncate the list, ie, "Delete All Data From List"

3. Execute "Delete All Data From List" via the REST API call to clear list--> completed with 200 return code . Verified in Anaplan as well

4. Create a text file with 2 rows (structurally the same as original upload file).

5. Upload file via REST API using the File ID (retrieved by getting list of all Server Files) --> completed with 204 return code

6. Trigger Import task via REST API --> completed with 200 return code but not as expected

 

Expected result: List will now contain 2 rows

Actual result: Import process picked up the file I uploaded via the User Interface.

13:25:26.916 [main] INFO com.fw. - successRowCount - 166598
13:25:26.916 [main] INFO com.fw.x - successCreateCount - 166598

 

What could be missing here?

 

@Ernie_Goff 

@Fwolf 

 

Additional Details since Anaplan will not accept my replies (made 3 replies, none showing up):

 

This is a PUT request hence the response (as per standards) should be 204 and not 200. See https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/204

 

I am following this API Request: Upload a single chunk

 

Example from Documentation. I also tried the option to upload by chunks (Set Chunk count, upload 1 chunk, set upload as complete...Still didn't work).

 

 

 

curl -X PUT \
htt ps://api.anaplan.com/2/0/workspaces/8a8196b15b7dbae6015b8694411d13fe/models/75A40874E6B64FA3AE0743278996850F/files/113000000008 \
-H 'authorization: AnaplanAuthToken {auth_token}' \
-H "Content-Type:application/octet-stream" --upload-file Tests.txt

 

 

 

My API call:

 

Request: URL: htt ps:  //api.anaplan.com/2/0/workspaces/x/models/x/files/113000000039

 

File to be uploaded:

File uploaded via browser had 160K rows. THis file will be smaller

 

 

ID,Timestamp,Email,Description
1,7/1/2019 9:55,test.user2@comap.com,Change Line Item
2,7/1/2019 9:58,test.user2@comap.com,Change Module
19,7/1/2019 11:03,test.user2@comap.com,Change Line Item Write Access Driver
20,7/1/2019 11:03,test.user2@comap.com,Change Line Item Read Access Driver
21,7/1/2019 11:03,test.user2@comap.com,Change Line Item Read Access Driver
22,7/1/2019 11:03,test.user2@comap.com,Change Line Item Write Access Driver

 

 

Upload file details (from REST API call)

 

ServerFileData: id: 113000000039 name: Activity_log_4_160K-20190915.csv code: null chunkCount: 1 origin: null format: txt language: null country: null encoding: ISO-8859-1 separator: , delimiter: " headerRow: 1 firstDataRow: 2 files[2]: 

 

 

Import Details:

 

imports[1] id: 112000000038 | name: Activity Log from Activity_log_4_160K-20190915.csv | code: null| importType: HIERARCHY_DATA | importDataSourceId: 113000000039 

 

 

Java Code:

 

    static void uploadFile(CloseableHttpClient client, String apiURL, String authToken, String fileNameWithPath) {

        // My URL: URL: htt ps://api.anap lan.com/2/0/workspaces/x/models/x/files/113000000039
     
   // Sample (Docs, Unload as single chunk): htt ps://api.anapla n.com/2/0/workspaces/8a8196b15b7dbae6015b8694411d13fe/models/75A40874E6B64FA3AE0743278996850F/files/113000000008

        System.out.println("URL: " + apiURL);

        StopWatch stopWatch = new StopWatch();
        stopWatch.start();

        try {
            HttpPut request = new HttpPut(apiURL);
            request.setHeaders(getHeadersForUpload(authToken, false));

            File source = new File(fileNameWithPath);

            if (source == null || !source.exists() || !source.isFile() || !source.canRead()) {
                throw new RuntimeException("Input file must be a valid file. File must be read-able. Check Source file. ");
            }

            byte[] byte_data = org.apache.commons.io.FileUtils.readFileToByteArray(source);

            if (byte_data == null || byte_data.length == 0) {
                //TODO determine if this should be allowed
                throw new RuntimeException("Source File seems to be empty. No Data to upload. ");
            }

            System.out.println("Data Size (mb): "+byte_data.length /1024);

            // create data stream and set to request
            InputStreamEntity entity = new InputStreamEntity(new ByteArrayInputStream(byte_data), byte_data.length);
            request.setEntity(entity);

            System.out.println("Begin File upload...");

            HttpResponse response = client.execute(request);
            stopWatch.stop();
            System.out.println("Upload request done. Took: " + stopWatch.toString());

            int responseCode = response.getStatusLine().getStatusCode();
            String respText = response.getEntity() != null ? EntityUtils.toString(response.getEntity()) : "[no response text]";
            System.out.println("Response Code: " + responseCode);

            if (responseCode != 204) {
                System.out.println("Error: " + respText);
                throw new RuntimeException("Upload may have failed. API Return code was: " + responseCode + ". Expect Code 204");
            } else {
                System.out.println("File Uploaded!!");
            }
        } catch (IOException e) {
            //TODO add proper logging and custom exception
            e.printStackTrace();
            throw new RuntimeException("IOException thrown. File upload failed. Error: " + e.getMessage());
        }
    }

 

 

Response:

 

URL: http s://api.anaplan.com/2/0/workspaces/x/mode ls/x/files/113000000039
contentTypeValue: application/octet-stream
Data Size: 443
Begin File upload...
Upload request done. Took: 00:00:02.722
Response Code: 204
File Uploaded!!

 

 

 

Results from Import Task:

Expecting only 6 rows to be upload. Import task picked up the file i uploaded via the browser (166,598  rows)

 

 

8:12:56.858 [main] INFO com.fw.x.x.anaplan.restapi.client.logging.LogUtils - - - - - - - - - - - - - - - - - - - - - - 
08:12:56.859 [main] INFO com.fw.x.anaplan.restapi.client.Task - Activity Log: 166598 (0/166598) rows successful, 0 ignored 
08:12:56.859 [main] INFO com.fw.x.anaplan.restapi.client.Task - hierarchyName - Activity Log
08:12:56.859 [main] INFO com.fw.x.anaplan.restapi.client.Task - successRowCount - 166598
08:12:56.859 [main] INFO com.fw.x.anaplan.restapi.client.Task - successCreateCount - 0
08:12:56.859 [main] INFO com.fw.x.anaplan.restapi.client.Task - successUpdateCount - 166598
08:12:56.859 [main] INFO com.fw.x.anaplan.restapi.client.Task - warningsRowCount - 0
08:12:56.859 [main] INFO com.fw.x.anaplan.restapi.client.Task - warningsCreateCount - 0
08:12:56.859 [main] INFO com.fw.x.anaplan.restapi.client.Task - warningsUpdateCount - 0
08:12:56.859 [main] INFO com.fw.x.anaplan.restapi.client.Task - failedCount - 0
08:12:56.859 [main] INFO com.fw.x.anaplan.restapi.client.Task - ignoredCount - 0
08:12:56.860 [main] INFO com.fw.x.anaplan.restapi.client.Task - totalRowCount - 166598
08:12:56.860 [main] INFO com.fw.x.anaplan.restapi.client.Task - totalCreateCount - 0
08:12:56.860 [main] INFO com.fw.x.anaplan.restapi.client.Task - totalUpdateCount - 166598
08:12:56.860 [main] INFO com.fw.x.anaplan.restapi.client.Task - invalidCount - 0
08:12:56.860 [main] INFO com.fw.x.anaplan.restapi.client.Task - updatedCount - 166598
08:12:56.860 [main] INFO com.fw.x.anaplan.restapi.client.Task - renamedCount - 166598
08:12:56.860 [main] INFO com.fw.x.anaplan.restapi.client.Task - createdCount - 0
08:12:56.864 [main] INFO com.fw.x.anaplan.restapi.client.Task - Task ID 407A8B7967A94CFEA00E8491CBE98F35 executed by anaplanfrsysdev Complete. Took: 00:00:01.316

 

 

 

14 REPLIES 14
Highlighted
Community Boss

Re: Uploading file Via REST API and Triggering Imports

Hi,

 

the response should be 200.

 

Are you following the doc ?

https://anaplan.docs.apiary.io/#reference/upload-files 

 

Can you show us your request/code to the API ?

Nathan Rudman
Anaplan Model Builder
Contributor

Re: Uploading file Via REST API and Triggering Imports

Yes, I have reviewed https://anaplanbulkapi20.docs.apiary.io/#reference/upload-files

 

The upload REST API is a PUT hence the response should be 204 and not 200. See https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/204 

 

The document clearly states a successfully response will return 204.

 

The API I am calling is 'upload a single file

 

Upload a single chunk

  • Request
curl -X PUT \
https://api.anaplan.com/2/0/workspaces/8a8196b15b7dbae6015b8694411d13fe/models/75A40874E6B64FA3AE0743278996850F/files/113000000008 \
-H 'authorization: AnaplanAuthToken {auth_token}' \
-H "Content-Type:application/octet-stream" --upload-file Tests.txt
  • Headers

          Authorization:AnaplanAuthToken {anaplan_auth_token}
          Content-Type:application/octet-stream
    • Body

        curl https://api.anaplan.com/2/0/workspaces/8a8196b15b7dbae6015b8694411d13fe/models/75A40874E6B64FA3AE0743278996850F/files/113000000008 --upload-file Tests.txt
  • Parameters

    • workspaceID (required, string, 8a8196b15b7dbae6015b8694411d13fe) ... String id of the workspace.
    • modelID (required, string, 75A40874E6B64FA3AE0743278996850F) ... String id of the model.
    • fileId (required, string, 113000000008) ... String fileId of the file.
  • Response 204

My Code.

File to be upload in text format, size is ~9 MB

 

 

 

static void uploadFile(CloseableHttpClient client, String apiURL, String authToken, String fileNameWithPath) {

        // My URL: ---<a href="https://api.anaplan.com/2/0/workspaces/x/models/x/files/113000000042" target="_blank">https://api.anaplan.com/2/0/workspaces/x/models/x/files/113000000042</a>
        // Sample (Docs, Unload as single chunk): ------<a href="https://api.anaplan.com/2/0/workspaces/8a8196b15b7dbae6015b8694411d13fe/models/75A40874E6B64FA3AE0743278996850F/files/113000000008" target="_blank">https://api.anaplan.com/2/0/workspaces/8a8196b15b7dbae6015b8694411d13fe/models/75A40874E6B64FA3AE0743278996850F/files/113000000008</a>

        System.out.println("URL: " + apiURL);

        StopWatch stopWatch = new StopWatch();
        stopWatch.start();

        try {
            HttpPut request = new HttpPut(apiURL);
            request.setHeaders(getHeadersForUpload(authToken, false));

            File source = new File(fileNameWithPath);

            if (source == null || !source.exists() || !source.isFile() || !source.canRead()) {
                throw new RuntimeException("Input file must be a valid file. File must be read-able. Check Source file. ");
            }

            byte[] byte_data = org.apache.commons.io.FileUtils.readFileToByteArray(source);
            System.out.println("Data size: " + byte_data == null ? "[NULL]" : byte_data.length);

            if (byte_data == null || byte_data.length == 0) {
                //TODO determine if this should be allowed
                throw new RuntimeException("Source File seems to be empty. No Data to upload. ");
            }

            // create data stream and set to request
            InputStreamEntity entity = new InputStreamEntity(new ByteArrayInputStream(byte_data), byte_data.length);
            request.setEntity(entity);

            System.out.println("Begin File upload...");

            HttpResponse response = client.execute(request);
            stopWatch.stop();
            System.out.println("Upload request done. Took: " + stopWatch.toString());

            int responseCode = response.getStatusLine().getStatusCode();
            String respText = response.getEntity() != null ? EntityUtils.toString(response.getEntity()) : "[no response text]";
            System.out.println("Response Code: " + responseCode);

            if (responseCode != 204) {
                System.out.println("Error: " + respText);
                throw new RuntimeException("Upload may have failed. API Return code was: " + responseCode + ". Expect Code 204");
            } else {
                System.out.println("File Uploaded!!");
            }
        } catch (IOException e) {
            //TODO add proper logging and custom exception
            e.printStackTrace();
            throw new RuntimeException("IOException thrown. File upload failed. Error: " + e.getMessage());
        }
    }

 

 

 

 

Output:

 

 

 

================================================================================
URL: <a href="<a href="https://api.anaplan.com/2/0/workspaces/x/models/x/files/113000000042" target="_blank">https://api.anaplan.com/2/0/workspaces/x/models/x/files/113000000042</a>" target="_blank"><a href="https://api.anaplan.com/2/0/workspaces/x/models/x/files/113000000042</a" target="_blank">https://api.anaplan.com/2/0/workspaces/x/models/x/files/113000000042</a</a>> 
contentTypeValue: application/octet-stream
Begin File upload...
Upload request done. Took: 00:00:08.020
Response Code: 204
File Uploaded!!

 

 

 

 

Highlighted
Contributor

Re: Uploading file Via REST API and Triggering Imports

I have responded to this now 3 times with the necessary details yet my response is not showing up
Highlighted
Community Boss

Re: Uploading file Via REST API and Triggering Imports

send me a DM if you want
Nathan Rudman
Anaplan Model Builder
Highlighted
Community Boss

Re: Uploading file Via REST API and Triggering Imports

Your step 3 (permission on import/upload file will be sent to "Admin Only") is actually setting the initial upload as a "default file" that is available to any other admin who has not uploaded their own copy of the file, so they can for example edit the import definition with representative source data. I think this default file is being picked up by the API-invoked import because the API file upload has not completed but need more detail on what API endpoints were used for the upload to figure out what is missing.

Highlighted
Contributor

Re: Uploading file Via REST API and Triggering Imports

I have updated my original post to include API details.

 

As noted above, i tried several upload options:

  1. Upload by Chunks 
    1. set the chunk count
    2. upload chunks
    3. mark upload as success
  2. Upload a single chunk
    1. upload a single file as noted above

In both cases, the upload picked up the file i uploaded via the UI and not sent via API even though I used the proper File ID

 

Highlighted
Contributor

Re: Uploading file Via REST API and Triggering Imports

@ben_speight 

Any thoughts on this?

Highlighted
Community Boss

Re: Uploading file Via REST API and Triggering Imports

I think you're uploading the file and running the import as different users - make sure they are the same.

It would be helpful to fix the "Data Size (mb):" message to either convert to megabytes (/1048576) or report it as kilobytes, because that threw me. The output doesn't include that line though - check the compiled class is newer than the source code.

Highlighted
Contributor

Re: Uploading file Via REST API and Triggering Imports

"I think you're uploading the file and running the import as different users - make sure they are the same."

 

This will never be the same as the Upload and Import are initiated by a human via the User interface.

 

The batch-based process is a special integration account (not human)....I cannot add my email and password in a batch system.

 

So are you saying if I initiated an upload/import via the browser with my personal account, set access to "Admin Only" then proceed to perform an upload to the same file with another account via the REST API, Anaplan will not recognize the newly uploaded file? The import process will not pick up the newly API-uploaded-file but the file i uploaded via the browser?