Data Hub Imports without manual uploads, how?

DeveloperCYT
DeveloperCYT
in Modeling

In the Data Hub training it says 'DO NOT' use manual uploads to the data hub. Use ETL Tools. We receive data in flat files (csv) for many of our data sources, and have no 'link' to their systems. So how do I utilise my data hub it if am not supposed to manually upload these files?

Tagged:

Answers

  • elena56brown
    elena56brown
    edited January 22

    To utilize your data hub without manually uploading files, you'll need to use ETL (Extract, Transform, Load) tools. These tools automate the process of extracting data from your flat files (CSV), transforming it into the required format, and loading it into the data hub. Here are some steps to get started:

    Choose an ETL Tool: Select an ETL tool that supports your data sources and data hub. Some popular options include Apache NiFi, Talend, and Informatica subway bogo footlong

    Extract Data: Configure the ETL tool to extract data from your CSV files.

    Transform Data: Apply any necessary transformations to the data, such as cleaning, filtering, or aggregating.

    Load Data: Load the transformed data into your data hub.

    By automating this process, you can ensure consistent and efficient data integration without manual uploads.

  • DeveloperCYT
    DeveloperCYT

    Hi, thanks, do these tools all have a cost associated with them?

  • alexpavel
    alexpavel

    Indeed, this is about automating the uploads in Anaplan.

    In your case, where you have the CSV created from the source system, a costless solution (from the ETL tool point of view) is to use Anaplan Connect which is an utility that upload and import available file.

    You can find more info here:

    https://help.anaplan.com/anaplan-connect-e3a9f00c-3924-4cfb-aed0-1ec14233821b

    In the same Anapedia area, you can alsocheck other ETL tools used in data integration.

    Hope it helps

    Alex

  • DeveloperCYT
    DeveloperCYT
    edited January 21

    Hi Alex,

    I already have written and used approximately 100 Anaplan Connect scripts, and I would class that as 'manual' uploading, but if that's not classed as manual, then I will continue to use Connect to import the data.
    The trouble we have is we are uploading customer orders maybe 20 times a day, as we are FMCG we need this live order processing.
    We receive CSV files of the orders, and then they need uploading.
    Users cannot run Anaplan Connect scripts.
    So currently they upload directly into the main Model, using an Import action.
    I wanted them to upload to the HUB, process the orders and then send clean data to the model.
    This is not possible unless we allow manual uploads to the HUB.


    thanks

Categories