Page History
To enhance your DataSync integration for Parquet, you can configure the Parquet Bulk Load Meshlet to the directives listed below.
Anchor | ||||
---|---|---|---|---|
|
Panel | ||||||
---|---|---|---|---|---|---|
| ||||||
|
To check out the general meshlet configurations, see General Meshlet Configurations for DataSync.
Directive | Default Value | Description | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
maxFileSize | Required Configuration. This configuration specifies the max size of records for each parquet file. 10000 will be used if input is over 10000 to prevent possible performance and memory issues. A suggested value is 5000.
| ||||||||||||||
customFileName | $table-$randomid | Dynamic naming convention that will be replaced when creating files using the following keywords. File names MUST be unique.
| |||||||||||||
fileDirectory | /files | Directory where the locally created files get made. (In respects to where application is running)
| |||||||||||||
postInterval | 2 | Minutes to check dead periods. Check every x minutes to compare if the in memory collection is the same as the last x minutes. If so, write records to file and push to parquet
| |||||||||||||
dateFormat | yyyy-MM-dd'T'HH:mm:ss.SSSZ | Date format used to create the file name. A valid SimpleDateFormat required.
| |||||||||||||
timeZone | GMT | ID of timezone to be used in $zonedatetime.
| |||||||||||||
file_prefix | Prefix used for file naming.
| ||||||||||||||
file_suffix | Suffix used for file naming.
|
Azure External Storage
To enable sharing Parquet files to Azure, use the following directives:
Directive | Default Value | Description | ||||
---|---|---|---|---|---|---|
connectionString | Connection URL for your Azure. To access the URL, go to Azure Portal > Storage Account > Access Keys > Show Keys > Connection String.
| |||||
destinationContainer | Name of your Azure Blob Storage container, including subdirectories if desired, to specify where the records will be uploaded e.g. container/folder1/folder2. For example, the following will save records into the pspcontainer blob storage container:
If the following is configured:
If an incident record is being processed and uploaded to the Azure Blob Storage container, then the record will be saved in the pspcontainer container and in the /tables/incident directory in that container, creating the directoriy incident automatically.
|
UI Text Box | ||
---|---|---|
| ||
To enable the sharing of Parquet files to Azure, the spring profile will need to include azure:
|