To enhance your DataSync integration for Snowflake, you can configure the Snowflake meshlet to the directives listed below:
| Directive | Default Value | Description |
|---|---|---|
maxFileSize | 15000 | Total of individual records (per table) for each file. After x records have been received by the meshlet, a file will be created with x records and an immediate push of all records in that file to Snowflake will occur. |
buildHeader | false | Each CSV file will include headers of column names in the first row. |
customFileName | $table-$randomid | Names file with format of table - random id. File names MUST be unique. |
fileDirectory | /files | Directory where the locally created files get made. (In respects to where application is running) |
postInterval | 2 | Minutes to check dead periods. Check every x minutes to compare if the in memory collection is the same as the last x minutes. If so, write records to file and push to Snowflake |
deleteFiles | true | Boolean to indicate whether you want to keep or delete locally created CSV files. Will not have a noticeable performance hit. |
fileCleanerInterval | 4 | How many hours for a job to clean up local files. (Like ServiceNow DataCleaner Scheduled Job) |
fileMaxAge | 1 | Any csv file in the filesDirectory older than fileMaxAge in hours, will be automatically deleted. |
