To enhance your DataSync integration for Snowflake, you can configure the Snowflake Bulk Load Meshlet to the directives listed below:
Directive | Default Value | Description |
---|---|---|
maxFileSize | 15000 | Total of individual records (per table) for each file. After x records have been received by the meshlet, a file will be created with x records and an immediate push of all records in that file to Snowflake will occur. perspectium: filesubscriber: maxFileSize: 15000 |
customFileName | $table-$randomid | Names file with format of table - random id. File names MUST be unique. perspectium: filesubscriber: customFileName: $table-$randomid |
fileDirectory | /files | Directory where the locally created files get made. (In respects to where application is running) perspectium: filesubscriber: fileDirectory: /files |
postInterval | 2 | Minutes to check dead periods. Check every x minutes to compare if the in memory collection is the same as the last x minutes. If so, write records to file and push to Snowflake perspectium: snowflake: postInterval: 2 |
deleteFiles | true | Indicates whether you want to keep or delete locally created CSV files. Will not have a noticeable performance hit. perspectium: filesubscriber: deleteFiles: true |
fileCleanerInterval | 4 | How often (in minutes) the file cleaner job will run to clean up local files created by the meshlet. This job is similar to the ServiceNow Data Cleaner Scheduled Job. For example, value of 4 will run the job every four minutes. The default value is 1. perspectium: filesubscriber: fileCleanerInterval: 4 |
fileMaxAge | 1 | Any csv file in the filesDirectory older than fileMaxAge in minutes, will be automatically deleted. The default value is 1. perspectium: filesubscriber: fileMaxAge: 1 |
legacyDbViewId | false | Derivative to use the legacy version for database view tables of concatenating GUID values into a sys_id field. If false, meshlet will use the pre-constructed encoded sys_id created by ServiceNow. perspectium: filesubscriber: legacyDbViewId: false |
useReceipts | false | Enable data guarantee. perspectium: message: receipts: enable: false |
receiptOutboundQueue | Queue receipts are returned to, typically default subscribed to on a ServiceNow instance. perspectium: message: receiptOutboundQueue |