To enhance your DataSync integration for Snowflake, you can configure the Snowflake Bulk Load Meshlet to the directives listed below:
WARNING:
- If you will be setting up the DataSync Agent to also connect to the same database, you will want to configure the Agent to use the same column cases. By default, the Snowflake Bulk Load Meshlet uses lowercase for the column names. Also, the QUOTED_IDENTIFIERS_IGNORE_CASE setting in Snowflake should be kept with its default value of false.
- The meshlet uses the TIMEZONE session parameter to connect to Snowflake in order to save all timestamps in UTC . By default, glide_date_time field is mapped to the TIMESTAMP_LTZ(9) column type. This allows you to query using a local timezone as needed.
| Directive | Default Value | Description |
|---|---|---|
maxFileSize | 15000 | Total of individual records (per table) for each file. After x records have been received by the meshlet, a file will be created with x records and an immediate push of all records in that file to Snowflake will occur. perspectium: filesubscriber: maxFileSize: 15000 |
customFileName | $table-$randomid | Names file with format of table - random id. File names MUST be unique. perspectium: filesubscriber: customFileName: $table-$randomid |
fileDirectory | /files | Directory where the locally created files get made. (In respects to where application is running) perspectium: filesubscriber: fileDirectory: /files |
postInterval | 2 | Minutes to check dead periods. Check every x minutes to compare if the in memory collection is the same as the last x minutes. If so, write records to file and push to Snowflake perspectium: snowflake: postInterval: 2 |
deleteFiles | true | Indicates whether you want to keep or delete locally created CSV files. Will not have a noticeable performance hit. perspectium: filesubscriber: deleteFiles: true |
fileCleanerInterval | 4 | How many minutes for a job to clean up local files. (Like ServiceNow DataCleaner Scheduled Job) perspectium: filesubscriber: fileCleanerInterval: 4 |
fileMaxAge | 1 | Any csv file in the filesDirectory older than fileMaxAge in minutes, will be automatically deleted. perspectium: filesubscriber: fileMaxAge: 1 |
| legacyDbViewId | false | Derivative to use the legacy version for database view tables of concatenating GUID values into a sys_id field. If false, meshlet will use the pre-constructed encoded sys_id created by ServiceNow. perspectium: filesubscriber: legacyDbViewId: false |
| useReceipts | false | Enable data guarantee. perspectium: message: receipts: enable: false |
| receiptOutboundQueue | Queue receipts are returned to, typically default subscribed to on a ServiceNow instance. perspectium: message: receiptOutboundQueue |