Versions Compared

Old Version 3

changes.mady.by.user user-1bc08

Saved on

compared with

New Version Current

changes.mady.by.user user-b885f

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

(warning) WARNING:

If you will be setting up the DataSync Agent to also connect to the same database, you will want to configure the Agent to use the same column cases

To enhance your DataSync integration for Parquet, you can configure the Parquet Bulk Load Meshlet to the directives listed below:

Warning
iconfalse

.

By default, the Parquet Bulk Load Meshlet uses lowercase for the column names. Also, the QUOTED_IDENTIFIERS_IGNORE_CASE setting in Parquet should be kept with its default value of

 

false.
  • The meshlet uses the TIMEZONE session parameter to connect to Parquet in order to save all timestamps in UTC . By default, glide_date_time field is mapped to the TIMESTAMP_LTZ(9) column type. This allows you to query using a local timezone as needed. 

To check out the general meshlet configurations, see General Meshlet Configurations for DataSync.


DirectiveDefault ValueDescription

maxFileSize


Required Configuration. This configuration specifies the max size for temporary files created as the meshlet pushes records to parquet. of records for each parquet file. 10000 will be used if input is over 10000 to prevent possible performance and memory issues. A suggested value is 5000.

(info) NOTE: Negative values are not supported and will result in 1 record for each parquet file. 

Code Block
languageyml
perspectium:
	filesubscriber: 
		maxFileSize: 150005000


customFileName

$table-$randomid

Dynamic naming convention that will be replaced when creating files using the following keywords. File names MUST be unique.

KeyDescription
$tableName of table file
$zonedatetime

Time of file writing  - Default format: yyyy-MM-dd'T'HH:mm:ss.SSSZ

(info) NOTE: Format can be changed (see dateFormat configuration)

$randomIDRandom ID to ensure unique file naming


Code Block
languageyml
perspectium:
	filesubscriber: 
		customFileName: $table-$randomid


fileDirectory

/files

Directory where the locally created files get made. (In respects to where application is running)

Code Block
languageyml
perspectium:
	filesubscriber: 
		fileDirectory: /files


postInterval

2

Minutes to check dead periods. Check every x minutes to compare if the in memory collection is the same as the last x minutes. If so, write records to file and push to parquet

Code Block
languageyml
perspectium:
	parquet: 
		postInterval: 2


dateFormatyyyy-MM-dd'T'HH:mm:ss.SSSZ

Date format used to create the file name. A valid SimpleDateFormat required.

Code Block
languageyml
perspectium:
	parquet: 
		postInterval: 2


timeZoneGMT

ID of timezone to be used in $zonedatetime.

Code Block
languageyml
perspectium:
	parquet: 
		timeZone: GMT


file_prefix

Prefix used for file naming.

Code Block
languageyml
perspectium:
	parquet: 
		file_prefix: psp


file_suffix

Suffix used for file naming.

Code Block
languageyml
perspectium:
	parquet: 
		file_suffix: bt