Page History
For optional advance configurations for your DataSync Agent, contact Perspectium Support if you have any questions.
Anchor | ||||
---|---|---|---|---|
|
Panel | ||||
---|---|---|---|---|
| ||||
|
Panel | ||||
---|---|---|---|---|
| ||||
|
Agent Column Case Configuration
The DataSync Agent supports forcing of the column name to a configured case. This configuration must be done prior to the Agent creating the table. Two directives are available for inclusion within the agent.xml configuration file. The chosen directive must be placed within the <task> tag.
You would use either of these directives in your agent.xml:
Expand | |||||
---|---|---|---|---|---|
| |||||
|
The default (none of these directives) simply uses the original case of the column. If you happen to specify both, uppercase will be used.
NOTE: This directive affects column names only. It does not affect table names.
Change data type
To ensure multibyte or foreign characters are saved properly for string column types, the Agent creates column types that can store unicode string data such as NVARCHAR (MySQL, MS SQL Server, etc). You can use other data types other than N type fields by editing the databases.xml.
Before you begin, request the databases.xml file for your DataSync Agent by contacting Perspectium Support.
Divbox | |||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||
|
DataSync Agent field type mappings
To ensure data is formatted correctly, the DataSync agent will map ServiceNow field types to Database field types. Consult the table below to see how ServiceNow field types are mapped for supported databases.
Expand | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Enable upsert to a database
For .bulk messages, the DataSync Agent will query for a record to see if it exists and if so, update that record. Otherwise if the record doesn't exist, an insert will occur. However for databases that support it, the Agent can perform an upsert action where updating or inserting is done in one action to help improve performance on large tables that have hundreds of millions or billions of records. To enable this, you will need to use the <enable_upsert/> directive.
Note | ||
---|---|---|
| ||
Enabling upsert is only available for SAP Hana. |
Divbox | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||
|
Excluding column DataSync Agent configuration
The DataSync Agent supports excluding columns from replication. Configuring columns to be excluded should be done prior to the Agent creating the table, which contains the excluded column for clarity. However, it can be done after with no ill affect. That column will no longer be populated.
You can configure multiple columns to be excluded. To set this up you will have one <exclude_columns>…</exclude_columns> set of tags. Within the tag, you will define each column to be excluded by individual <exclude_column>…</exclude_column> tags. This will be placed within the <subscribe> element located in the agent.xml configuration file.
NOTE: The preferred approach for excluding columns from being replicated is to leverage capabilities of the producer, such as Perspectium Views, which provides the ability to define which columns should be replicated.
In the example below, the necessary configuration is added to exclude the columns:
fx_price
sys_tags
sys_translated_text
Currently, when a column is excluded, it is applied to every table which may contain the column.
Expand | |||||
---|---|---|---|---|---|
| |||||
|
Set up a Highly Available Agent
Perspectium DataSync Agents support an “exclusive consumer” mode that provides a Highly Available Agent solution while maintaining the order of messages in a queue. This active-passive Highly Available approach will exclusively allow only one DataSync Agent to actively read messages from a queue. Other DataSync Agents that passively try to connect to the queue will only be given access to the queue if another DataSync Agent is no longer connected.
DataSync Agents use the queue in the message connection and locks that queue for exclusive use. If multiple tasks are made using the same message connection, only one subscriber task will be created and used. When running another DataSync Agent, a warning will be logged in the perspectium.log file indicating that the queue is in exclusive use and that consumer wait time will be increased (default is to increase to 30 seconds). Once the first running DataSync Agent is stopped or fails, the backup DataSync Agent will start consuming messages when the queue becomes available and the consumer wait time will be reset.
NOTE: The Highly Available Agent is only available when your DataSync Agent is set to connect to the Perspectium Mesh via the AMQP/AMQPS protocol (HTTP/HTTPS is not supported).
Divbox | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||
|
Set up temporal data
To compile timestamped "snapshots" of your ServiceNow data, you can configure your DataSync Agent with the <temporal> directive. <temporal> will allow you to indicate times when your record's data is/was valid from and when the data is/was valid to.
NOTE: To set up a temporal data, the table you are syncing data to must not contain any records (i.e., Temporal data cannot be captured for tables that are already being synced to a database with a DataSync Agent).
Warning |
---|
WARNING! If you have already configured an integration with a DataSync Agent and have been saving records in a database, but you would now like to enable or disable temporal replication, update the value within the agent.xml's <database> directive to a new database. Otherwise records will not be processed properly. |
The following are a list of database the temporal agent supports:
- Oracle
- MySQL
- MSSQL
- SAP Hana 2.0
- PostGresSQL
Divbox | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Share Table Schemas to a database
By default, table schemas are created by passing your ServiceNow instance username and password into the <instance_connection> tag of the agent.xml file created upon installation of the DataSync Agent. Alternatively, you can send ServiceNow table schemas directly to a database that your DataSync Agent replicates to.
NOTE: Table schemas can be shared through bulk share only.
Divbox | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||
|
Skip altering of database tables
By default, your Perspectium DataSync Agent will make changes to your database table(s) so that the table(s) in your application match with your database. For example, you DataSync Agent will add database columns and increase the size of database columns to ensure the complete syncing of data. You can, however, turn off this functionality by adding the <skip_alter/> directive within your agent.xml configuration file.
Divbox | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||
|
Specify Working Directories for Agent Files
By default the conf, logs and perspectium_data directories will be saved in the directory where the DataSync Agent is installed. However, you can specify a different “working directory” for these files/directories.
The directories that can be moved to a different working directory contain the following files:
- conf - The configuration directory holds the agent.xml and log4j2.xml configuration files
- logs - The directory that holds the Agent's log files
- perspectium_data - The directory that holds temporary data while the Agent runs. By default, the perspectium_data directory is created in the bin directory. The perspectium_data directory is generated automatically when the Agent starts.
NOTE: You will still need to keep a conf directory in the directory where the Agent is installed. However, in the new working directory that you specify you must also create a conf directory containing your agent.xml and log4j2.xml files. The perspectium_data directory will be generated in the first level of the working directory alongside these two directories when the Agent starts.
Divbox | |||||
---|---|---|---|---|---|
| |||||
|
Divbox | |||||
---|---|---|---|---|---|
| |||||
|
Truncate string values
In the DataSync Agent, you can truncate string values if you have exceeded a column's max length (UTF-8). If you are using UTF-8 characters and prefer using VARCHAR or a string data type where the specified length is strictly the number of bytes, you will be able to truncate the string value as well.
NOTE: UTF-8 characters may have multiple bytes per character.
If you use multiple UTF-8 characters and the number of bytes exceeds what is allowed in a column, an error will occur. You can prevent this by either adding <byte_padding/> or if you want to keep the original length and truncate the string values, add <truncate_utf8/> directive within your agent.xml configuration file.
If you use a different character encoding, you can instead use <database_truncate_encoding> and set your specified encoding.
UI Text Box | ||
---|---|---|
| ||
NOTE: Currently, the <database_truncate_encoding> directive only allow these encoding values:
|
Request the databases.xml file for your DataSync Agent by contacting Perspectium Support.
Divbox | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||
|
Show If | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||
Enable multiple queues or database per Agent You can only configure the Agent to point to one queue or database. If you want to override that restriction, you will need to add the following under the <agent> directive:
|
Key Set Processing for DataSync Agent
Key Set Processing is a capability of the DataSync Agent which can be enabled when replicating messages from a ServiceNow instance to a database. The Key Set capability optimizes throughput while further ensuring message ordering.
NOTE: It's strongly recommended to enable this capability when replicating from a ServiceNow instance.
This is enabled on a task by task basis by adding the following configuration direction within the <task> definition.
Code Block | ||
---|---|---|
| ||
<task> <enable_keyset_processing/> ... ... </task> |
The following directives can be added to the agent.xml when enabling the Key Set Processing:
Directive | Description | Example | |||||
---|---|---|---|---|---|---|---|
<keyset_size> | Indicates the number of key set processor that will maintain the connection to the database.
|
| |||||
<queue_consumer_polling_interval> | Indicates the time interval the queue will be consumed. By default, the key set is scheduled to read from the queue every two seconds. |
| |||||
<keyset_read_wait_time> | Indicates the time each key set processor will wait when reading from its queue. By default, the wait time is 200 milliseconds. |
|
Expand | |||||
---|---|---|---|---|---|
| |||||
|
Indexing Database tables
Database indexing is used to speed up data retrieval when querying for records from database tables. Enabling the Share Index option allows you to share the indexes of a table in ServiceNow through a bulk share so the same indexes can be created in the database where you've shared data to.
With this option enabled, the bulk share shares out an index message that contains the columns that are indexed in the table as its configured in ServiceNow. The DataSync Agent will then pick up this message and create indexes on the columns of the database table to match.
NOTE: Indexing database tables is compatible with Oracle, SQL Server, MySQL, and Postgres databases, and is only used for DataSync. Snowflake does not support indexing, therefore, this feature is not compatible for Snowflake.
UI Text Box | ||
---|---|---|
| ||
Some databases, such as SQL Server and Oracle, do not support the creation of indexes on columns that are of large object (LOB) data types. To learn more about CLOB columns in DataSync Agent configurations, see CLOB Configuration for DataSync Agent. |
Divbox | |||||
---|---|---|---|---|---|
| |||||
|
Match Field Lengths
By default, whenever a new text field is added to an existing table, the DataSync Agent will use 40 for lengths that are less than 40. For text fields greater than 40 in length, it will use the field length from the schema. To enable matching the length of the text fields, add the <match_field_lengths/> directive within your agent.xml configuration file.
NOTE: This directive will work for new fields only.
Divbox | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||
|
Batch Processing
With batch processing, records are inserted or updated in batches versus being executed one by one. When there are no more records received in the given flush interval, <batch_flush_interval>, or when the number of records reaches the maximum batch size, <max_batch_size> (whichever happens first), then the DataSync Agent will insert or update all the records from the current batch with the use of <batch_insert/> and <batch_update/>.
Using batch processing can improve the DataSync Agent's overall performance.
To use batch processing with DataSync Agent, configure your agent.xml with the directives below:
Directive | Example | Description |
---|---|---|
<batch_insert/> | Self closing tag that configures your Agent to batch SQL INSERT statements. | |
<batch_update/> | Self closing tag that configures your Agent to batch SQL UPDATE statements. | |
<batch_flush_interval> | <batch_flush_interval>20</batch_flush_interval> | Number of seconds that will trigger a batch SQL statement execution. Acceptable values are integers from 1 to 30 (seconds). |
<max_batch_size> | <max_batch_size>200</max_batch_size> | Number of SQL statements that will trigger a batch SQL statement execution. A larger suggested value is 200. |
Expand | |||||
---|---|---|---|---|---|
| |||||
|
Truncate Before Share
In the DataSync Agent, you can truncate and delete all the records for a selected table in your database before bulk sharing. This is best for when you want to avoid using business rules to capture your deletes. Instead, you can use bulk share with Truncate Before Share enabled to send a message to the agent that will indicate a delete in all the records in the specified table.
This feature is NOT compatible with custom schemas.
For example,
- For tables cmdb_rel_ci, they get rebuilt nightly through discovery, you can use this feature to keep track
- For tables like that where we cant capture deletes since they’re done silently, it allows you to be up to date
- Runs discovery -> clears out table and recreates relations in the table -> so now the database needs to get in sync
UI Text Box | ||
---|---|---|
| ||
This feature requires configuration in both the DataSync Agent and the DataSync application in ServiceNow. |
NOTE: This is an advanced feature that will delete data from your database and should be used with caution. Incorrect use of this feature may result in data being lost that cannot be recovered. Contact Perspectium Support with any questions before enabling this feature.
To truncate records for your DataSync integration, configure your agent.xml with the directive below:
This will be added within the <task> directive(s).
Directive | Example | Description | |
---|---|---|---|
<truncate_table> | <truncate_table>true</truncate_table> | Enables truncate and deleting all records for a selected table in your database before bulk sharing. | |
Attributes | Description | ||
delete_only | To issue a DELETE command instead of TRUNCATE. <truncate_table delete_only="true">true</truncate_table> |
Divbox | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||
|
Dynamic Columns Configuration
Dynamic columns provides the ability to configure one or more columns that will be added for each destination table. For example use Dynamic columns when you want to track the exact time in which a table entry has been made. You can create the desired column using the Dynamic columns feature and then manage the content of the column via the Plugin Framework.
Code Block |
---|
<dynamic_columns> <dynamic_column column_name="my_column" column_type="93" column_size="255"></dynamic_column> </dynamic_columns> |
The column_name must contain a valid column name per the requirements of the target database type. The column_type must be a valid Java SQL type. The column_size field is required but will only be used when your column type is character oriented.
Expand | ||
---|---|---|
| ||
|
Installing Third-party Libraries
After installing your agent, you may need to download and install third party libraries that you have licenses to in order to add extra functionality to your agent.
For example, if you are running the agent on Windows and want to support DPAPI to encrypt values, you will need to provide the DPAPI library to the agent as follows:
Start by opening up the directory where your agent is installed. In this directory you will see the extlib directory.
- Next, open the extlib directory. Drag and drop the desired library, or libraries, into the directory. Make sure that you obtain the version of the library that's right for the target architecture of the machine that's running the agent, e.g. 64 bit versus 32 bit.
Configuring Multiple Agents
Although you can run multiple tasks within a single DataSync Agent, you can also run multiple DataSync Agents as services on the same machine. A common strategy is to have one DataSync Agent for production work and another one for testing/development work.
Running multiple DataSync Agents requires modifying entries within each Agent's wrapper.conf file (under ../conf).
The wrapper.ntservice.name entry is required to be unique for each DataSync Agent on the same machine.
The other entries (wrapper.ntservice.displayname and wrapper.ntservice.description) listed below are not required to be unique but can be changed to help make it easier to distinguish the different DataSync Agents when looking at your operating system's services console.
NOTE: When installing the DataSync Agent through the Installer, the Agent Name you enter will be used in the wrapper.ntservice.name entry. You can run the Installer multiple times to install multiple Agents and must specify different installation paths and different Agent Name values each time you run the installer.
For example, the wrapper.conf for one DataSync Agent will look as follows:
Code Block | ||
---|---|---|
| ||
# Name of the service wrapper.ntservice.name=agent1 # Display name of the service wrapper.ntservice.displayname=Perspectium Replicator Agent1 # Description of the service wrapper.ntservice.description=Perspectium Replicator Agent1 |
A second Agent would have entries such as the following:
Code Block | ||
---|---|---|
| ||
# Name of the service wrapper.ntservice.name=agent2 # Display name of the service wrapper.ntservice.displayname=Perspectium Replicator Agent2 # Description of the service wrapper.ntservice.description=Perspectium Replicator Agent2 |
Configuring the DataSync Agent to Leverage a Proxy
The DataSync Agent supports accessing the Perspectium Integration Mesh and replication sources such as ServiceNow through a Proxy. Several organizations use such a proxy to provide a more secure or controlled method in which to access servers within the Internet.
Access to the Proxy can be configured to use either the HTTP or HTTPS protocol. The destination server, such as a ServiceNow instance, can also be configured to leverage either protocol. Furthermore, the DataSync Agent supports the ability to authenticate itself to the proxy using either the Basic Authentication or NTLM protocols. Additionally, you can configure authentication support for the destination server as well.
Divbox | |||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||||||||||||||||||||||||||
|
Set a custom primary key
By default, the Perspectium DataSync Agent uses a GUID type field as the primary key for a table. If you require a separate auto-incrementing primary key as the primary key, you can use the directive <custom_primary_key/> and set the name for this column inside the directive.
For SQLServer, this will also create a non-clustered index for sys_id. To make sure this works, you can look up index ix_psp_old_primary and see if it is set to nonclustered.
NOTE:
- If a name is not specified, it will default to psp_id
- If you also included the <primary_key> tag, it will not create the custom primary key (and index if using SQLServer)
- This applies when creating new tables, existing tables are not affected
Divbox | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||
|
Heartbeat
The Heartbeat configuration also requires information on how to contact the Perspectium Message Bus. The directives used are exactly those used by the core configuration file config.xml but it’s possible that a completely different Message Bus or user credentials will be used. It’s important to note that the Heartbeat configuration is not intended to be modified by the customer.
Directive | Example | Description | Required |
---|---|---|---|
heartbeat_queue | <heartbeat_queue>psp.in.heartbeat</heartbeat_queue> | heartbeat message bus queue | Yes |
update_interval | <update_interval>60</update_interval> | heartbeat interval | Yes |
amqp_uri | <amqp_uri>amqp://localhost</amqp_uri> | Primary Message Bus Location | No |
amqp_user | <amqp_user>georvi</amqp_user> | User for logging into primary bus | No |
amqp_password | <amqp_password>greetingsFromGeorvi</amqp_password> | User's password for message bus | No |
Code Block | ||
---|---|---|
| ||
<heartbeat_configuration> <amqp_uri>amqp://localhost</amqp_uri> <amqp_user>admin</amqp_user> <amqp_password>adminadmin</amqp_password> <heartbeat_queue>psp.in.heartbeat</heartbeat_queue> <update_interval>60</update_interval> </heartbeat_configuration> |
Encrypting Fields
Encrypting sensitive field values is supported in the config.xml and agent.xml files. The agent will utilize encrypted fields based on the encrypted: and encrypt: prefixes.
When the Agent (or the Validate Configuration) starts, any configuration values prefixed with encrypt: will be encrypted and the configuration file will be rewritten with the prefixes changed to encrypted:.
You can encrypt passwords attributes and elements, as well as your decryption keys if desired.
The following is an example of adding encrypt: in the agent.xml:
Expand | |||||
---|---|---|---|---|---|
| |||||
|
The following is an example of the agent.xml after startup:
Expand | |||||
---|---|---|---|---|---|
| |||||
|
Encrypting Fields using DP API
To leverage the Data Protection (DP) API for encrypting fields within your agent.xml configuration file, the Agent will utilize encrypted fields based on dpapi: prefixes. This approach ensures no 3rd party software by itself can decrypt fields that you have encrypted.
Expand | |||||
---|---|---|---|---|---|
| |||||
|
The following is an example of the agent.xml after startup:
Expand | |||||
---|---|---|---|---|---|
| |||||
|