Page History
For optional advance configurations for your DataSync Agent, contact Perspectium Support if you have any questions.
Panel | ||||
---|---|---|---|---|
| ||||
|
Panel | ||||
---|---|---|---|---|
| ||||
|
Anchor | ||||
---|---|---|---|---|
|
Agent Column Case Configuration
The DataSync Agent supports forcing of the column name to a configured case. This configuration must be done prior to the Agent creating the table. Two directives are available for inclusion within the agent.xml configuration file. The chosen directive must be placed within the <task> tag.
You would use either of these directives in your agent.xml:
Expand | |||||
---|---|---|---|---|---|
| |||||
|
The default (none of these directives) simply uses the original case of the column. If you happen to specify both, uppercase will be used.
NOTE: This directive affects column names only. It does not affect table names.
Change data type
To ensure multibyte or foreign characters are saved properly for string column types, the Agent creates column types that can store unicode string data such as NVARCHAR (MySQL, MS SQL Server, etc). You can use other data types other than N type fields by editing the databases.xml.
Before you begin, request the databases.xml file for your DataSync Agent by contacting Perspectium Support.
Expand | |||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||||||||
|
DataSync Agent field type mappings
To ensure data is formatted correctly, the DataSync agent will map ServiceNow field types to Database field types. Consult the table below to see how ServiceNow field types are mapped for supported databases.
Expand | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Enable upsert to a database
For .bulk messages, the DataSync Agent will query for a record to see if it exists and if so, update that record. Otherwise if the record doesn't exist, an insert will occur. However for databases that support it, the Agent can perform an upsert action where updating or inserting is done in one action to help improve performance on large tables that have hundreds of millions or billions of records. To enable this, you will need to use the <enable_upsert/> directive.
Note | ||
---|---|---|
| ||
Enabling upsert is only available for SAP Hana. |
Expand | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||
|
Excluding column DataSync Agent configuration
The DataSync Agent supports excluding columns from replication. Configuring columns to be excluded should be done prior to the Agent creating the table, which contains the excluded column for clarity. However, it can be done after with no ill affect. That column will no longer be populated.
You can configure multiple columns to be excluded. To set this up you will have one <exclude_columns>…</exclude_columns> set of tags. Within the tag, you will define each column to be excluded by individual <exclude_column>…</exclude_column> tags. This will be placed within the <subscribe> element located in the agent.xml configuration file.
NOTE: The preferred approach for excluding columns from being replicated is to leverage capabilities of the producer, such as Perspectium Views, which provides the ability to define which columns should be replicated.
In the example below, the necessary configuration is added to exclude the columns:
fx_price
sys_tags
sys_translated_text
Currently, when a column is excluded, it is applied to every table which may contain the column.
Expand | |||||
---|---|---|---|---|---|
| |||||
|
Monitoring DataSync Agent logs
The DataSync Agent generate log messages in the logs directory that is created upon installation of the agent. To view logs for your agent, open the perspectium.log file within the logs directory.
Here are some examples of log messages that might appear within your perspectium.log file that indicate potential errors with your DataSync Agent:
Error type | Sample log message(s) | Suggested action |
---|---|---|
Connection timeout | 2019-07-11 11:15:54.064 ERROR - main - Replicator - new connection attempt to target: https://example.perspectium.net as user: example password length: 10 failed: com.perspectium.api.MessageBusException: Send Error: Connect to example.perspectium.net:443 [example.perspectium.net] failed: Connection timed out: connect, GET https://example.net/customerstatus HTTP/1.1 | Check your network connection and that your Agent has outbound access through your organization's firewall. Also, check that your <message_connection> URL is correct. |
Invalid credentials | 2019-07-11 11:21:35.854 ERROR - main - Replicator - new connection attempt to target: https://ofc.example.net:4443/ as user: admin12 password length: 10 failed: com.perspectium.api.MessageBusException: Send Error: Unexpected response status: 401 reason: , GET https://ofc.example.net:4443//customerstatus HTTP/1.1 (Please check that your queue exists and that your credentials are correct) | Check that your username, password, and queue name are correct. |
Database primary key error | 2019-07-11 11:27:49.778 ERROR - main - Scheduler - java.lang.IllegalStateException: Unable to locate the database entry using task defined type: randomsql | Check that the database that you're trying to update exists. |
Database connection error | 2019-07-11 11:32:12.745 WARN - main - TaskDatabase - The TCP/IP connection to the host localhost, port 1411 has failed. Error: "Connection refused: connect. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.". | Check your network connection and that your Agent has outbound access through your organization's firewall. Also, check that your database port and name are correct. |
Database login error | 2019-07-11 11:29:43.223 ERROR - main - SQLDataSource - Error: Cannot create PoolableConnectionFactory (Login failed for user 'exampleuser'. ClientConnectionId:98e4b368-e34d-47f8-9019-19a2f6d66715) | Check that your database username and password are correct. |
Invalid decryption key | 2019-07-11 11:37:03.935 ERROR - test_temporal_subscribe - SubscriberTask - Subscriber Error: com.perspectium.replicator.SubscribeException: Failed to decrypt message, make sure the shared secret keys match! | Check that your decryption key is correct (i.e. matches the key as used to encrypt the data in your source instance such as ServiceNow). |
Invalid agent.xml format | 2019-07-11 11:39:25.392 ERROR - main - Replicator - (stderr) [Fatal Error] :27:12: The element type "random" must be terminated by the matching end-tag "</random>". 2019-07-11 11:39:25.392 ERROR - main - Replicator - Error parsing null 2019-07-11 11:41:37.659 ERROR - main - Replicator - message_connection configuration directive is missing! | Validate that your agent.xml file is formatted correctly. Also, check that your agent.xml is configured correctly. |
Set up a Highly Available Agent
Perspectium DataSync Agents support an “exclusive consumer” mode that provides a Highly Available Agent solution while maintaining the order of messages in a queue. This active-passive Highly Available approach will exclusively allow only one DataSync Agent to actively read messages from a queue. Other DataSync Agents that passively try to connect to the queue will only be given access to the queue if another DataSync Agent is no longer connected.
DataSync Agents use the queue in the message connection and locks that queue for exclusive use. If multiple tasks are made using the same message connection, only one subscriber task will be created and used. When running another DataSync Agent, a warning will be logged in the perspectium.log file indicating that the queue is in exclusive use and that consumer wait time will be increased (default is to increase to 30 seconds). Once the first running DataSync Agent is stopped or fails, the backup DataSync Agent will start consuming messages when the queue becomes available and the consumer wait time will be reset.
NOTE: The Highly Available Agent is only available when your DataSync Agent is set to connect to the Perspectium Mesh via the AMQP/AMQPS protocol (HTTP/HTTPS is not supported).
Expand | ||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||
|