Page History
For troubleshooting configurations for your DataSync Agent, contact Perspectium Support if you have any questions.
Panel | ||||
---|---|---|---|---|
| ||||
|
Panel | ||||
---|---|---|---|---|
| ||||
|
Anchor | ||||
---|---|---|---|---|
|
Monitoring DataSync Agent logs
The DataSync Agent generate log messages in the logs directory that is created upon installation of the agent. To view logs for your agent, open the perspectium.log file within the logs directory.
Here are some examples of log messages that might appear within your perspectium.log file that indicate potential errors with your DataSync Agent:
Error type | Sample log message(s) | Suggested action |
---|---|---|
Connection timeout | 2019-07-11 11:15:54.064 ERROR - main - Replicator - new connection attempt to target: https://example.perspectium.net as user: example password length: 10 failed: com.perspectium.api.MessageBusException: Send Error: Connect to example.perspectium.net:443 [example.perspectium.net] failed: Connection timed out: connect, GET https://example.net/customerstatus HTTP/1.1 | Check your network connection and that your Agent has outbound access through your organization's firewall. Also, check that your <message_connection> URL is correct. |
Invalid credentials | 2019-07-11 11:21:35.854 ERROR - main - Replicator - new connection attempt to target: https://ofc.example.net:4443/ as user: admin12 password length: 10 failed: com.perspectium.api.MessageBusException: Send Error: Unexpected response status: 401 reason: , GET https://ofc.example.net:4443//customerstatus HTTP/1.1 (Please check that your queue exists and that your credentials are correct) | Check that your username, password, and queue name are correct. |
Database primary key error | 2019-07-11 11:27:49.778 ERROR - main - Scheduler - java.lang.IllegalStateException: Unable to locate the database entry using task defined type: randomsql | Check that the database that you're trying to update exists. |
Database connection error | 2019-07-11 11:32:12.745 WARN - main - TaskDatabase - The TCP/IP connection to the host localhost, port 1411 has failed. Error: "Connection refused: connect. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.". | Check your network connection and that your Agent has outbound access through your organization's firewall. Also, check that your database port and name are correct. |
Database login error | 2019-07-11 11:29:43.223 ERROR - main - SQLDataSource - Error: Cannot create PoolableConnectionFactory (Login failed for user 'exampleuser'. ClientConnectionId:98e4b368-e34d-47f8-9019-19a2f6d66715) | Check that your database username and password are correct. |
Invalid decryption key | 2019-07-11 11:37:03.935 ERROR - test_temporal_subscribe - SubscriberTask - Subscriber Error: com.perspectium.replicator.SubscribeException: Failed to decrypt message, make sure the shared secret keys match! | Check that your decryption key is correct (i.e. matches the key as used to encrypt the data in your source instance such as ServiceNow). |
Invalid agent.xml format | 2019-07-11 11:39:25.392 ERROR - main - Replicator - (stderr) [Fatal Error] :27:12: The element type "random" must be terminated by the matching end-tag "</random>". 2019-07-11 11:39:25.392 ERROR - main - Replicator - Error parsing null 2019-07-11 11:41:37.659 ERROR - main - Replicator - message_connection configuration directive is missing! | Validate that your agent.xml file is formatted correctly. Also, check that your agent.xml is configured correctly. |
Timeout Values
It may be necessary to adjust the timeout values that the DataSync Agent is using. You can do this using the attributes of <message_connection> with connection_request_timeout and connection_timeout. For example, you adjust these values to 120000 for 120 seconds.
NOTE: This is only applicable if you are using https for the Agent.
Expand | ||
---|---|---|
| ||
|
CLOB Configuration
By default, the DataSync Agent's maximum column size is 251 characters for character string type columns such as NVARCHAR. When the Agent sees a ServiceNow column defined with more than 251 characters, it will create the column as a CLOB type in the database the Agent is connected to. Columns are created with this length and type to ensure all content is preserved from ServiceNow.
Since ServiceNow uses their own translations to map column types they define for “fields” to the column types of the backend database that powers their platform, these fields don’t always have the exact same length as what’s shown in the ServiceNow UI. When a field is created with less than 251 characters in ServiceNow, that field is considered a single line string which maps to an NVARCHAR type with the same number of characters in the ServiceNow platform's backend database (generally MySQL). If it’s more than 251 characters, that field would be mapped and created as a CLOB type in their backend database.
As a result, when you define a field in ServiceNow with a type of “String” and a length of 400 and because this is mapped to a CLOB type in the backend database, you can actually enter a value greater than 400 characters in the field and it will save properly in ServiceNow both in the UI and in background scripts. When the data is sent out from ServiceNow using the Perspectium DataSync application, this field’s data will appear with more than 400 characters.
If the agent were to create a column in the database with 400 characters, this would result in the data being truncated and loss of content. To prevent this, the Agent creates the field with a larger length and uses a CLOB type that has no length restrictions.
The Agent does provide a way to customize this value to a different one than the default value of 251. To do this, add the following directive to the agent.xml file saved in your Agent's installation directory:
Directive | Value |
---|---|
<database_column_max_size> | A number to be the maximum before converting to a CLOB. This value cannot be greater than the database maximum size for a character string column (see below). |
NOTE: Each Agent-supported database has a maximum value that a text column can be created as. Any column size greater than the value specified below will be created as a CLOB type.
Database | Maximum Column Size |
---|---|
Oracle | 2000 |
SQL Server (MSSQL) | 4000 |
MySQL* | 4000 |
SAP Hana | 5000 |
Amazon Redshift | 65535 |
Amazon Aurora | 65535 |
IBM DB2 | 32672 |
HP Vertica | 65000 |
PostGreSQL | 10485760 |
Snowflake | 4000 |
Sybase | 32767 |
NOTE: MySQL also has table row size limits that should be considered before setting this value too high as you may get "Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs" exceptions if you create too many columns as character string types and not CLOB types.
Expand | |||||||
---|---|---|---|---|---|---|---|
| |||||||
|
Error Directory
Users can specify a log directory for messages that have caused errors. These logs contain the messages' data to help you review and recover records.
To add an error directory, follow these steps:
- Navigate to the directory where you saved your agent.xml file when installing your DataSync Agent.
Place the tag <error_directory></error_directory> within the agent.xml.
For example: <error_directory>../errors</error_directory> will create a directory named errors at the base Agent directory. See more examples below.
- Save the agent.xml and restart the agent.
Expand | ||||
---|---|---|---|---|
| ||||
Task Level - Place the <error_directory> directive within the <task> brackets. Defining it explicitly (or avoiding it) for each task. You can do this if you want to configure different error directories per task / queue.
Agent Level - Place the <error_directory> directive within the <agent> level to have all the tasks inherit this behavior. You can do this if you just want one single error directory for all your tasks / queues.
Example: |
Instance Filter
The DataSync Agent will process all data which comes into its queue. This means that by default multiple instances can write to the same queue and the agent will process them all. If you do not want this behavior you can modify your configuration so the agent will only process data from a specific instance and skip the rest.
To set up instance filter for the DataSync Agent, follow these steps:
- Navigate to the directory where you saved your agent.xml file when installing your DataSync Agent.
Add the <key> directive to the <task> directive of your agent.xml. For example if you wanted to only process data from the ServiceNow instance dev12345 than you would put:
Code Block <key>dev12345</key>
- Save the agent.xml and restart the agent.
NOTE: If you have multiple tasks in your agent.xml you should place this tag in each task. You can have one task have one key and a different task with a different key.
Expand | |||||
---|---|---|---|---|---|
| |||||
|