Page History
Anchor | ||||
---|---|---|---|---|
|
If you have issues installing or configuring the Agent, see the Installation/Configuration troubleshooting page for help troubleshooting. Below are FAQs and some common issues you may come across while running the DataSync Agent and subscribing to records. Contact support@perspectium.com if your issue is not listed below or you have any other questions.
Panel | ||||
---|---|---|---|---|
| ||||
|
FAQs
Core Application
UI Expand | ||
---|---|---|
|
Core Application
title | Does DataSync work with ServiceNow Database Encryption? |
---|
style | background:white; |
---|
ServiceNow provides an option for Database Encryption where data is encrypted at rest in the database. Since Database Encryption happens at the database layer and the Perspectium application runs at the application layer, by the time we call to get data from ServiceNow, the data will be accessible to our application to be shared out. ServiceNow's documentation mentions with Database Encryption that you can add another level of encryption by also encrypting at the application layer which is what our application supports as well.
DataSync Agent
UI Expand | |||||||
---|---|---|---|---|---|---|---|
| |||||||
| |||||||
UI Expand | |||||||
| |||||||
Divbox | |||||||
| |||||||
|
DataSync Agent
UI Expand | |||||||
---|---|---|---|---|---|---|---|
| |||||||
|
UI Expand | |||||
---|---|---|---|---|---|
| |||||
|
UI Expand | |||||
---|---|---|---|---|---|
| |||||
|
ui-ui-expand | |||||
---|---|---|---|---|---|
| |||||
| |||||
UI Expand | |||||
|
Divbox | ||
---|---|---|
| ||
The HDFSFileSubcriber handler requires the use of the Hadoop external library. You will need the jar files from each of these Hadoop librarires:
The Maven repository for these files can be found here. Once you have these files they need to be placed in the extlib folder in the File Replicator agent's directory. |
title | How can I improve my DataSync Agent's overall performance? |
---|
style | background:white; |
---|
- If you find that your DataSync Agent is running slowly due to a large number of SQL statements being processed, you may want to configure the Agent to perform SQL statements in batches.
NOTE: This feature should only be used when you send messages of the same table into one queue. If your queue has messages from different tables (including sys_audit, sys_journal_field, sys_attachment and sys_attachment_doc records), do not enable this feature as it will cause errors in saving records. To properly use this feature, separate your dynamic and bulk shares to save each type of table record to a different queue.
- To do this, open the agent.xml file that was created upon installation of your Agent in a text editing application. Within the <task> directive of the agent.xml file, nest the following directives:
Number of SQL statements that will trigger a batch SQL statement execution. A larger suggested value is 200.
NOTE: By default, this directive's value will be set to 10.
- In your agent.xml, change the <polling_interval> to a lower number
- In your agent.xml, increase the number of <tasks>
NOTE: Increasing the number of task will require more memory and connections to the database. We do not recommend setting tasks to more than 10.
- In your agent.xml, change <message_connection> attributes so use_basic_consume=false
- In ServiceNow, change your bulk share configuration to Insert Only=true
title | Handling of long Table and Column names when replicating to the Oracle database |
---|
Divbox | ||
---|---|---|
| ||
Since Oracle only permits a string of 30 characters by default, a table or a column name that has over 30 characters will be truncated upon replication. The first 15 characters of the name will be displayed, followed by a “_” and then the last 14 of the table name. The “_” denotes the truncated values between the first 15 and the last 14 characters. For example, when replicating the table “Workflow Estimated Runtime Configuration” (41 characters total) to Oracle database, the replicated name will be “u_workflow_esti__configuration”.
|
Security
|
UI Expand | |||||
---|---|---|---|---|---|
| |||||
|
UI Expand | |||||
---|---|---|---|---|---|
| |||||
|
UI Expand | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||
|
UI Expand | |||||
---|---|---|---|---|---|
| |||||
|
UI Expand | |||||||
---|---|---|---|---|---|---|---|
| |||||||
|
Security
UI Expand | |||||
---|---|---|---|---|---|
| |||||
|
UI Expand | |||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||
|
UI Expand | |||||
---|---|---|---|---|---|
| |||||
|
UI Expand | |||||
---|---|---|---|---|---|
| |||||
|
UI Expand | |||||
---|---|---|---|---|---|
| |||||
|
UI Expand | |||||
---|---|---|---|---|---|
| |||||
|
UI Expand | |||||
---|---|---|---|---|---|
| |||||
|
UI Expand | |||||
---|---|---|---|---|---|
| |||||
|
UI Expand | |||||
---|---|---|---|---|---|
| |||||
|
Troubleshooting
General Issues
UI Expand | ||
---|---|---|
| ||
Open the wrapper.conf file located in the Agent's conf folder and change the following configuration: #wrapper.java.maxmemory=64 Removing the “#” and putting a numeric value higher than 64. This numeric value is a size in MB for the Java memory heap space the agent can use. Generally, you would base this value on the memory available on the server where the Agent is running. For example, if the server has 1GB of memory, you can set it to be 512MB here: wrapper.java.maxmemory=512 |
UI Expand | ||
---|---|---|
| ||
Start by enabling the timing feature in the Agent to see the times the Agent takes to get messages from the queue in the Integration Mesh as well the time it takes to build statements and execute those against the database. This will help determine where the performance bottlenecks are. Contact support@perspectium.com for more information. |
UI Expand | ||
---|---|---|
| ||
You can alter the default <connection_request_timeout> by setting it to 120000. This should give your connection plenty more room to handle all the IO of large transaction. You would place it within your agent.xml like so: <config> <agent> <subscribe> <task> <task_name>timeout_example</task_name> <message_connection connection_request_timeout="120000" user="XXX" password="XXX" >your_url</message_connection> ... <task> <subscribe> </agent> </config> This should be placed on the <message_connection> within the task level of the desired connection. This attribute will only be set for the specified <message_connection>, so if you have separate connections for monitoring or replicating data they will use the default unless specified. Another option is if you have firewall access to both your https and AMQPS connections (https://your_instance.perspectium.net & amqps://your_instance-amqp.perspectium.net) you can try either
|
UI Expand | |||||
---|---|---|---|---|---|
| |||||
|
UI Expand | ||
---|---|---|
| ||
You can add a loginTimeout database parameter to the agent.xml configuration file to control the DB connection timeout. In your agent.xml, under each <task> entry, add <database_parms>loginTimeout=NN</database_parms> where nn is in seconds. For example: <database_parms>loginTimeout=30</database_parms> If you already have <database_parms> configured, then append the loginTimeout parameter using: <database_parms>integratedSecurity=true;loginTimeout=30</database_parms> |
UI Expand | |||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||
|
UI Expand | ||
---|---|---|
| ||
|
MySQL Specific Issues
UI Expand | |||||
---|---|---|---|---|---|
| |||||
|
Microsoft SQL Server Specific Issues
UI Expand | ||
---|---|---|
| ||
The DataSync Agent saves data into the default schema of the database user specified in the agent.xml configuration file. To change the default schema of a user in your SQL Server database, see the DEFAULT_SCHEMA argument.
|
UI Expand | ||
---|---|---|
| ||
If your Agent is continuously showing logs similar to this: 2021-01-01 11:31:35.766 ERROR - psp_agent_subscribe - StatementBuilder - [incident] index: 1 error: org.apache.commons.dbcp2.DelegatingPreparedStatement with address: "NULL" is closed. This may be due to database configurations in your SQL Server causing the connection to timeout before the Agent is able to finish processing messages. This is shown by the "NULL" is closed message and the Agent having to re-establish the connection. To help troubleshoot if this is the case, do the following:
|
Oracle Specific Issues
UI Expand | ||
---|---|---|
| ||
The reason the you're receiving this error is due to Oracle's parameters for SHARED_POOL_SIZE. Note that when SGA_TARGET is set and the parameter is not specified, then the default is 0 (internally determined by the Oracle Database), but if the parameter is specified, then your specified value indicates a minimum value for the memory pool. In the case where a value was set for SGA_TARGET, that would be the value you would need to update rather than the SHARED_POOL_SIZE since by setting SGA_TARGET, you are using automatic SGA management. Hence, there is no need to manually set the value of SHARED_POOL_SIZE because Oracle will internally transfer memory between the SGA components. In the case where you are more concerned with setting a larger value for SGA_TARGET, you can also make a larger value for SHARED_POOL_SIZE but the value must be smaller than SGA_TARGET to avoid encountering the following issue: SGA_TARGET = 1GB
SHARED_POOL_SIZE = no value You will encounter an issue when the value of SHARED_POOL_SIZE exceeds the value of SGA_TARGET. It is recommended to set the SGA_TARGET value at a minimum of 5GB. Therefore, if the SHARED_POOL_SIZE value is at 1GB, the SGA_TARGET will still have at least 4GB for allocation of other memory components that are concurrently stored in SGA_TARGET.
|
UI Expand | |||||
---|---|---|---|---|---|
| |||||
|
Foreign Character Issues
UI Expand | ||
---|---|---|
| ||
Do one of the following:
The format is: <!-- MySQL multibyte decryption --> <database_parms>characterEncoding=UTF-8</database_parms> <!-- SQL Server multibyte decryption (Windows) --> <!-- Note: Requires Agent V3.11.0 or greater --> <database_parms>SendStringParametersAsUnicode=true</database_parms> |
UI Expand | ||
---|---|---|
| ||
Ensure that your database is using the correct character set. Oracle requires the AL32UTF8 character set. MySQL requires utf8mb4 or utf8 character set and utf8_general_ci collation for its character set and collation. In MySQL, you can update this by running the following command on the database: ALTER DATABASE [DBNAME] CHARACTER SET utf8 COLLATE utf8_general_ci; Also verify that you are using the <byte_padding> option in your agent.xml. To setup byte padding please see DataSync Agent configurations. | ||
UI Expand | ||
|
Divbox | ||
---|---|---|
| ||
Yes, the data at rest inside the Perpectium Cloud database can be secured by encryption for an additional cost. Perspectium uses Amazon Web Services which allows the encryption of data using the industry standard AES-256 encryption algorithm. Please refer to the following link for further details on Encrypting Amazon RDS Resources. |
title | Is my connection to the database in the Perspectium Cloud secure? |
---|
Divbox | ||
---|---|---|
| ||
The connection to our database is highly secure. We can also set it up as a secure SSL connection for an additional cost. Please refer to the following link for further details on Using SSL to Encrypt a Connection to a DB Instance. |
title | Which encryption algorithm does Perspectium used for data encrypted at ServiceNow? |
---|
Divbox | ||
---|---|---|
| ||
The cipher Perspectium uses is Triple DES by default and Advanced Encryption Standard 128 (AES-128) as an option. |
title | Does Perspectium support 128 bit AES encryption? |
---|
Divbox | ||
---|---|---|
| ||
Yes, we support 128 bit AES encryption. |
title | When we change our encryption key in ServiceNow, how does that get updated in the conf XML file? |
---|
Divbox | ||
---|---|---|
| ||
This is done manually using a text editor. |
title | Is the authentication data (username/password) encrypted between the agent and the cloud? Or is it sent in clear text? |
---|
Divbox | ||
---|---|---|
| ||
This data can be sent encrypted by utilizing AMQPS on port 5671 or HTTPS. |
title | How does data get encrypted at rest and in transit? |
---|
Divbox | ||
---|---|---|
| ||
The data is encrypted within the ServiceNow Instance before being transmitted to to Perspectium cloud using HTTPS. The payload remains encrypted while at rest within the Perspectium Message Bus until it’s consumed by either a ServiceNow Instance or a DataSync Agent. The former will decrypt the data just prior to being inserted into a ServiceNow table and in the later case the DataSync Agent decrypts the data just before it’s sent to the database server. |
title | Does the Perspectium QA team assess security aspects of the offering during code reviews? |
---|
Divbox | ||
---|---|---|
| ||
Perspectium performs a weekly infrastructure review which includes security. We also perform a daily code review as part of our sprints which includes security as needed. |
title | What are the security features that Perspectium offers? |
---|
Divbox | ||
---|---|---|
| ||
All Perspectium data is transmitted using HTTPS and AMQP/AMQPS as secure protocols. An option of having the data encrypted while in Perspectium's Cloud based Database is available. The on-premise DataSync Agent supports a proxy using HTTP/HTTPS to talk to external servers such as those within the Perspectium Cloud environment. VPN's can be set up at an additional cost. |
title | Can the Perspectium Agent connect to ServiceNow through a proxy server? |
---|
style | background:white |
---|
Can't find what you're looking for?
See additional DataSync Agent troubleshooting topicsContact Perspectium Support or browse the Perspectium Community Forum.