Versions Compared

Old Version 5

changes.mady.by.user user-b885f

Saved on

compared with

New Version Current

changes.mady.by.user user-1bc08

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


HTML
<style>
.release-box {
	height: 30px; 
	width: 100px; 
	padding-top: 8px;
	text-align: center; 
	border-radius: 5px; 
	font-weight: bold; 
	background-color: #0ACA8B;  
	border-color: #FCE28A;
}

.release-box:hover {
  	cursor: hand;
    cursor: pointer;
    opacity: .9; 
}
</style>
<meta name="robots" content="noindex">
<meta name="robots" content="noindex">

<div class="release-box">
<a href="https://docs.perspectium.com/display/fluorineplus" style="text-decoration: none; color: #FFFFFF; display: block;">
Fluorine+
</a>
</div>



ServiceNow App

Divbox


Expand
titleDoes DataSync work with ServiceNow Database Encryption?

ServiceNow provides an option for

ServiceNow FAQs

Database Encryption where data is encrypted at rest in the database. Since Database Encryption happens at the database layer and the Perspectium application runs at the application layer, by the time we call to get data from ServiceNow, the data will be accessible to our application to be shared out.  ServiceNow's documentation mentions with Database Encryption that you can add another level of encryption by also encrypting at the application layer which is what our application supports as well.



Divbox


Expand
titleVerify if your Quebec instance has a properly functioning GlideEncrypter


UI Steps
sizesmall


UI Step

Log in to your ServiceNow instance with admin privileges.


UI Step

In the Filter Navigator, type in Scripts - Background.


UI Step

Run the following script:

Code Block
var ge = new GlideEncrypter();
var plainText = "Some encryption key here";
var encrypted = ge.encrypt(plainText);
gs.print("Encrypting: " + plainText + ", and got: " + encrypted);
var decrypted = ge.decrypt(encrypted);
gs.print("Decrypting: " + encrypted + ", and got: " + decrypted);


If the result is successful, you will see the following:

Code Block
*** Script: Encrypting: Some encryption key here, and got: plzF5fF0yab+qzzglBWoW+co191O2CUx+3l9W2kqQdA=
*** Script: Decrypting: plzF5fF0yab+qzzglBWoW+co191O2CUx+3l9W2kqQdA=, and got: Some encryption key here


if the result is NOT successful, there will be an error displayed with a large stack trace in the ServiceNow System Logs.






DataSync Agent

Divbox


Expand
titleWhat Java processes does the DataSync Agent run?

The Perspectium DataSync Agent leverages two Java virtual machine instances, or processes, to run.

These Java processes can be started as a service or interactively through the executables in the Agent's bin folder.



Divbox


Expand
titleHow can I improve my DataSync Agent's overall performance?

If you find that your DataSync Agent is running slowly due to a large number of SQL statements being processed, you may want to configure the Agent to perform SQL statements in batches.

(info) NOTE: This feature should only be used when you send messages of the same table into one queue. If your queue has messages from different tables (including sys_audit, sys_journal_field, sys_attachment and sys_attachment_doc records), do not enable this feature as it will cause errors in saving records.  To properly use this feature, separate your dynamic and bulk shares to save each type of table record to a different queue.

To do this, open the agent.xml file that was created upon installation of your Agent in a text editing application. Within the <task> directive of the agent.xml file, nest the following directives:

DirectiveDescription
<batch_update/>Self closing tag that configures your Agent to batch SQL UPDATE statements
<batch_insert/>Self closing tag that configures your Agent to batch SQL INSERT statements
<max_batch_size>

Number of SQL statements that will trigger a batch SQL statement execution. A larger suggested value is 200.

(info) NOTE: By default, this directive's value will be set to 10. 




Divbox


Expand
titleShould I configure my DataSync Agent to connect to the Perspectium Mesh via HTTPS or AMQPS?

When you received your Perspectium Mesh credentials, you may have been given two different addresses. Such as:

The choice between these different protocols will vary per customer (largely firewall rules). ServiceNow does not handle AMQP connections, so do not include AMQP within the ServiceNow instance URL for any of your <instance_connection> directives.



Divbox


Expand
titleHow does the Agent handle ServiceNow schema changes?

The DataSync Agent handles schema changes in your ServiceNow instance as follows:

  • Columns that are added to a ServiceNow table will be automatically added to the table in the database.

  • When a column's max size is increased, the Agent will automatically increase the column's size to the maximum size for that database. In the case of MySQL, the column will automatically transition to a CLOB data type.

  • If a column is changed from a different data type to another data type, the data in this column will be skipped (the record itself will insert/update all other columns).

Divbox
Expand
titleWhat if I'm having issues with multibyte characters and foreign language encryption/decryption?

Do one of the following:

  • If you are expecting multibyte characters from ServiceNow it is recommended to turn on multibyte encryption within the Perspectium Properties page.

  • If you are running a MySQL agent it is recommended to place characterEncoding=UTF-8 within the database_parms tag.

  • If you are running a SQL Server agent on Windows then you must be using at least Agent V3.11.0 and include SendStringParametersAsUnicode=false within the Database Parms tag.

The format is:

<!-- MySQL multibyte decryption -->
<database_parms>characterEncoding=UTF-8</database_parms>
<!-- SQL Server multibyte decryption (Windows) -->
<!-- Note: Requires Agent V3.11.0 or greater -->
<database_parms>SendStringParametersAsUnicode=false</database_parms>
Divbox
Expand
titleWhat if I keep seeing timeout errors in my Agent logs?

You can alter the default <connection_request_timeout> by setting it to 120000. This should give your connection plenty more room to handle all the IO of large transaction. You would place it within your agent.xml like so:

<config>
    <agent>
        <subscribe>
            <task>
                <task_name>timeout_example</task_name>
                <message_connection connection_request_timeout="120000" user="XXX" password="XXX" >your_url</message_connection>
                ...
            <task>
        <subscribe>
    </agent>
</config>

This should be placed on the <message_connection> within the task level of the desired connection. This attribute will only be set for the specified <message_connection>, so if you have separate connections for monitoring or replicating data they will use the default unless specified.

Another option is if you have firewall access to both your https and AMQPS connections (https://your_instance.perspectium.net & amqps://your_instance-amqp.perspectium.net

  • )

you can try either

  • Setting your <max_reads_per_connect> to 1 and use the HTTPS connection

  • Setting your <max_reads_per_connect> to 4000 and use the AMQPS connection

Divbox
Expand
titleWhat if I keep seeing database connection timeout errors in my Agent logs?

You can add a loginTimeout database parameter to the agent.xml configuration file to control the DB connection timeout.

In your agent.xml, under each <task> entry, add <database_parms>loginTimeout=NN</database_parms> where nn is in seconds.

For example:

 <database_parms>loginTimeout=30</database_parms>

If you already have <database_parms> configured, then append the loginTimeout parameter using:

 <database_parms>integratedSecurity=true;loginTimeout=30</database_parms>
Divbox
Expand
titleWhat if I'm seeing the following error: java.lang.OutOfMemoryError: Java heap space?

Open the wrapper.conf file located in the Agent's conf folder and change the following configuration:

#wrapper.java.maxmemory=64

Removing the “#” and putting a numeric value higher than 64. This numeric value is a size in MB for the Java memory heap space the agent can use. Generally, you would base this value on the memory available on the server where the Agent is running. For example, if the server has 1GB of memory, you can set it to be 512MB here:

wrapper.java

  • .

maxmemory=512



Divbox


Expand
titleHow does the SQL agent commit to the database?

The DataSync AQL SQL Agent leverages the default connection commit strategy of the JDBC driver for which Oracle is auto commit. The agent does not explicitly decide when to perform a commit. The JDBC driver makes this decision.

The Agent retrieves a message from the message store in the order they were published, performs the required processing such as decryption, validation etc., possibly determines the type of SQL operation required (such as update or insert), and then issues the request to the database. The Agent then determines the response and does any further processing required. Once completed, the Agent will fetch the next message from the message store in the queue.

(info) NOTE: You can configure either multiple tasks to run against a single queue or you can configure multiple instances of a single task to run against a single queue. This is done primarily when throughput of the Agent is an issue. Both of these configurations introduce more than a single consumer of the queue and so the order in which the database transaction occurs could be different than the order of the messages within the message store due to scheduling of the task or thread.



Divbox


ServiceNow provides an option for Database Encryption where data is encrypted at rest in the database. Since Database Encryption happens at the database layer and the Perspectium application runs at the application layer, by the time we call to get data from ServiceNow, the data will be accessible to our application to be shared out.  ServiceNow's documentation mentions with Database Encryption that you can add another level of encryption by also encrypting at the application layer which is what our application supports as well

.

Expand
titleWhy am I getting the following error "ORA-04031: unable to allocate 592 bytes (<-byte size may vary) of shared memory ("shared pool","unknown object","sga heap(1,1)","KGLHD")" on my Agent when I'm replicating to an Oracle Database?

The reason the you're receiving this error is due to Oracle's parameters for SHARED_POOL_SIZE. Note that when SGA_TARGET is set and the parameter is not specified, then the default is 0 (internally determined by the Oracle Database), but if the parameter is specified, then your specified value indicates a minimum value for the memory pool.

In the case where a value was set for SGA_TARGET, that would be the value you would need to update rather than the SHARED_POOL_SIZE since by setting SGA_TARGET, you are using automatic SGA management. Hence, there is no need to manually set the value of SHARED_POOL_SIZE because Oracle will internally transfer memory between the SGA components.

In the case where you are more concerned with setting a larger value for SGA_TARGET, you can also make a larger value for SHARED_POOL_SIZE but the value must be smaller than SGA_TARGET to avoid encountering the following issue:

SGA_TARGET = 1GB
SHARED_POOL_SIZE = no value

You will encounter an issue when the value of SHARED_POOL_SIZE exceeds the value of SGA_TARGET.

It is recommended to set the SGA_TARGET value at a minimum of 5GB. Therefore, if the SHARED_POOL_SIZE value is at 1GB, the SGA_TARGET will still have at least 4GB for allocation of other memory components that are concurrently stored in SGA_TARGET.

(info) NOTE: Be sure to restart the Oracle Database after making the described value changes. For additional information, refer to SHARED_POOL_SIZE or SGA_TARGET.

Should I use a read replica of my database for running my reporting queries?

We only suggest making a read replica database of the database the DataSync Agent is writing if your volumes are above 15 million transactions/day. Our experience is that anything less than 15 million is fine to run your queries against the same database the Agent is writing records to

Divbox
Expand
titleDoes DataSync work with ServiceNow Database Encryption?




Can't find what you're looking for?  

See additional DataSync Agent troubleshooting topics or browse the Perspectium Community Forum.



Similar topics

Content by Label
showLabelsfalse
max5
showSpacefalse
sortmodified
cqllabel = "data-sync" and space = currentSpace()

Contact Perspectium Support

Image ModifiedUS: 1 888 620 8880

UK: 44 208 068 5953

support@perspectium.com