ServiceNow provides an option for Database Encryption where data is encrypted at rest in the database. Since Database Encryption happens at the database layer and the Perspectium application runs at the application layer, by the time we call to get data from ServiceNow, the data will be accessible to our application to be shared out.  ServiceNow's documentation mentions with Database Encryption that you can add another level of encryption by also encrypting at the application layer which is what our application supports as well.

To check what version of the Agent you are running you can use the following command in the command window from the agent directory:

bin\version.bat

You could also double-click the version.bat file in the Agent's bin directory.

The Perspectium DataSync Agent leverages two Java virtual machine instances, or processes, to run.

These Java processes can be started as a service or interactively through the executables in the Agent's bin folder.

When you received your Perspectium Mesh credentials, you may have been given two different addresses. Such as:

• https://user.perspectium.net

• amqps://user.perspectium.net

The choice between these different protocols will vary per customer (largely firewall rules). ServiceNow does not handle AMQP connections, so do not include AMQP within the ServiceNow instance URL for any of your <instance_connection> directives.

The DataSync Agent handles schema changes in your ServiceNow instance as follows:

• Columns that are added to a ServiceNow table will be automatically added to the table in the database.

• When a column's max size is increased, the Agent will automatically increase the column's size to the maximum size for that database. In the case of MySQL, the column will automatically transition to a CLOB data type.

• If a column is changed from a different data type to another data type, the data in this column will be skipped (the record itself will insert/update all other columns).

The DataSync SQL Agent leverages the default connection commit strategy of the JDBC driver for which Oracle is auto commit. The agent does not explicitly decide when to perform a commit. The JDBC driver makes this decision.

The Agent retrieves a message from the message store in the order they were published, performs the required processing such as decryption, validation etc., possibly determines the type of SQL operation required (such as update or insert), and then issues the request to the database. The Agent then determines the response and does any further processing required. Once completed, the Agent will fetch the next message from the message store in the queue.

 NOTE: You can configure either multiple tasks to run against a single queue or you can configure multiple instances of a single task to run against a single queue. This is done primarily when throughput of the Agent is an issue. Both of these configurations introduce more than a single consumer of the queue and so the order in which the database transaction occurs could be different than the order of the messages within the message store due to scheduling of the task or thread.

The HDFSFileSubcriber handler requires the use of the Hadoop external library. You will need the jar files from each of these Hadoop librarires:

1. Hadoop Annotations

2. Hadoop Auth

3. Hadoop Cient

4. Hadoop Common

5. Hadoop HDFS

6. Hadoop Mapreduce

7. Hadoop Yarn

The Maven repository for these files can be found here. Once you have these files they need to be placed in the extlib folder in the File Replicator agent's directory.

• If you find that your DataSync Agent is running slowly due to a large number of SQL statements being processed, you may want to configure the Agent to perform SQL statements in batches.

 NOTE: This feature should only be used when you send messages of the same table into one queue. If your queue has messages from different tables (including sys_audit, sys_journal_field, sys_attachment and sys_attachment_doc records), do not enable this feature as it will cause errors in saving records.  To properly use this feature, separate your dynamic and bulk shares to save each type of table record to a different queue.

• To do this, open the agent.xml file that was created upon installation of your Agent in a text editing application. Within the <task> directive of the agent.xml file, nest the following directives:

• In your agent.xml, change the <polling_interval> to a lower number
• In your agent.xml, increase the number of <tasks>
  •  NOTE: Increasing the number of task will require more memory and connections to the database. We do not recommend setting tasks to more than 10. 
• In your agent.xml, change <message_connection> attributes so use_basic_consume=false
• In ServiceNow, change your bulk share configuration to Insert Only=true

Since Oracle only permits a string of 30 characters by default, a table or a column name that has over 30 characters will be truncated upon replication. The first 15 characters of the name will be displayed, followed by a “_” and then the last 14 of the table name. The “_” denotes the truncated values between the first 15 and the last 14 characters.

For example, when replicating the table “Workflow Estimated Runtime Configuration” (41 characters total) to Oracle database, the replicated name will be “u_workflow_esti__configuration”.

 NOTE: When replicating a table or column that have similar names, meaning if the first 15 and the last 14 of the name are the same, only one of the two records will be replicated since after truncation, both of the names will be the same.

 NOTE: If you are using Oracle 12.2+ with the 12.2 JDBC driver, you can set the table and column names to support Oracle 12.2's larger 128 bytes limit (prior to 12.2, the limit was 30 bytes meaning table and column names could only be up to 30 standard characters).  See here for further information.

Starting with Iodine 7.0.0 release, the Agent will requires Java 11 as the minimum version. By adding the following directive, you will be able to skip the Java version check with agent: 

<system_resource_check>false<system_resource_check/>

Alternatively, if you want to run the Agent without performing a minimum Java requirement check for testing purposes, see Running DataSync Agent in Test Mode.

Yes, the data at rest inside the Perpectium Cloud database can be secured by encryption for an additional cost. Perspectium uses Amazon Web Services which allows the encryption of data using the industry standard AES-256 encryption algorithm. Please refer to the following link for further details on Encrypting Amazon RDS Resources.

The connection to our database is highly secure. We can also set it up as a secure SSL connection for an additional cost. Please refer to the following link for further details on Using SSL to Encrypt a Connection to a DB Instance.

Yes, we support 128 bit AES encryption.

This is done manually using a text editor.

This data can be sent encrypted by utilizing AMQPS on port 5671 or HTTPS.

The data is encrypted within the ServiceNow Instance before being transmitted to to Perspectium cloud using HTTPS. The payload remains encrypted while at rest within the Perspectium Message Bus until it’s consumed by either a ServiceNow Instance or a DataSync Agent. The former will decrypt the data just prior to being inserted into a ServiceNow table and in the later case the DataSync Agent decrypts the data just before it’s sent to the database server.

Perspectium performs a weekly infrastructure review which includes security. We also perform a daily code review as part of our sprints which includes security as needed.

All Perspectium data is transmitted using HTTPS and AMQP/AMQPS as secure protocols. An option of having the data encrypted while in Perspectium's Cloud based Database is available. The on-premise DataSync Agent supports a proxy using HTTP/HTTPS to talk to external servers such as those within the Perspectium Cloud environment. VPN's can be set up at an additional cost.

Yes, you can use a proxy server to connect the Perspectium Agent to ServiceNow v3.2.2. For further details, see Configuring the DataSync Agent to Leverage a Proxy. 

If your records cannot be shared out from ServiceNow to the DataSync Agent successfully, try the following:

1. In ServiceNow, navigate to Perspectium > Tools > Receipts.

2. Check the checkbox next to the Receipt record(s) with a Pending or Error Delivery Status.

3. In the bottom left-hand corner of the screen, click the Resend Message(s) button. This will create a new receipt record in pending status and set the original receipt record to success status. The new resend receipt will remain in the pending status until it gets an acknowledgement from the agent.

4. Refresh the page and check the Delivery Status for your Receipt record(s) again.

If you are seeing the following errors: 

“Windows could not start the Perspectium DataSync Agent service on Local Computer. 
Error 1053: The service did not respond to the start or control request in a timely fashion.”
 
“Windows could not start the Perspectium DataSync Agent service on Local Computer. 
Error 2: The system cannot find the file specified.”

This is usually related to the wrong set up of Java paths and environment variables. Here are the best practice instructions to configure and troubleshoot:

• Check if the Java version installed is supported by the respective agent version DataSync Agent requirements.
  
  

• Verify java runs successfully on the local machine by running on the command line (CMD): 

  java -version

  If you get a positive result with the java version installed then we know java is installed successfully.
  
  

• Verify that the installService.bat script is executed As Administrator to install the service. If not uninstall it and install again “As Administrator”. 
  
  
• Try to reinstall the agent from the command line As Administrator and not from the Windows UI if not done initially. 
  1. Open CMD “As Administrator” 
     
     

  2. Go to the folder where the agent install file is located
     

     cd /folderName/

  3. Enter the following 

     java -jar <installation_file.jar>

  4. Hit Enter
     
     

• Check wrapper.conf file. If you can run “java -version” then in wrapper.conf you can only use wrapper.java.command=java and the JAVA_HOME variable does not need to be set. Like in the example below: 

  # Java Application
  #set.JAVA_HOME= 
  wrapper.java.command=java

• If you still want to use JAVA_HOME environment variable you need to make sure that this variable matches the actual java path to the /bin folder both in wrapper.conf and in Windows Environment Variables
  
  
• Check System Environment Variables - Path variable. When selecting “Edit” for the Path variable you should see in the list a path to the java /bin folder. If not, add one.

If you notice that delete sync is not working properly and you are using MBS3, it may be due to ServiceNow’s limitation of String object of 32 MB. See Server side scripting is failing with Error "String object would exceed maximum permitted size of 33554432". - Support and Troubleshooting - Now Support Portal. 

When sending messages to MBS 3.0, you can put any number of messages in each batch and it will create a file for that batch.

When retrieving the messages from MBS 3.0, it will retrieve one file at a time.

You can configure the Agent to send a set amount of messages in each batch so that ServiceNow can process these messages.

To do so, add <message_batch_size>number of messages in each batch</message_batch_size>. The number must be between 1 - 5000. 

To see if ServiceNow can handle the number of bytes, use the following formula and see if it is less than 33554432 (ServiceNow's limit):

<number set in <message_batch_size>> * <number of characters in one idList message> * 2

NOTE: 2 represent how in Javascript, which ServiceNow use, each character use 2 bytes in a String object.

An idList message can have around 44422 characters, which can be less or more based on the field values (i.e. attributes, key, name) and JSON formatting. You can take an idList message and use a character counter to count the number of characters.

If you are unable to consume messages in your agent, one possible reason could be the log file size. To increase the size, locate the innodb_log_file_size directive, and increase the limit. 

Oracle does not allow data type changes on from NVARCHAR2 to NCLOB. One thing to prevent this from happening for new tables, add in <database_column_max_size> and set a value to your agent configuration. This will allow new columns for new tables to be created as NCLOB if it exceeds the value in <database_column_max_size>. The default value is 251.

Another alternative is to go to your ServiceNow instance and to the related table's fields. For any field that may require a large amount of text, set the max length higher than 251.