The file that controls the configuration settings for Perspectium DataSync Agent is the agent.xml file. This file is automatically created within the Agent's bin directory upon installation. To make changes to the configuration settings for your DataSync Agent, open this agent.xml file in a text editing application and modify the directives described in the sections below.
NOTE: The directives described below are only the essential configurations that must be set for your DataSync Agent. For additional DataSync Agent configuration settings, see the Similar topics list below or browse the topics under DataSync Agent.
Configuration
The Perspectium SQL Subscriber Handler uses the newest version of each of the associated JDBC driver. In a nutshell, the Perspectium SQL subscriber is configured to specify the following:
The IP address or fully qualified domain name of the Database Server
The Port which the database server is listening on
A user which has been configured to access and perform the required database operations
The password associated with the configured database user
The name of the database which will contain the replicated tables
Optionally, the SID if it's an Oracle database and a SID is required
Permissions
After installation has been performed and the Replicator Agent is started, the agent establishes a connection to the database server using the credentials configured as outlined above. This task must be completed successfully before the agent will proceed with subsequent processing. Once the connection has been established, the agent will attempt to create the configured database. This step requires that the configured user has the proper permissions to create a new database. If this is not the case then an error will occur. As an alternative, the required database can be created by something or someone other than the agent. When this approach is taken you add the configuration directive <skip_database_creation> to the configuration and the agent will not attempt to create the database since this tell the agent that the database already exists.
Once replicated data is available, the agent will determine if the table associated with the replicated data exists, and if it's not it will create it. This means that the configured user must have the permissions required to create tables within the defined database.
Once the required table has been created the agent will perform an insert in order to populate the table with the replicated row. This means that the agent must have the permissions to perform an insert into the table.
When you share out data to be consumed by the Replicator Agent, you'll typically specify whether the data is being shared in bulk or dynamically. When the data is shared in bulk and based on how the agent has been configured, it will typically perform a query in order to determine if the row already exists in the database. The agent must therefore have permission to query the database and the associated tables. If the agent determines that the row already exist then it performs an update, otherwise it will insert the row. The agent must have permission to perform both these actions.
Additionally, the agent wants to ensure that all data is replicated. It's possible that the structure of the source table has changed. For example, an additional column is added to the table or the size of a column is increased. The agent performs the analysis required to make this determination and by default will alter the table in order to align it with the source table. Currently, Perspectium supports adding a column, increasing the size of a column to it's maximum and if based upon the database and the type of column altering the data type. The agent must therefore have the permissions to alter the table. It is possible to constrain the agent from altering tables by including the <skip_alter/> directive within the SQL Subscribers <task> definition.
Lastly, in order to support the deletion of rows from a table, the database user must also have permission to delete a table row.
Message batching directives (optional)
To improve the performance for your DataSync Agent, you can optionally add some batching directives that specify what constitutes a batch of messages that will trigger message processing.
To enable these functions, nest one or many of the following directives within the agent.xml's <agent> directive:
Directive | Description |
---|---|
<max_reads_per_connect> | Maximum number of messages to consumer per connection interval |
<max_writes_per_connect> | Maximum number of messages to produce per connection interval |
<polling_interval> | Interval for how often the agent pulls from the queue (seconds)
|
<skip_message_set_processing/> | Self-closing tag that disables Message Set posting for data shared from ServiceNow |
![](/download/attachments/12128742/Message_directives.png)
Share & subscribe
DataSync Agents can perform two main functions, share and subscribe. To enable these functions, nest one or many of the following directives within the agent.xml's <agent> directive:
Directive | Description |
---|---|
<share> | An Agent configured with <share> functionality (also known as a producer) monitors static data sources and places messages from those sources into queues in the Perspectium Mesh. |
<subscribe> | An Agent configured with <subscribe> functionality (also known as a consumer) monitors queues in the Perspectium Mesh and processes any messages in those queues, typically by routing those messages to a static data source.
|
![](/download/attachments/12128742/Share_Subscribe_directives.png)
Task & handler
The <task> directive defines a specific <share> or <subscribe> function that your DataSync Agent should perform. Multiple <task> directives can be nested within the <share> and <subscribe> directives, and individual configurations can be defined for each <task> by nesting further configuration directives (described in the following sections) within each <task>.
Directive | Description |
---|---|
<task> | Task that handles either sharing or subscribing. You can configure separate settings by defining directives within each <task> directive. You can also specify an instances attribute here and give it a value of 4 (default) so that 4 instances of your <share> or <subscribe> task can run concurrently, thus improving throughput for data processing.
|
<task_name> | Name for your <task>
|
<handler> | Name of the Java handler class. In most cases, the value for this directive will be com.perspectium.replicator.sql.SQLSubscriber. |
![](/download/attachments/12128742/Task_directive.png)
Task Configurations
Under the <task> directive are configurations related to the task your DataSync Agent is performing such as saving data to a database. These different configurations are as follows:
Message, instance, and schema connection
The <message_connection> and <instance_connection> directives define how your DataSync Agent connects to the Perspectium Mesh and ServiceNow instance, respectively. For your DataSync Agent to connect to your Salesforce organization, you will need to include the <message_connection> and <schema_connection> directives.
Directive | Required attributes | Description | ||||
---|---|---|---|---|---|---|
<message_connection> | user = Perspectium Mesh username | Perspectium Mesh URL e.g., https://mymesh.perspectium.net
| ||||
password = Perspectium Mesh password
| ||||||
queue = Perspectium Mesh queue name | ||||||
The use_basic_consume attribute will depend on the type of messaging protocol you will be using. e.g., https://mymesh.perspectium.net
| ||||||
<instance_connection> | user = ServiceNow instance username
| ServiceNow instance URL e.g., https://myinstance.service-now.com | ||||
password = ServiceNow instance password
| ||||||
<schema_connection> | user = Salesforce org username | Salesforce org URL e.g., https://login.salesforce.com/services/oauth2/token | ||||
password = Salesforce org password
| ||||||
client_id = Salesforce Consumer Key for the Perspectium Replicator app | ||||||
client_secret = Salesforce Consumer Secret for the Perspectium Replicator app | ||||||
<schema/> | none | Determines which sys_processor ServiceNow will use to retrieve table schemas
|
Encryption key & decryption key
The <encryption_key> directive is used with <share> tasks to encrypt data that is being shared from a static data source to an application. The <decryption_key> directive is used with <subscribe> tasks to decrypt data that was encrypted by a sharing application. The value for the <encryption_key> directive should match the subscribing application's decryption key, and the value for the <decryption_key> direction should match the sharing application's encryption key.
Directive | Description |
---|---|
<encryption_key> | Encryption key used to encrypt data being shared from a static data source to an application |
<decryption_key> | Decryption key used to decrypt data being shared from an application |
Database
There are several directives related to your static data source, or database, that the DataSync Agent connects to. For more information on how to configure your database information in the agent.xml configuration file, see Perspectium SQL Replicator Agent Configuration Guide.
Directive | Description | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
<database_type> | Type of database you are syncing your data to. Supported databases and associated values for the <database_type>:
| ||||||||||||||||||||
<database_server> | URL for your database. If your database is installed on your local machine, set this value to localhost. | ||||||||||||||||||||
<database_port> | Port number for your database | ||||||||||||||||||||
<database_user> | Username used to connect to your database | ||||||||||||||||||||
<database_password> | Password used to connect to your database | ||||||||||||||||||||
<database_parms> | (Optional) Can be used to pass additional parameters to the database server during establishment of the connection For MySQL, this directive will have a default value of autoReconnect=true & characterEncoding=UTF-8 & useSSL=false.
| ||||||||||||||||||||
<database> | Name of your database | ||||||||||||||||||||
<database_column_max_size> | Maximum size of an object that can appear in your database's columns | ||||||||||||||||||||
<database_static_column_size> | Prevents altering the size of an object in your database's columns
| ||||||||||||||||||||
<database_sid> | SID used for your Oracle database
| ||||||||||||||||||||
<byte_padding> | (Optional) Can be used for multi-byte characters by multiplying the size of the columns. Adding an integer to this directive will create columns X times larger.
| ||||||||||||||||||||
<column_sort> | Sorts your database's columns in a specified order.
| ||||||||||||||||||||
<sql_retries> | (Optional) Can be used to configure the number of times the DataSync agent will retry after a SQLException. This directive defaults to 3. NOTE: | ||||||||||||||||||||
<sql_retries_delay> | (Optional) Can be used to configure how many milliseconds the DataSync agent will wait between retries. This directive defaults to 250. NOTE: | ||||||||||||||||||||
<match_field_lengths/> | Extends the column’s length when the ServiceNow table field’s length gets extended. It will never shorten the length. The agent will make sure your column’s length matches with the ServiceNow table field’s length.
|
Other Configurations
Directive | Description |
---|---|
<skip_performance_report> | When enabled, the directive will stop the Agent from reporting performance reporting (statsx) messages to the Integration Mesh.
|
Data Guarantee
These directives configure how the DataSync agent works with Data Guarantee.
Directive | Description |
---|---|
<receipt_send_interval> | (Optional) Indicates the number of seconds in which the Agent will regularly send receipts back to the sharing/publishing instance. This directive defaults to 1800. NOTE: |
<ack_batch_size> | (Optional) Can be used to specify the maximum size of acknowledged receipt messages per batch.
|
<error_batch_size> | (Optional) Can be used to specify the maximum size of error receipt messages per batch.
|
<skip_report/> | (Optional) Adding the self-closing skip_report tag stops the Agent from sending message receipts back to the ServiceNow instance you are sharing from. This option will also turn off the Agent performance reporting as well. |
Example basic agent.xml configuration
The example below shows some of the basic configurations that must be set for a Perspectium DataSync Agent. For additional DataSync Agent configuration settings, see the Similar topics list below or browse the topics under DataSync Agent.
<?xml version="1.0" encoding="ISO-8859-1" standalone="no"?> <config> <agent> <max_reads_per_connect>4000</max_reads_per_connect> <polling_interval>5</polling_interval> <skip_message_set_processing/> <share/> <subscribe> <task instances="1"> <task_name>datasyncforservicenow_to_mysql_subscribe</task_name> <message_connection password="encrypted:vlOtU71yu8N/EFIJH85SSBtaIt7qEEfvqiqft9VZyYE=" queue="psp.out.replicator.testqueue" user="admin" use_basic_consume="false">https://testperspectium.net</message_connection> <instance_connection password="encrypted:vlOtU71yu8N/EFIJH85SSPN9aF0P5/YViVwPEVFcGW4=" user="test.user">https://myinstance.service-now.com</instance_connection> <handler>com.perspectium.replicator.sql.SQLSubscriber</handler> <decryption_key>The cow jumped over the moon</decryption_key> <database_type>mysql</database_type> <database_server>localhost</database_server> <database_port>3306</database_port> <database_user>root</database_user> <database_password>encrypted:vlOtU71yu8N/EFIJH85SSMoilKLTeJHQrNZPJ7c5tFU=</database_password> <database_parms>autoReconnect=true & characterEncoding=UTF-8 & useSSL=false</database_parms> <database_column_max_size>251</database_column_max_size> <database_static_column_size/> <database>psp_repl</database> <byte_padding>4</byte_padding> <column_sort>alpha</column_sort> </task> </subscribe> </agent> </config>