DataSync Agent Tasks
The DataSync Agent can be configured to have zero or more tasks. An agent should always have at least one task, since the tasks and their associated handler perform the actual processing.
Each task will have access to its own section of the overall configuration in addition to those directives outlined within the <agent> section of the configuration.
For example here is an agent that:
Subscribes to the queue psp.out.replicator.acme and places the records in psp_repl
Shares the table incident from Test to the queue psp.out.replicator.acme_test
Shares the table problem from psp_repl to the queue psp.out.repcliator.acme_dev
<?xml version="1.0" encoding="ISO-8859-1" standalone="no"?>
<config>
<agent>
<max_reads_per_connect>4000</max_reads_per_connect>
<!-- Subscribing Task -->
<subscribe>
<task>
<task_name>Subscribe</task_name>
<message_connection queue="psp.out.replicator.acme" password="XXXX" user="acme">https://acme.perspectium.net</message_connection>
<instance_connection password="XXXX" user="acme">https://acme.service-now.com</instance_connection>
<handler>com.perspectium.replicator.sql.SQLSubscriber</handler>
<decryption_key>The cow jumped over the moon</decryption_key>
<schedule>* * * * *</schedule>
<database_type>mysql</database_type>
<database_server>localhost</database_server>
<database_port>3306</database_port>
<database_user>root</database_user>
<database_password></database_password>
<database_parms></database_parms>
<database>psp_repl</database>
</task>
</subscribe>
<!-- Sharing Task -->
<share>
<task>
<task_name>Share to Test</task_name>
<handler>com.perspectium.replicator.sql.sharer.SQLSharer</handler>
<message_connection queue="psp.out.replicator.acme_test" password="XXXX" user="acme">https://acme.perspectium.net</message_connection>
<encryption_key>The cow jumped over the moon</encryption_key>
<max_writes_per_connect>50</max_writes_per_connect>
<polling_interval>90</polling_interval>
<database_type>mysql</database_type>
<database_server>localhost</database_server>
<database_port>3306</database_port>
<database_user>root</database_user>
<database_password></database_password>
<database>Test</database>
<table_name>incident</table_name>
<topic>replicator</topic>
<type>agent</type>
<name>.bulk</name>
</task>
</share>
<!-- Sharing Task -->
<share>
<task>
<task_name>Share To Dev</task_name>
<handler>com.perspectium.replicator.sql.sharer.SQLSharer</handler>
<message_connection queue="psp.out.replicator.acme_dev" password="XXXX" user="acme">https://acme.perspectium.net</message_connection>
<encryption_key>The cow jumped over the moon</encryption_key>
<max_writes_per_connect>50</max_writes_per_connect>
<polling_interval>60</polling_interval>
<database_type>mysql</database_type>
<database_server>localhost</database_server>
<database_port>3306</database_port>
<database_user>root</database_user>
<database_password></database_password>
<database>psp_repl</database>
<table_name>problem</table_name>
<topic>replicator</topic>
<type>agent</type>
<name>.bulk</name>
</task>
</share>
</agent>
</config>
Multiple Instances on Subscriber Task
The DataSync Agent can also be configured to have one or more instances running on a single subscriber task. By default, the agent configures each <subscriber> task to use four instances during the replication process. However, you can manually configure your subscriber tasks to use multiple instances. By specifying multiple instances, the agent will run multiple parallel threads when subscribing, which allows it to replicate more data over a given period of time.
For example, here is an agent which configures a task with 4 instances and subscribes to the queue psp.out.replicator.acme.
<?xml version="1.0" encoding="ISO-8859-1" standalone="no"?>
<config>
<agent>
<polling_interval>30</polling_interval>
<!-- Subscribing Task -->
<subscribe>
<task instances="4">
<task_name>Subscribe</task_name>
<message_connection queue="psp.out.replicator.acme" password="XXXX" user="acme">https://acme.perspectium.net</message_connection>
<instance_connection password="XXXX" user="acme">https://acme.service-now.com</instance_connection>
<handler>com.perspectium.replicator.sql.SQLSubscriber</handler>
.
.
.
</task>
</subscribe>
</agent>
</config>
When running the agent, you will see the threads being executed. Here is the sample log output of the four threads created above (Subscribe, Subscribe_2, Subscribe_3, Subscribe_4) running:
INFO - PerformanceReportGenerator - PerformanceReport - Agent: <agent_name>.Subscribe_2 Processed: 128.0 Messages Errors: 0 Queue: psp.out.replicator.acme monitor 4.27 messages per second 379351 bytes processed INFO - PerformanceReportGenerator - PerformanceReport - incident.bulk=[117, 344208]: 117 messages processed and 344208 bytes processed INFO - PerformanceReportGenerator - PerformanceReport - change_request.bulk=[3, 12237]: 3 messages processed and 12237 bytes processed INFO - PerformanceReportGenerator - PerformanceReport - Agent: <agent_name>.Subscribe_4 Processed: 114.0 Messages Errors: 0 Queue: psp.out.replicator.acme monitor 3.80 messages per second 333427 bytes processed INFO - PerformanceReportGenerator - PerformanceReport - incident.bulk=[103, 303011]: 103 messages processed and 303011 bytes processed INFO - PerformanceReportGenerator - PerformanceReport - cert_follow_on_task.bulk=[9, 24560]: 9 messages processed and 24560 bytes processed INFO - PerformanceReportGenerator - PerformanceReport - Agent: <agent_name>.Subscribe Processed: 113.0 Messages Errors: 0 Queue: psp.out.replicator.acme monitor 3.77 messages per second 331079 bytes processed INFO - PerformanceReportGenerator - PerformanceReport - sysapproval_group.bulk=[1, 2928]: 1 messages processed and 2928 bytes processed INFO - PerformanceReportGenerator - PerformanceReport - incident.bulk=[106, 311844]: 106 messages processed and 311844 bytes processed INFO - PerformanceReportGenerator - PerformanceReport - Agent: <agent_name>.Subscribe_3 Processed: 115.0 Messages Errors: 0 Queue: psp.out.replicator.acme monitor 3.83 messages per second 341240 bytes processed INFO - PerformanceReportGenerator - PerformanceReport - sysapproval_group.bulk=[2, 6431]: 2 messages processed and 6431 bytes processed INFO - PerformanceReportGenerator - PerformanceReport - incident.bulk=[100, 294194]: 100 messages processed and 294194 bytes
