You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 12 Next »

Krypton


This guide will provide you the optimization of your DataSync integration for the best performance while avoiding issues with data updates happening incorrectly and out of order in your database.

To optimize performance while avoiding data loss, we will split the sharing on the hexadecimal numbering system (16 characters, 0-9 and A-F) used for ServiceNow sys_id values and share records to different queues. So you can optimize sharing by splitting it based on the 16 characters, having Perspectium jobs share data with 2, 4, 8 or 16 jobs running in parallel with each job sharing to a different queue.

This enables higher performance since multiple jobs are running in parallel to share data. With each job using a different queue and us splitting the sharing by the sys_id hexadecimal character, we ensure that any updates to one particular record goes to the same queue all the time. This way, we capture all updates to the record since the records are fed first in first out to the same queue and you avoid the issues of records being shared out of order and as a result causing changes to be missed.

For example, if you are splitting the sharing as follows:

One bulk share runs every minute to capture all updates to records starting with sys_id 1 go to psp.out.replicator.queue01

A second bulk share runs every minute to capture all updates to records starting with sys_id 2 go to psp.out.replicator.queue02

Then if you the following sequence happens:

  1. sys_id 1 record is created → goes to psp.out.replicator.queue01 (first message in the queue)
  2. sys_id 2 record is created → goes to psp.out.replicator.queue02 (first message in the queue)
  3. sys_id 1 record is updated → goes to psp.out.replicator.queue01 (second message in the queue)
  4. sys_id 1 record is updated → goes to psp.out.replicator.queue01 (third message in the queue)

And since we have two jobs running, these changes will be capture in parallel and sent to the two queues for processing by the DataSync Agent. With the DataSync Agent configured properly as described below with one thread each running against those two queues, it would consume as follows:

  1. DataSync Agent thread 1 configured to read psp.out.replicator.queue01 would read the first message
  2. DataSync Agent thread 2 configured to read psp.out.replicator.queue02 would read the second message
  3. DataSync Agent thread 1 configured to read psp.out.replicator.queue01 would read the second message
  4. DataSync Agent thread 1 configured to read psp.out.replicator.queue01 would read the third message

Thus ensuring we capture all updates and in the correct order.


If we instead had the legacy approach of creating two jobs running and sharing to the same queue we would have the records as follows in the queue:

  1. sys_id 1 record is created → goes to psp.out.replicator.queue (first message in the queue)
  2. sys_id 2 record is created → goes to psp.out.replicator.queue (second message in the queue)
  3. sys_id 1 record is updated → goes to psp.out.replicator.queue (third message in the queue)
  4. sys_id 1 record is updated → goes to psp.out.replicator.queue (fourth message in the queue)

And then if we had the legacy approach of the DataSync Agent with two threads processing from the same queue, it in some cases (depending on how fast each thread is performing) processes the records in the following sequence:

  1. DataSync Agent thread 1 configured to read psp.out.replicator.queue would read the first message and finishes processing
  2. DataSync Agent thread 2 configured to read psp.out.replicator.queue would read the second message and finishes processing
  3. DataSync Agent thread 1 configured to read psp.out.replicator.queue would read the third message and process this message slower this time
  4. DataSync Agent thread 2 configured to read psp.out.replicator.queue would read the fourth message and finish processing before thread 1 finishes the third message

Which would cause the database to then reflect an incorrect update as the database would reflect whichever thread finishes last (in this case thread 2), resulting in sys_id 1 record showing incorrect data since it reflects an older update.

So the goal of the configuration below is to ensure we optimize performance while at the same avoid any data loss.

(info) NOTE: Running multiple jobs in parallel means more ServiceNow scheduled job workers are being used. So you do want to choose the option that best suits your workload and data being shared to ensure optimal performance of your instance.

ServiceNow Configuration

Create a shared queue

Create a shared queue using the Number of Queues option, choosing an option to match the workload expected. 

Create a bulk share using the created shared queue


Agent Configuration

Create a message connection

For the shared queue you created in ServiceNow above, you will create a <message_connection> with the num_queues attribute matching what you specified in the shared queue. This will create threads to consume the child queues along with the base queue.

For example, if you specify queue=

For example, with the configuration:

<message_connection password="password" queue="psp.out.replicator.prodqueue" user="user" num_queues="4">amqp://instance.perspectium.net</message_connection>

The Agent will create 5 threads, one thread to read from the base queue psp.out.replicator.prodqueue as well as a separate thread for each of the four child queues as follows:

psp.out.replicator.prodqueue01

psp.out.replicator.prodqueue02

psp.out.replicator.prodqueue03

psp.out.replicator.prodqueue04

The base queue psp.out.replicator.prodqueue can be used as a "fast lane" for records that need to have priority for sharing into the database. By default with this optimization setup, the bulk share will use the child queues 01, 02, 03 and 04 to evenly distribute the records to be shared out of the table the bulk share is running on in ServiceNow. 


  • No labels