Page History
HTML |
---|
<style> .release-box { height: 30px; width: 100px; padding-top: 8px; text-align: center; border-radius: 5px; font-weight: bold; background-color: #8efeb3; border-color: #FCE28A; } .release-box:hover { cursor: hand; cursor: pointer; opacity: .9; } </style> <meta name="robots" content="noindex"> <div class="release-box"> <a href="https://docs.perspectium.com/display/krypton" style="text-decoration: none; color: #FFFFFF; display: block;"> Krypton </a> </div> |
This guide will provide you the optimization of your DataSync integration for the best performance while avoiding issues with data updates happening incorrectly and out of order in your database.
ServiceNow Configuration
Create a shared queue
Create a shared queue using the Number of Queues option, choosing an option to match the workload expected.To optimize performance while avoiding data loss, we will split the sharing on the hexadecimal numbering system (16 characters, 0-9 and A-F) used for ServiceNow sys_id values and share records to different queues. So you can optimize sharing by splitting
the sharingit based on the 16 characters, having Perspectium jobs
run toshare data with 2, 4, 8 or 16 jobs running in parallel with each job sharing to a different queue.
This enables higher performance since multiple jobs are running in parallel to share data. With each job using a different queue and us splitting the sharing by the sys_id hexadecimal character, we ensure that any updates to one particular record goes to the same queue all the time. This way, we capture all updates to the record since the records are fed first in first out to the same queue and you avoid the issues of records being shared out of order and as a result causing changes to be missed.
For example, if you are splitting the sharing as follows:
One bulk share runs every minute to capture all updates to records starting with sys_id 1 go to psp.out.replicator.queue01
A second bulk share runs every minute to capture all updates to records starting with sys_id 2 go to psp.out.replicator.queue02
Then if you the following sequence happens:
- sys_id 1 record is created → goes to psp.out.replicator.queue01 (first message in the queue)
- sys_id 2 record is created → goes to psp.out.replicator.queue02 (first message in the queue)
- sys_id 1 record is updated → goes to psp.out.replicator.queue01 (second message in the queue)
- sys_id 1 record is updated → goes to psp.out.replicator.queue01 (third message in the queue)
And since we have two jobs running, these changes will be capture in parallel and sent to the two queues for processing by the DataSync Agent. With the DataSync Agent configured properly as described below with one thread each running against those two queues, it would consume as follows:
- DataSync Agent thread 1 configured to read psp.out.replicator.queue01 would read the first message
- DataSync Agent thread 2 configured to read psp.out.replicator.queue02 would read the second message
- DataSync Agent thread 1 configured to read psp.out.replicator.queue01 would read the second message
- DataSync Agent thread 1 configured to read psp.out.replicator.queue01 would read the third message
Thus ensuring we capture all updates and in the correct order.
If we instead had the legacy approach of creating two jobs running and sharing to the same queue we would have the records as follows in the queue:
- sys_id 1 record is created → goes to psp.out.replicator.queue (first message in the queue)
- sys_id 2 record is created → goes to psp.out.replicator.queue (second message in the queue)
- sys_id 1 record is updated → goes to psp.out.replicator.queue (third message in the queue)
- sys_id 1 record is updated → goes to psp.out.replicator.queue (fourth message in the queue)
And then if we had the legacy approach of the DataSync Agent with two threads processing from the same queue, it in some cases (depending on how fast each thread is performing) processes the records in the following sequence:
- DataSync Agent thread 1 configured to read psp.out.replicator.queue would read the first message and finishes processing
- DataSync Agent thread 2 configured to read psp.out.replicator.queue would read the second message and finishes processing
- DataSync Agent thread 1 configured to read psp.out.replicator.queue would read the third message and process this message slower this time
- DataSync Agent thread 2 configured to read psp.out.replicator.queue would read the fourth message and finish processing before thread 1 finishes the third message
Which would cause the database to then reflect an incorrect update as the database would reflect whichever thread finishes last (in this case thread 2), resulting in sys_id 1 record showing incorrect data since it reflects an older update.
So the goal of the configuration below is to ensure we optimize performance while at the same avoid any data loss.
NOTE: Running multiple jobs in parallel means more ServiceNow scheduled job workers are being used. So you do want to choose the option that best suits your workload and data being shared to ensure optimal performance of your instance.
ServiceNow Configuration
UI Steps | ||||
---|---|---|---|---|
|
Agent Configuration
UI Steps | ||||
---|---|---|---|---|
|