Bulk Processing on a Queue

There are two main strategies behind creating multiple MultiOutput jobs. The strategy you will use will depend on your use case. 

The first strategy is bulk processing on a queue. This strategy refers to wanting to process a high volume of messages on a specific queue. If you are bulk sharing a large amount of messages for a single queue, then this is the path you should lean towards. This strategy sets up sharing to divide the work for a queue into small distinct chunks - and having multiple jobs each process a chunk. The primary way to do this is by querying off of the sys_id of the outbound message.

 NOTE: This method queries off the sys_id of the outbound message itself and not the record that the outbound message represents. Additionally, we normally share out the records in a way to preserve sequencing on a single queue, whereas this method does not honor that sequencing. This strategy is only recommended if you are bulk sharing a large set of data and are not concerned about the order they arrive in.

For an easy way to set up this strategy, go to Perspectium > Replicator Tools then click on Create Multiple MultiOutput Jobs. This will direct you to a UI where you can choose the number of output jobs you want running.

When the jobs are created, the default job Perspectium MultiOutput Processing is renamed XPerspectium MultiOutput Processing and set inactive, and new jobs are created in the format of Perspectium MultiOutput Processing [*] where * is a list of characters between 0-9 and a-f. The list of characters represents the first characters of sys_ids of outbound records that are processed by the job. For example, [8, 9, a, b] would mean any outbound records with sys_ids starting with 8, 9, a, or b will be processed by this output job. To restore default settings, first delete the new jobs then reactivate and rename the default job.

Procedure

To create multiple MultiOutput jobs using bulk processing on a queue strategy, follow these steps:

Create a filter on your current outbound messages based on the sys_id starts with X flag. The sys_ids for these records start with (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, a, b, c, d, e, f) for 16 values. We want to make sure we capture each of these distinctly.

The following is an example where we break these into groups of 4:

Create the other 3 jobs similarly to the step above. You may also want to limit this to a distinct queue by passing in a target queue into the encoded query.

The following is an example of the script: 

try {
    var encodedQuery="sys_idSTARTSWITH0^ORsys_idSTARTSWITH1^ORsys_idSTARTSWITH2^ORsys_idSTARTSWITH3";
 
    var psp = new Perspectium();
    psp.processMultiOutput(encodedQuery);
}
catch(e) {
    var logger = new PerspectiumLogger();
    logger.logError("error = " + e, "Perspectium MultiOutput Processing");
}