Monitoring queues

Set up monitor thresholds

This feature allows you to define alert and warning thresholds for queue connectivity issues and backlog of records in a cue.

Here's how:

1. In your sharing ServiceNow instance, go to Perspectium > Perspectium Core > Shared Queues.
   
   
2. On the resulting page, find and select the shared queue that you want to configure.
   
   

3. On the resulting shared queue form, click the Monitor box. This will reveal a variety of new monitoring fields below.
   
   

4. In the newly revealed monitoring fields, the only requirement is to select an option from the Monitor Polling Interval dropdown. This indicates the repeated interval that the background job will run at when checking to see if the thresholds are passed.
   
   
5. Now, use the various threshold fields to set connection thresholds, record thresholds, or both. Connection thresholds will trigger a warning/alert if your instance cannot properly connect to the queue for the provided duration. Record thresholds will trigger a warning/alert if the number of messages in the queue pass the indicated threshold. If both warnings and alerts thresholds are passed, the system will send out only an alert. 

View alerts and warnings

Once you've set up monitoring thresholds for your queue, here's how you can view the alerts and warnings.

Monitor Status

This appears at the bottom of the monitoring fields on the shared queue. It holds contents of alerts/warnings.

Dashboard

You can also view the monitor status right on the DataSync homepage.

Alert & Notification

You can set this up to create a log error int he Perspectium Logs (u_psp_log_message) on the instance, as well as an Alert (u_psp_alerts) on the instance. Then, you can use the Error Notifications module to set up a notification corresponding to this alert. You can also apply filters onto this alert table to capture only those with Name = Perspectium Queue Monitor, or whichever alerts you would like notifications from. You could similarly build a custom notification record based off of the Alerts table. 

Queue History

Back in the queue form under the monitoring fields, check the Track History to track the history of the queue. Then, you can click the View History link under Related Links to see a visualization similar to the one below:

This chart will load the last 7 days of data by default. You can pull up the last 3, 7, 14, or 30 days as well using the links underneath the chart. Data by default is also deleted when it is over 35 days old. This can be changed by going to the table u_psp_properties and adjusting the value of the record with name com.perspectium.queue_history.days_retained from 35 to the number of days you would like.

Additionally, you can view the metrics of your queue in the Queue History (u_psp_queue_history) table.

↑ Go to top of page

Purge queue

This feature allows you to purge a queue of all messages in it. This is useful if you run a large bulk share and no longer want or need a subscriber to consume the messages. 

Here's how to purge a queue:

1. In your ServiceNow instance, go to Perspectium > Perspectium Core > Shared Queues.
   
   
2. On the resulting page, find and click the queue you want to purge
   
   
3. On the resulting queue page, under Related Links, click Purge Queue. 

↑ Go to top of page

Number of Queues (Child Queues)

With Krypton 8.1.0+, the Number of Queues option on the Miscellaneous section allows you to specify the number of child shared queues created for sharing out data to optimize sharing.

This way you can create multiple queues to use with a bulk share without having to manually create each shared queue one by one. The queue specified in the shared queue record is considered the base shared queue while any other queues created related to this feature are considered child shared queues. All shared queues have the same capability for sharing data to using the Perspectium application as this feature is meant to simplify the shared queue creation process so you don't have to do it all manually.

Using the example of creating a queue named psp.out.replicator.yourqueue as specified in the Queue name field, the option you choose will create queues as follows:

 NOTE: 

• Child queues can't be modified except for setting them individually active or inactive, enabling/disabling queue monitoring or updating the Team field. All other options are controlled from the base queue so the configurations are copied down to the child queues. For example, if you change the Endpoint URL on the base queue, this will update the Endpoint URL value on all its related child queues as well.
• Changing the Number of Queues option to a smaller value will delete all its related child queues beyond the new range. For example, if you change Number of Queues from 8 to 4, this will delete child queues psp.out.replicator.yourqueue05, psp.out.replicator.yourqueue06, psp.out.replicator.yourqueue07 and psp.out.replicator.yourqueue08. Changing the value to 1 will delete all child queues as 1 indicates only the base queue.
• Deleting the base queue will delete all the related child queues.
• Get Queue Status works on each queue individually including child queues.

After setting up a shared queue with child queues using this feature, you can use these queues in a bulk share to share records to multiple queues in parallel to optimize performance while avoiding data loss. See bulk share for more information on how to split the sharing with this feature.

↑ Go to top of page

Get Queue Status

To check the status for your queue, click the Get Queue Status link under Related Links at the bottom left-hand corner of the form. Information about the queue status will be displayed in the Status field.

If you are using the Encrypt on Send option (see below for more information), a message stating NOTE: This queue uses the 'Encrypt on send' option to send one encrypted message with a batch of records each time data is sent to MBS. If you are using Perspectium MBS2, the queue count will represent only the encrypted batch messages and not each message you are sharing. will be added to the end of the status information:

With this approach, the number of records that are in the queue may not match the number of records you shared out.

This is due to the MBS you are connected to (which is purposely not available to the Perspectium Core application including the version number so as to not be flagged by application security testing platforms). Since MBS3 maintains a separate state of each queue, there are enhancements supported by both MBS3 and the Perspectium Core application so we can display the actual number of records posted when we post the entire batch encrypted.

With MBS2, this information isn't available so Get Queue Status will display the number of encrypted batches in the queue. For example, if there are 2000 records shared and we encrypted 2 batches (1000 records each) into the queue that have not been consumed, this will show that there are 2 records in the queue.  NOTE: The batching is not the same number of records each time as its depending on how large the combined total of all the records are.

Encrypt on Send

The Encrypt on Send feature will encrypt the entire batch of records sent to the Perspectium Integration Mesh (MBS) versus encrypting each record one by one when they are put into the outbound (psp_out_mesage) table when sharing.

With this option selected, any bulk/dynamic share that uses the queue will have their encryption method set to Base64 Encode Only to base64 encode the records as placed into the outbound table and then when the Perspectium Core application runs its scheduled job to take a batch of records from the outbound table and send those to MBS, the batch will first be encrypted with AES128 or AES256 using the encryption key specified in the shared queue before being sent to MBS.

Here's how you set up the Encrypt on Send option:

1. In your ServiceNow instance, go to Perspectium > Perspectium Core > Shared Queues.
   
   
2. On the resulting page, find and click the queue you want to enable the Encrypt on Send option.
   
   

3. Go to the Miscellaneous tab and check the Encrypt on Send box.
   
   

4. The Encryption Method dropdown will appear so you can select what type of encryption method to use. You can select either AES128 or AES256 as encryption options (Triple DES is not supported since it is no longer consider a secure encryption method, see ServiceNow Encryption Options for more information).  NOTE: AES256 requires an encryption key of 32+ characters.
   
   
5. Save your changes to the queue.
   
   
6. Execute your bulk share or share data out with your dynamic share. 

↑ Go to top of page

Generate Encryption Key

For cases where you want to enter a new encryption key for a queue (such as when you disabled copying the default encryption key or you are creating a new integration where the key will be different from other integrations), the Generate encryption key UI action provides a quick way to randomly create a new encryption key.

The key created will be 32 characters in length and be made up of a random mix of uppercase letters, lowercase letters, numbers and symbols.

The newly generated key will be displayed at the top of the queue form so you can copy it down to enter in the target application that will decrypt messages such as the DataSync Agent, Meshlets or another ServiceNow instance with the Perspectium application installed:

 NOTE: After you click Generate encryption key, this key will be copied into the queue's Queue Encryption Key field. You will need to save the record for the key to be saved into the queue's configuration. Otherwise the queue will use the previous encryption key saved.