Set up the DataSync Agent to share to Azure Blob Storage

Perspectium DataSync Agents support the replication of data from your app to an Azure Blob Storage container, or an Azure Blob Storage Subscriber Agent. By configuring your Azure Blob Storage Subscriber Agent, data from your app can be replicated and then saved as either .json or .xml file(s) in your Azure Blob Storage container.

Prerequisites

 First, you will need to set up the Perspectium DataSync Agent.

 You should also stop running your DataSync Agent before making any Agent configuration changes.

Procedure

The DataSync Agent supports saving records into files that can be pushed to Azure Blob Storage containers as follows:

• One Record Per File - Each record is saved into its own file
• Multiple Records Per File - Multiple records are saved into a file before a new file is created and you can control the max number of records or the maximum file size along with separating the files by table
• One File for All Records - All the records of a batch, regardless of which ServiceNow table it belongs to, will be saved into one file for pushing to a Azure Blob Storage container

See the Sample Configurations section for sample configurations for each.

To configure your DataSync Agent to run as an Azure Blob Storage Subscriber Agent, follow these steps:

<file_prefix>$table_$d{yyyyMMdd}_</file_prefix>

[
{"field1":"record1val1", "field2","record1val2"}
,{"field1":"record2val1", "field2","record2val2"}
,{"field1":"record3val1", "field2","record3val2"} 
]

Sample Configurations

You have three different ways of saving files that are then pushed to Azure Blob Storage containers. Below is a sample configurations for each along with details. In all examples, files will be named with a <randomized_unique_identifier> value (such as 00b470b7-901c-4447-9316-023a265d632f) to ensure there are no file naming collisions when saving to the Azure Blob Storage container.

One Record Per File

In this setup, each record is saved into its own file that is then pushed into the Azure Blob Storage container.

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <agent>
      <share />
      <subscribe>
         <task>             
			<task_name>azure_blob_storage_agent_subscribe</task_name>
            <message_connection password="password" user="user">https://<customer>.perspectium.net</message_connection>
    	    <instance_connection password="password" user="user">https://<instance>.service-now.com</instance_connection>   
            <handler>com.perspectium.replicator.file.AzureBlobStorageSubscriber</handler>             
			<decryption_key>Some decryption key here</decryption_key>             
			<abs_container>pspcontainer</abs_container>             
			<connection_string>your_connection_string</connection_string>             
			<file_format>json</file_format>
            <file_prefix>record_</file_prefix>
			<file_suffix>.json</file_suffix>
		 	<files_directory>datasyncfile/azureblobfiles</files_directory>
            <one_record_per_file/>
         </task>
      </subscribe>
      <polling_interval>40</polling_interval>
   </agent>
</config>

In this case, each record will be saved in its own file named <file_prefix><randomized_unique_identifier><file_suffix>.  Using the above configuration example, a file would be named record_00b470b7-901c-4447-9316-023a265d632f.json.

Multiple Records Per File

In this setup, multiple records are saved into a file before a new file is created. This way we're creating batches of records in one file so we're not creating too many files while also not creating files that are too large as well. This option also allows you to specify multiple records of one table into their own files so each file can only contain multiple records of the same table i.e. one file only contains incident records, one file only contains ticket records, etc.

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <agent>
      <share/>
      <subscribe>
         <task>             
			<task_name>azure_blob_storage_agent_subscribe</task_name>
            <message_connection password="password" user="user">https://<customer>.perspectium.net</message_connection>
    	    <instance_connection password="password" user="user">https://<instance>.service-now.com</instance_connection>   
            <handler>com.perspectium.replicator.file.AzureBlobStorageSubscriber</handler>             
			<decryption_key>Some decryption key here</decryption_key>             
			<abs_container>pspcontainer</abs_container>             
			<connection_string>your_connection_string</connection_string>             
			<file_format>json</file_format>              			
			<file_prefix>$table_$d{yyyyMMddHHmm}_</file_prefix>
            <file_suffix>.json</file_suffix>
		 	<files_directory>datasyncfile/azureblobfiles</files_directory>						
			<buffered_writes>300</buffered_writes> 
         </task>
      </subscribe>
      <polling_interval>40</polling_interval>
   </agent>
</config>

In the above example, we specify the files to be named <file_prefix><randomized_unique_identifier><file_suffix> which in this case will be based on the table and a datetime value that is today's date with the current hour and minutes, while saving up to 300 records per file using the <buffered_writes> configuration. Once the <buffered_writes> maximum has been reached, a new file at that datetime with a new randomized_unique identifier will be created. 

 NOTE: <buffered_writes> and <file_max_size> are used while processing each batch of records of records the Agent pulls down from MBS. As the Agent processes each batch, it will push the files created from this batch of records to the Azure Blob Storage container and then repeat the process with the next batch of records it pulls down from MBS, creating new files with new <file_prefix><randomized_unique_identifier><file_suffix> names. So this means that there may be files that have less records than <buffered_writes> or smaller in size than <file_max_size> that are pushed to the Azure Blob Storage container. For example, if incident records are shared and the Agent processes them in two different batches, this will result in files named:

incident_202501132122_4d747761-62d0-49ae-87fc-4998e259727d.json

incident_202501132122_5af73396-5bd9-49e1-8ac5-da8fb8a12047.json

incident_202501132124_6445a8ab-f694-482c-a315-01192cc84d7a.json

incident_202501132124_e4343ccb-a344-212c-b213-00055cc34a3b.json

See the <buffered_writes> and <file_max_size> configurations above for more information.

One File for All Records

In this setup, when the Agent receives a batch of records from MBS, all the records, regardless of which ServiceNow table it belongs to, will be saved into one file that is then pushed to Azure Blob Storage. This process is repeated each time the Agent receives a new batch of records from MBS (which is controlled by your <polling_interval> configuration for how often the Agent polls MBS for another batch of records).

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <agent>
      <share/>
      <subscribe>
         <task>
            <task_name>azure_blob_storage_agent_subscribe</task_name>
            <message_connection password="password" user="user">https://mesh.perspectium.net</message_connection>
    	    <instance_connection password="password" user="user">https://myinstance.service-now.com</instance_connection>   
            <handler>com.perspectium.replicator.file.AzureBlobStorageSubscriber</handler>
            <decryption_key>Some decryption key here</decryption_key>             
			<abs_container>pspcontainer</abs_container>             
			<connection_string>your_connection_string</connection_string>
		 	<file_format>json</file_format>
		 	<files_directory>datasyncfile/azureblobfiles</files_directory>
          </task>
      </subscribe>
      <polling_interval>40</polling_interval>
   </agent>
</config>

The files saved in the Azure Blob Storage container will be named <task_name>.<randomized_unique_identifier>.<file_format>.  Using the above configuration example, a file would be named azure_blob_storage_agent_subscribe.00b470b7-901c-4447-9316-023a265d632f.json and then this file would be pushed into your Azure Blob Storage container.