Set up the DataSync Agent to share to AWS S3

Perspectium DataSync Agents support the replication of data from your app to an Amazon Web Services (AWS) S3 bucket, or an AWS S3 Subscriber Agent. By configuring your AWS S3 Subscriber Agent, data from your app can be replicated and then saved as either .json or .xml file(s) in your AWS S3 bucket.

Prerequisites

 First, you will need to set up the Perspectium DataSync Agent.

 You should also stop running your DataSync Agent before making any Agent configuration changes.

Procedure

The DataSync Agent supports saving records into files that can be pushed to AWS S3 buckets as follows:

• One Record Per File - Each record is saved into its own file
• Multiple Records Per File - Multiple records are saved into a file before a new file is created and you can control the max number of records or the maximum file size along with separating the files by table
• One File for All Records - All the records of a batch, regardless of which ServiceNow table it belongs to, will be saved into one file for pushing to an AWS S3 bucket

See the Sample Configurations section for sample configurations for each.

To configure your DataSync Agent to run as an AWS S3 Subscriber Agent, follow these steps:

<file_prefix>$table_$d{yyyyMMdd}_</file_prefix>

[
{"field1":"record1val1", "field2","record1val2"}
,{"field1":"record2val1", "field2","record2val2"}
,{"field1":"record3val1", "field2","record3val2"} 
]

Sample Configurations

You have three different ways of saving files that are then pushed to AWS S3 buckets. Below is a sample configurations for each along with details. In all examples, files will be named with a <randomized_unique_identifier> value (such as 00b470b7-901c-4447-9316-023a265d632f) to ensure there are no file naming collisions when saving to the S3 bucket.

One Record Per File

In this setup, each record is saved into its own file that is then pushed into the AWS S3 bucket.

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <agent>
      <share />
      <subscribe>
         <task>
            <task_name>s3_agent_subscribe</task_name>
            <message_connection password="password" user="user">https://<customer>.perspectium.net</message_connection>
    	    <instance_connection password="password" user="user">https://<instance>.service-now.com</instance_connection>   
            <handler>com.perspectium.replicator.file.S3Subscriber</handler>
            <decryption_key>My special key for decrypting the data</decryption_key>
            <access_key>AccessKey</access_key>
            <secret_access_key>SecretAccessKey</secret_access_key>
            <region>us-west-2</region>
            <s3_bucket>examples3bucket</s3_bucket>
            <file_format>json</file_format>
            <file_prefix>record_</file_prefix>
			<file_suffix>.json</file_suffix>
		 	<files_directory>datasyncfile/azureblobfiles</files_directory>
            <one_record_per_file/>
         </task>
      </subscribe>
      <polling_interval>40</polling_interval>
   </agent>
</config>

In this case, each record will be saved in its own file named <file_prefix><randomized_unique_identifier><file_suffix>.  Using the above configuration example, a file would be named record_00b470b7-901c-4447-9316-023a265d632f.xml.

Multiple Records Per File

In this setup, multiple records are saved into a file before a new file is created. This way we're creating batches of records in one file so we're not creating too many files while also not creating files that are too large as well. This option also allows you to specify multiple records of one table into their own files so each file can only contain multiple records of the same table i.e. one file only contains incident records, one file only contains ticket records, etc.

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <agent>
      <share />
      <subscribe>
         <task>
            <task_name>s3_agent_subscribe</task_name>
            <message_connection password="password" user="user">https://<customer>.perspectium.net</message_connection>
    	    <instance_connection password="password" user="user">https://<instance>.service-now.com</instance_connection>   
            <handler>com.perspectium.replicator.file.S3Subscriber</handler>
            <decryption_key>My special key for decrypting the data</decryption_key>
            <access_key>AccessKey</access_key>
            <secret_access_key>SecretAccessKey</secret_access_key>
            <region>us-west-2</region>
            <s3_bucket>examples3bucket</s3_bucket>
            <file_format>json</file_format>			
			<file_prefix>$table_$d{yyyyMMddHHmm}_</file_prefix>
            <file_suffix>.json</file_suffix>	
		 	<files_directory>datasyncfile/azureblobfiles</files_directory>					
			<buffered_writes>300</buffered_writes>			
         </task>
      </subscribe>
      <polling_interval>40</polling_interval>
   </agent>
</config>

In the above example, we specify the files to be named <file_prefix><randomized_unique_identifier><file_suffix> which in this case will be based on the table and a datetime value that is today's date with the current hour and minutes, while saving up to 300 records per file using the <buffered_writes> configuration. Once the <buffered_writes> maximum has been reached, a new file at that datetime with a new randomized_unique identifier will be created. 

 NOTE: <buffered_writes> and <file_max_size> are used while processing each batch of records of records the Agent pulls down from MBS. As the Agent processes each batch, it will push the files created from this batch of records to the AWS S3 bucket and then repeat the process with the next batch of records it pulls down from MBS, creating new files with new <file_prefix><randomized_unique_identifier><file_suffix> names. So this means that there may be files that have less records than <buffered_writes> or smaller in size than <file_max_size> that are pushed to the AWS S3 bucket. For example, if incident records are shared and the Agent processes them in two different batches, this will result in files named:

incident_202501132122_4d747761-62d0-49ae-87fc-4998e259727d.json

incident_202501132122_5af73396-5bd9-49e1-8ac5-da8fb8a12047.json

incident_202501132124_6445a8ab-f694-482c-a315-01192cc84d7a.json

incident_202501132124_e4343ccb-a344-212c-b213-00055cc34a3b.json

See the <buffered_writes> and <file_max_size> configurations above for more information.

One File for All Records

In this setup, when the Agent receives a batch of records from MBS, all the records, regardless of which ServiceNow table it belongs to, will be saved into one file that is then pushed to AWS S3. This process is repeated each time the Agent receives a new batch of records from MBS (which is controlled by your <polling_interval> configuration for how often the Agent polls MBS for another batch of records).

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <agent>
      <share />
      <subscribe>
         <task>
            <task_name>s3_agent_subscribe</task_name>
            <message_connection password="password" user="user">https://mesh.perspectium.net</message_connection>
    	    <instance_connection password="password" user="user">https://myinstance.service-now.com</instance_connection>   
            <handler>com.perspectium.replicator.file.S3Subscriber</handler>
            <decryption_key>My special key for decrypting the data</decryption_key>
            <access_key>AccessKey</access_key>
            <secret_access_key>SecretAccessKey</secret_access_key>
            <region>us-west-2</region>
            <s3_bucket>examples3bucket</s3_bucket>
            <file_format>json</file_format>
		 	<files_directory>datasyncfile/azureblobfiles</files_directory>
         </task>
      </subscribe>
      <polling_interval>40</polling_interval>
   </agent>
</config>

The files saved in the AWS S3 bucket will be named <task_name>.<randomized_unique_identifier>.<file_format>.  Using the above configuration example, a file would be named s3_agent_subscribe.00b470b7-901c-4447-9316-023a265d632f.json and then this file would be pushed into your AWS S3 bucket.