Set up the DataSync Agent to share to AWS S3

Perspectium DataSync Agents support the replication of data from your app to an Amazon Web Services (AWS) S3 bucket, or an AWS S3 Subscriber Agent. By configuring your AWS S3 Subscriber Agent, data from your app can be replicated and then saved as either .json or .xml file(s) in your AWS S3 bucket.

Prerequisites

 First, you will need to set up the Perspectium DataSync Agent.

 You should also stop running your DataSync Agent before making any Agent configuration changes.

Procedure

To configure your DataSync Agent to run as an AWS S3 Subscriber Agent, follow these steps:

<file_prefix>$table_$d{yyyyMMdd}_</file_prefix>

Sample Configurations

You have three different ways of saving files that are then pushed to AWS S3 buckets. Below is a sample configurations for each along with details. In all examples, files will be named with a <randomized_unique_identifier> value (such as 00b470b7-901c-4447-9316-023a265d632f) to ensure there are no file naming collisions when saving to the S3 bucket.

One Record Per File

In this setup, each record is saved into its own file that is then pushed into the AWS S3 bucket.

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <agent>
      <share />
      <subscribe>
         <task>
            <task_name>s3_agent_subscribe</task_name>
            <message_connection password="password" user="user">https://<customer>.perspectium.net</message_connection>
    	    <instance_connection password="password" user="user">https://<instance>.service-now.com</instance_connection>   
            <handler>com.perspectium.replicator.file.S3Subscriber</handler>
            <decryption_key>My special key for decrypting the data</decryption_key>
            <access_key>AccessKey</access_key>
            <secret_access_key>SecretAccessKey</secret_access_key>
            <region>us-west-2</region>
            <s3_bucket>examples3bucket</s3_bucket>
            <file_format>json</file_format>
            <file_prefix>record_</file_prefix>
			<file_suffix>.json</file_suffix>
            <one_record_per_file/>
         </task>
      </subscribe>
      <polling_interval>40</polling_interval>
   </agent>
</config>

In this case, each record will be saved in its own file named <file_prefix><randomized_unique_identifier><file_suffix>.  Using the above configuration example, a file would be named record_00b470b7-901c-4447-9316-023a265d632f.xml.

Multiple Records Per File

In this setup, multiple records are saved into a file before a new file is created. This way we're creating batches of records in one file so we're not creating too many files while also not creating files that are too large as well. This option also allows you to specify multiple records of one table into their own files so each file can only contain multiple records of the same table i.e. one file only contains incident records, one file only contains ticket records, etc.

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <agent>
      <share />
      <subscribe>
         <task>
            <task_name>s3_agent_subscribe</task_name>
            <message_connection password="password" user="user">https://<customer>.perspectium.net</message_connection>
    	    <instance_connection password="password" user="user">https://<instance>.service-now.com</instance_connection>   
            <handler>com.perspectium.replicator.file.S3Subscriber</handler>
            <decryption_key>My special key for decrypting the data</decryption_key>
            <access_key>AccessKey</access_key>
            <secret_access_key>SecretAccessKey</secret_access_key>
            <region>us-west-2</region>
            <s3_bucket>examples3bucket</s3_bucket>
            <file_format>json</file_format>			
			<file_prefix>$table_$d{yyyyMMddHH_}</file_prefix>
            <file_suffix>.json</file_suffix>						
			<buffered_writes>300</buffered_writes>			
         </task>
      </subscribe>
      <polling_interval>40</polling_interval>
   </agent>
</config>

In the above example, we specify the files to be named <file_prefix><randomized_unique_identifier><file_suffix> which in this case will be based on the table and a datetime value that is today's date with the current hour, while saving up to 300 records per file using the <buffered_writes> configuration. Once the <buffered_writes> maximum has been reached, a new file at that datetime with a new randomized_unique identifier. For example, if incident records are shared, this will result in files named:

incident_2025011322_4d747761-62d0-49ae-87fc-4998e259727d.json

incident_2025011322_5af73396-5bd9-49e1-8ac5-da8fb8a12047.json

incident_2025011323_6445a8ab-f694-482c-a315-01192cc84d7a.json

One File for All Records

In this setup, when the Agent receives a batch of records from MBS, all the records, regardless of which ServiceNow table it belongs to, will be saved into one file that is then pushed to AWS S3. This process is repeated each time the Agent receives a new batch of records from MBS (which is controlled by your <polling_interval> configuration for how often the Agent polls MBS for another batch of records).

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <agent>
      <share />
      <subscribe>
         <task>
            <task_name>s3_agent_subscribe</task_name>
            <message_connection password="password" user="user">https://mesh.perspectium.net</message_connection>
    	    <instance_connection password="password" user="user">https://myinstance.service-now.com</instance_connection>   
            <handler>com.perspectium.replicator.file.S3Subscriber</handler>
            <decryption_key>My special key for decrypting the data</decryption_key>
            <access_key>AccessKey</access_key>
            <secret_access_key>SecretAccessKey</secret_access_key>
            <region>us-west-2</region>
            <s3_bucket>examples3bucket</s3_bucket>
            <file_format>json</file_format>
         </task>
      </subscribe>
      <polling_interval>40</polling_interval>
   </agent>
</config>

The files saved in the AWS S3 bucket will be named <task_name>.<randomized_unique_identifier>.<file_format>.  Using the above configuration example, a file would be named s3_agent_subscribe.00b470b7-901c-4447-9316-023a265d632f.json and then this file would be pushed into your AWS S3 bucket.