Perspectium DataSync Agents support the replication of data from your app to an Amazon Web Services (AWS) S3 bucket, or an AWS S3 Subscriber Agent. By configuring your AWS S3 Subscriber Agent, data from your app can be replicated and then saved as either .json or .xml file(s) in your AWS S3 bucket.


Prerequisites


(warning) First, you will need to set up one of the Perspectium DataSync Agents.

(warning) You should also stop running your DataSync Agent before making any Agent configuration changes.

Procedure

To configure your DataSync Agent to run as an AWS S3 Subscriber Agent, follow these steps:


Add joda-time and aws-java-sdk libraries

Add the following .jar files to your DataSync Agent's extlib directory:

While newer versions may work, they have not been tested and it is suggested you use the versions listed above which have been confirmed to work with this release. 

Access your agent.xml configuration file

Navigate to the directory where you saved your agent.xml file when installing your DataSync Agent.

Delete database directives

Open your agent.xml file in a text editing application and delete the following directives nested within the <task> tag: 

  • <database_type>
  • <database_server>
  • <database_port>
  • <database_user>
  • <database_password>
  • <database_parms>
  • <database_column_max_size>
  • <database>

Update the values for <task_name> and <handler>

Locate the <task_name> and <handler> directives nested within the <task> tag and update their values as follows:

DirectiveUpdate value to...
<task_name>s3_agent_subscribe
<handler>com.perspectium.replicator.file.S3Subscriber

Add AWS directives

Within the <task> tag, nest the following directives:

DirectiveDescriptionRequired?
<access_key>Access Key associated with your AWS accountYes
<secret_access_key>Secret Access Key associated with your AWS accountYes
<region>Region that your AWS S3 bucket resides inYes
<s3_bucket>Name of your AWS S3 bucketYes
<file_format>

Format you want to save your data records in

e.g., json or xml

Yes

Save your agent.xml

Save the changes you've made to your agent.xml and close the file. An example agent.xml configuration for an AWS S3 Subscriber Agent is shown below:

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <agent>
      <share />
      <subscribe>
         <task>
            <task_name>s3_agent_subscribe</task_name>
            <message_connection password="password" user="user">https://mesh.perspectium.net</message_connection>
    	    <instance_connection password="password" user="user">https://myinstance.service-now.com</instance_connection>   
            <handler>com.perspectium.replicator.file.S3Subscriber</handler>
            <decryption_key>The cow jumped over the moon</decryption_key>
            <access_key>AccessKey</access_key>
            <secret_access_key>SecretAccessKey</secret_access_key>
            <region>us-west-2</region>
            <s3_bucket>examples3bucket</s3_bucket>
            <file_format>json</file_format>
         </task>
      </subscribe>
      <polling_interval>40</polling_interval>
   </agent>
</config>

(info) NOTE: In this configuration example, your data records will be saved in your AWS S3 bucket as one file. To save each record from your app as an individual file in your AWS S3 bucket, use the following agent.xml configuration example as a guide:

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <agent>
      <share />
      <subscribe>
         <task>
            <task_name>s3_agent_subscribe</task_name>
            <message_connection password="password" user="user">https://<customer>.perspectium.net</message_connection>
    	    <instance_connection password="password" user="user">https://<instance>.service-now.com</instance_connection>   
            <handler>com.perspectium.replicator.file.S3Subscriber</handler>
            <decryption_key>The cow jumped over the moon</decryption_key>
            <access_key>AccessKey</access_key>
            <secret_access_key>SecretAccessKey</secret_access_key>
            <region>us-west-2</region>
            <s3_bucket>examples3bucket</s3_bucket>
            <file_format>json</file_format>
            <one_record_per_file/>
         </task>
      </subscribe>
      <polling_interval>40</polling_interval>
   </agent>
</config>

Files saved in the AWS S3 bucket will be named <task_name>.<randomized_unique_identifier>.<file_format>.  Using the above configuration example, a file would be named s3_agent_subscribe.00b470b7-901c-4447-9316-023a265d632f.json.

Run your AWS S3 Subscriber DataSync Agent

After configuring your agent.xml file to support your AWS S3 Subscriber Agent, start running your DataSync Agent again.