Perspectium DataSync Agents support the replication of data from your app to an Amazon Web Services (AWS) S3 bucket, or an AWS S3 Subscriber Agent. By configuring your AWS S3 Subscriber Agent, data from your app can be replicated and then saved as either .json or .xml file(s) in your AWS S3 bucket.
Prerequisites
First, you will need to set up one of the Perspectium DataSync Agents.
You should also stop running your DataSync Agent before making any Agent configuration changes.
Procedure
To configure your DataSync Agent to run as an AWS S3 Subscriber Agent, follow these steps:
Add joda-time and aws-java-sdk libraries
Add the following .jar files to your DataSync Agent's extlib directory:
joda-time (v2.10.3)
aws-java-sdk-core (v1.11.729)
aws-java-sdk-kms (v1.11.729)
aws-java-sdk-s3 (v1.11.729)
While newer versions may work, they have not been tested and it is suggested you use the versions listed above which have been confirmed to work with this release.
Access your agent.xml configuration file
Navigate to the directory where you saved your agent.xml file when installing your DataSync Agent.
Delete database directives
Open your agent.xml file in a text editing application and delete all database directives (those starting with <database) nested within the <task> tag. Some of the more commonly found database directives include:
- <database_type>
- <database_server>
- <database_port>
- <database_user>
- <database_password>
- <database_parms>
- <database>
Update the values for <task_name> and <handler>
Locate the <task_name> and <handler> directives nested within the <task> tag and update their values as follows:
| Directive | Update value to... |
|---|---|
| <task_name> | s3_agent_subscribe |
| <handler> | com.perspectium.replicator.file.S3Subscriber |
Add AWS directives
Within the <task> tag, nest the following directives:
| Directive | Description | Required? |
|---|---|---|
| <access_key> | Access Key associated with your AWS account | Yes |
| <secret_access_key> | Secret Access Key associated with your AWS account | Yes |
| <region> | Region that your AWS S3 bucket resides in | Yes |
| <s3_bucket> | Name of your AWS S3 bucket | Yes |
| <file_format> | Format you want to save your data records in e.g., json or xml | Yes |
Save your agent.xml
Save the changes you've made to your agent.xml and close the file. An example agent.xml configuration for an AWS S3 Subscriber Agent is shown below:
<?xml version="1.0" encoding="UTF-8"?>
<config>
<agent>
<share />
<subscribe>
<task>
<task_name>s3_agent_subscribe</task_name>
<message_connection password="password" user="user">https://mesh.perspectium.net</message_connection>
<instance_connection password="password" user="user">https://myinstance.service-now.com</instance_connection>
<handler>com.perspectium.replicator.file.S3Subscriber</handler>
<decryption_key>The cow jumped over the moon</decryption_key>
<access_key>AccessKey</access_key>
<secret_access_key>SecretAccessKey</secret_access_key>
<region>us-west-2</region>
<s3_bucket>examples3bucket</s3_bucket>
<file_format>json</file_format>
</task>
</subscribe>
<polling_interval>40</polling_interval>
</agent>
</config>
NOTE: In this configuration example, your data records will be saved in your AWS S3 bucket as one file. To save each record from your app as an individual file in your AWS S3 bucket, use the following agent.xml configuration example as a guide:
<?xml version="1.0" encoding="UTF-8"?>
<config>
<agent>
<share />
<subscribe>
<task>
<task_name>s3_agent_subscribe</task_name>
<message_connection password="password" user="user">https://<customer>.perspectium.net</message_connection>
<instance_connection password="password" user="user">https://<instance>.service-now.com</instance_connection>
<handler>com.perspectium.replicator.file.S3Subscriber</handler>
<decryption_key>The cow jumped over the moon</decryption_key>
<access_key>AccessKey</access_key>
<secret_access_key>SecretAccessKey</secret_access_key>
<region>us-west-2</region>
<s3_bucket>examples3bucket</s3_bucket>
<file_format>json</file_format>
<one_record_per_file/>
</task>
</subscribe>
<polling_interval>40</polling_interval>
</agent>
</config>
Files saved in the AWS S3 bucket will be named <task_name>.<randomized_unique_identifier>.<file_format>. Using the above configuration example, a file would be named s3_agent_subscribe.00b470b7-901c-4447-9316-023a265d632f.json.
Run your AWS S3 Subscriber DataSync Agent
After configuring your agent.xml file to support your AWS S3 Subscriber Agent, start running your DataSync Agent again.
