The DataSync Agent can be set up to replicate table records from a Service Now instance to local file(s) on the machine where the DataSync Agent is running. This can be useful for the case where you have a separate application that is able to read files to import data.
Records can be saved in CSV, JSON, and XML formats and each record is inserted into the file (i.e. the previous version is not updated) when the Agent processes a message.
Prerequisites
First, you will need to set up the Perspectium DataSync Agent.
You should also stop running your DataSync Agent before making any Agent configuration changes.
Procedure
To enable file replication for the DataSync Agent, follow these steps:
Navigate to the directory where you saved your agent.xml file when installing your DataSync Agent.
Within the <task> tag, nest the following directives with your choice of how you want to save your records:
- All Records in One File
- One Record per File
- Records to Multiple Files
These three options are detailed below:
Records to Multiple Files
If you want to save your records to multiple files, use the following directives:
Directive | Example | Use | Required? | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
<handler> | <handler>com.perspectium.replicator.file.XMLFileSubscriber</handler>
| The name of the file handler class | Yes | ||||||||
<buffered_writes> | <buffered_writes>250</buffered_writes> | The maximum number of records to buffer before writing to a file (to improve performance and not write to the file upon reading each record). | Yes | ||||||||
<files_directory> | For Linux: <files_directory>/Downloads/subscribefiles/</files_directory> For Windows: <files_directory>Users\Downloads\subscribefiles\</files_directory> | The directory that contains the file of the saved records
| Yes | ||||||||
<file_prefix> | <file_prefix>record</file_prefix>
You can modify yyyyMMdd with other date format of your choice. For example, hourly will need a yyyyMMddHH value. For other date format, see Date Format. <file_prefix>$table_$d{yyyyMMdd}_$i</file_prefix> | A prefix for the file name of each record. If this directive is not specified, “psp.replicator.” will be used as the prefix.
| Varies (Required if using <file_suffix>) | ||||||||
<file_suffix> | <file_suffix>.xml</file_suffix>
| A suffix for the file name of each record. If this directive is not specified, “.xml” will be used as the suffix. | Varies (Required if using <file_prefix>) | ||||||||
<file_name> | <file_name>records.csv</file_name> | The name of the file to which you want to save the records | Varies (Required if NOT using <file_prefix> and <file_suffix>) | ||||||||
<separate_files> | <separate_files>table</separate_files> | Indicates that the files will be separated by table. | Yes | ||||||||
<enable_audit_log/> | <enable_audit_log/> | A self-closing directive that will generate an audit file. The audit file has information about when the records are processed, name of the file, and number of records processed. | No | ||||||||
<translate_newline> | <translate_newline>%13</translate_newline> | This directive will replace record content newline entries with the value you input. | Varies | ||||||||
<file_max_size> | <file_max_size>50KB</file_max_size>
| Sets the maximum size for each file. Once the maximum size has been reached, a new file will be created using the current timestamp as specified in the <file_prefix> directive.
| No | ||||||||
<close_file_interval> | <close_file_interval>180</close_file_interval> | Sets the amount of time (in seconds) the Agent will wait for tmp files to be modified. When the DataSync Agent is creating the file, the file name will have a prefix “tmp” in front to indicate that the file is writing records until file_max_size or buffered_writes is reached. Once the DataSync Agent finishes creating the file due to file_max_size or buffered_writes reached, the prefix “tmp” will be removed from the file name. When the tmp file hasn’t been modified for certain amount of time, the “tmp” prefix will be removed from the filename and the file will be closed. | No | ||||||||
<translations> | Available in Krypton 8.0.6 and newer <translations> <translation> <search_value>\</search_value> <replacement_value>\\\</replacement_value> <order>7</order> </translation> <translation> <search_value>\r</search_value> <replacement_value>\ r</replacement_value> <order>3</order> </translation> <translation> <search_value>\b</search_value> <replacement_value>\ b</replacement_value> <order>2</order> </translation> <translation> <search_value>\n</search_value> <replacement_value>\ n</replacement_value> <order>1</order> </translation> <translation> <search_value>\t</search_value> <replacement_value>\ t</replacement_value> <order>4</order> </translation> <translation> <search_value>\"</search_value> <replacement_value>\ "</replacement_value> <order>5</order> </translation> <translation> <search_value>\f</search_value> <replacement_value>\ f</replacement_value> <order>6</order> </translation> </translations> For example, to translate the value \n into \ n (with a space between \ and n) and have this as the first translation to be processed, you would specify: <translation> <search_value>\n</search_value> <replacement_value>\ n</replacement_value> <order>1</order> </translation> | Allows you to specify how values in each record should be "translated" for saving into the file. This is useful for when you want to do your own escaping of the content before its saved into the file. The translation is done on the entire content of the record (i.e. every value in every field of the record). For example, you can use this feature if you want to escape newlines (\n) so they save into the file as \ n (with a space in between) so the JSON can be read properly by a different system. To configure, under <translations> you specify a <translation> for each translation you want to do. Each <translation> should have the following configurations: <search_value> - The value in the record that should be searched for and replaced when the record is saved in the file. <replacement_value> - The value that should be saved as the replacement when the record is saved in the file. <order> - The order to process the translations. This value is an integer (whole number). The <translation> record with the lowest order number will be processed first (i.e. if you have translations with the orders 1, 2 and 3, 1 will be processed first followed by 2, 3, etc.).
| No |
Save the changes you've made to your agent.xml and close the file.
An example agent.xml configuration for saving all records to a single file is shown below:
<?xml version="1.0" encoding="ISO-8859-1" standalone="no"?> <config> <agent> <max_reads_per_connect>10</max_reads_per_connect> <polling_interval>20</polling_interval> <subscribe> <task> <task_name>file_subscribe</task_name> <message_connection password="password" user="user">https://<customer>.perspectium.net</message_connection> <instance_connection password="password" user="user">https://<instance>.service-now.com</instance_connection> <decryption_key>The cow jumped over the moon</decryption_key> <handler>com.perspectium.replicator.file.XMLFileSubscriber</handler> <file_name>records.xml</file_name> <files_directory>/Users/user/Downloads</files_directory> <exclude_xml_header/> <buffered_writes>250</buffered_writes> </task> </subscribe> </agent> </config>
An example agent.xml configuration for saving one record per file is shown below:
<?xml version="1.0" encoding="ISO-8859-1" standalone="no"?> <config> <agent> <max_reads_per_connect>10</max_reads_per_connect> <polling_interval>20</polling_interval> <subscribe> <task> <task_name>file_subscribe</task_name> <message_connection password="password" user="user">https://<customer>.perspectium.net</message_connection> <instance_connection password="password" user="user">https://<instance>.service-now.com</instance_connection> <decryption_key>The cow jumped over the moon</decryption_key> <handler>com.perspectium.replicator.file.XMLFileSubscriber</handler> <one_record_per_file/> <files_directory>/tmp</files_directory> <file_prefix>records</file_prefix> <file_suffix>.xml</file_suffix> </task> </subscribe> </agent> </config>
An example agent.xml configuration for saving records to multiple files is shown below:
<?xml version="1.0" encoding="ISO-8859-1" ?> <config> <agent> <!-- the following subscribe fragment defines subscribing class --> <!-- and its arguments --> <subscribe> <task> <task_name>test_file_subscriber</task_name> <message_connection password="password_here" user="admin" queue="psp.in.meshlet.example">https://<customer>.perspectium.net</message_connection> <instance_connection password="Adminadmin1" user="admin">https://<instance>.service-now.com</instance_connection> <max_reads_per_connect>1</max_reads_per_connect> <polling_interval>3</polling_interval> <decryption_key>Example_decryption_key_here</decryption_key> <handler>com.perspectium.replicator.file.JSONFileSubscriber</handler> <buffered_writes>10</buffered_writes> <files_directory>/Users/You/Downloads/Example</files_directory> <file_prefix>$table_$d{yyyyMMdd}_$i</file_prefix> <file_suffix>.json</file_suffix> <file_max_size>50KB</file_max_size> <translate_newline>%13</translate_newline> <separate_files>table</separate_files> <enable_audit_log/> <close_file_interval>180</close_file_interval> </task> </subscribe> </agent> </config>
After configuring your agent.xml file to enable file replication, start running your DataSync Agent again.