Troubleshooting Agent Installation and Configuration Errors

You should ensure that your firewall and or proxy allow outbound connectivity to the Perspectium Integration Mesh and your ServiceNow Instance.

Note: Running the configuration validation tool will perform connectivity tests for all elements defined within the agent.xml configuration file and will tell you if the test succeeded or failed. The following is recommended if additional trouble shooting is required.

Depending on the protocol you plan on using you should test the corresponding DNS resolved name and port number. If you plan on operating through a proxy then you should test it as well.

The connection to the destination host and port can be tested by calling telnet from the command line as necessary with the ports:

If you're running the Agent on a Windows server and need assistance installed the telnet client please reference this link: https://social.technet.microsoft.com/wiki/contents/articles/22715.how-to-enable-telnet-in-windows-server-2012.aspx

If you're running the Agent on a Linux distribution that supports yum then you should be able to install the client by issuing the following command which requires sudo privileges.

sudo yum install telnet

If your system does not leverage yum then ask your system administrator for assistance.

The following reflects a successful test to the host <your_url>.perspectium.net. The telnet command specifies the destination host's fully qualified domain name followed by the target port which in this case is 5672. The message starting with 'Connected to' states that the connection has been established which is the desired outcome.

# Good Connections (the IP address will vary per region)
telnet <your_url>.perspectium.net 5672
Trying 52.8.132.178 ...
Connected to <your_url>.perspectium.net.
Escape character is '^]'.

The next example is also successful at establishing a connection to the host <your_url>.perspectium.net which accepts connections directed to port 443 or HTTPS. 

telnet <your_url>.perspectium.net 443
Trying 52.8.203.198 ...
Connected to <your_url>.perspectium.net.
Escape character is '^]'.

The next example of a successful test is performed to ensure that the agent can connect to your ServiceNow instance which accepts HTTPS connections via port 443.

telnet <your_instance>.service-now.com 443
Trying 103.23.66.118 ...
Connected to <your_instance>.service-now.com.
Escape character is '^]'.

Now let's take a look at what failed connection attempts look like.

In the following example we attempt to connect to the host my_domaine.perspectium.net using port 443 or HTTPS. The error message could be due to a typo in the spelling of the fully qualified domain name. Perhaps the name you were assigned is my_domain but what was actually typed during the test or perhaps entered into the agent.xml configuration file was my_domaine.perspectium.net (Note the extraneous 'e' at the end of domain):

# Failed Connections (you may get a failed response or no response at all)
telnet my_domaine.perspectium.net 443
bad_domain.perspectium.net: nodename nor servname provided, or not known

Here are some additional examples or error responses that can happen when connection attempts fail:

# Failed Connections (you may get a failed response or no response at all)
telnet blocked_domain.perspectiun.net 443
telnet: connect to address XX.XX.XXX.XX: Connection refused
telnet: Unable to connect to remote host

It's also a good idea to see if you have general internet connectivity. For example, you should be able to connect to google via telnet:

telnet www.google.com 443

All my tests fail, what should I do? If all of your tests fail then it's likely that you have one or more of a few standard issues.

• Your company has a security policy in place such that outbound connections to certain destinations such as Perspectium must be configured. Discuss your issue with your network support team.

• Your company leverages an outbound proxy. A outbound proxy server is a server (a computer system or an application) that acts as an intermediary for requests from clients seeking resources from other servers, typically those in the public internet. Please contact your proxy support team, or possibly network support for assistance. 

If you see the following error with Connection refused:

ERROR AMQP - new connection attempt failed: Connection refused
ERROR SubscriberTask - Subscriber Error: Failed to get connection to URI: amqps://<your_url>.perspectium.net VHost: / User: your_instance
ERROR SubscriberTask - Can't report status - MessageBus is not set or not open

This is a result of an incorrect protocol/DNS relation. This is solved by correctly configuring the <message_connection> tag which depends on the assigned DNS given to you by Perspectium Support. Your connection will likely be either:

• amqps://<your_url>.perspectium.net

• https://<your_url>.perspectium.net

Incorrect Subscriber - Perspectium Credentials

ERROR SubscriberTask - Subscriber Error: Failed to authenticate to URI: https://<your_url>.perspectium.net VHost: / User: your_instance
ERROR SubscriberTask - Can't report status - MessageBus is not set or not open!

If your subscriber task has incorrect login credentials you cannot authenticate to our servers and you may see these error messages. Double check that your credentials are correct.

Incorrect Subscriber - ServiceNow Credentials

ERROR ServiceNowSchema - Error: Send Error: Unexpected response status: 401
WARN  SchemaDocument - com.perspectium.replicator.sql.subscriber.SchemaAccessException: Send Error: Unexpected response status: 401
WARN  SchemaDocument - retrying ... 0
ERROR ServiceNowSchema - Error: Send Error: Unexpected response status: 401
WARN  SchemaDocument - com.perspectium.replicator.sql.subscriber.SchemaAccessException: Send Error: Unexpected response status: 401
WARN  SchemaDocument - retrying ... 2
WARN  SchemaDocument - Failed to fetch schema for table: incident trying cached version
ERROR SQLTableSubscriber - Unable to establish table meta data.

If your subscriber task has incorrect credentials for the instance connection to ServiceNow you may see these errors upon pulling messages from the queue. Double check that your credentials are correct.

Incorrect Share - Perspectium Credentials

ERROR SharerTaskMessagePublisher - post error: Send Error: Send Error: Unexpected response status: 401
ERROR SharerTaskMessagePublisher - closeQueue Error: Send Error: Send Error: Unexpected response status: 401

If your share task has incorrect credentials than you may see this error upon connecting to our server.

Incorrect SQL Credentials

WARN  TaskDatabase - Access denied for user 'root'@'localhost' (using password: YES)
ERROR SQLDataSource - Error: Access denied for user 'root'@'localhost' (using password: YES)

If your SQL credentials are incorrect than you may see that your access is denied and receive the following error messages.

This is something that occurs more often during installation but can occur other times. When your agent is Subscribed it will store schemas from ServiceNow within the bin folder. When you receive a message the agent will check this schema and if this schema is corrupted, doesn't exist, is out of date, etc. the agent will check ServiceNow and update the schema. If your ServiceNow credentials are out of date/incorrect than you may see the following errors.

ERROR ServiceNowSchema - Error: Send Error: Unexpected response status: 403
WARN  SchemaDocument - com.perspectium.replicator.sql.subscriber.SchemaAccessException: Send Error: Unexpected response status: 403
WARN  SchemaDocument - retrying ... 0
...
WARN  SchemaDocument - Failed to fetch schema for table: //table_in_message// trying cached version
ERROR SubscriberTask - …….
    com.perspectium.replicator.SubscribeException: Error: SQLSubscriber processMessage error against table: //table_in_message// error: Unable to obtain or locate the schema for table: //table_in_message//

To fix this you can remove the stored schema for this table, do this by going to the bin/<your_instance>.service-now.com.schemas_directory folder and finding the tables' XML and delete it. Then double check that your ServiceNow credentials are correct and restart the agent. It should then re-acquire the correct schema from ServiceNow.

WARN  AMQP - com.perspectium.api.AMQPException: Failed close channel on URI 'amqp://<your_url>.perspectium.net' com.rabbitmq.client.AlreadyClosedException: channel is already closed due to channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'psp.out.replicator.<your_queue>' in vhost '/', class-id=60, method-id=70)

This is an issue of subscribing to a queue that cannot be found in the Perspectium Integration Mesh. Verify the queue you specified in the message connection tag is the queue you are sharing data to from your ServiceNow instance and that you have started sharing (the queue is automatically created when you first start sharing).

ERROR Encrypt - Error decrypting value [encrypted:vlOtU71Xu8N/EFQJJ5SS+o2+kRzuseHBK+]

This is generally a sign that at some point your encrypted field was edited post encryption causing it to fail to decrypt. This will also normally cascade causing some sort of authentication or 401 error. Check your if you have a Before Share Script that is modifying an encrypted field.

Encrypting sensitive password field values is supported in the config.xml and agent.xml files. The installer by default will tag any password field due for encryption with the prefix encrypt:. When the agent starts, any configuration values prefixed with encrypt: will be encrypted and the configuration file will be rewritten with the prefixes changed to encrypted: and the password encrypted.

If you change the password to any of your credentials then you will have to stop the agent, replace encrypted:password with encrypt:new_password where necessary, and restart the agent.

For example:

<share>
    <task>
        <task_name>My_Sharer</task_name>
        <handler>com.perspectium.replicator.sql.sharer.SQLSharer</handler>
        <message_connection user="user" password="encrypt:newpassword">https://<your_url>.perspectium.net</message_connection>      
 
        <database_type>mysql</database_type>
        <database_server>localhost</database_server>
        <database_port>3306</database_port>
        <database_user>root</database_user>
        <database_password>encrypt:newpassword</database_password>
 
        <database>repl_db</database>
        <table_name>incident</table_name>
        <topic>replicator</topic>
        <type>agent</type>
        <name>.bulk</name>			
    </task>
</share>

If you are trying to install the agent on Windows in a console window (such as using Putty to SSH into a different server) and get an error about the DISPLAY variable or initDisplay, this error is due to the Agent installer trying to create a GUI for installing the agent.

In this case, you will want to run the agent with the console switch:

java -jar perspectium-replicator-installer.jar -console

The following error displays the failure to create connections to both the Perspectium Integration Mesh and the database while the connection to the instance was fine. In this case, the failures were just due to bad credentials and using the wrong port to connect to my database (3305 instead of 3306).

Configuration Validation Report: Fri Feb 02 09:18:58 PST 2018

Agent Information:
	Agent version: Argon_3270
	Agent name: 'example_agent'

Configured Tasks:
	Task: example_agent_subscribe Type: subscribe
		example_agent_subscribe instances: 4

		Attempting to connect to the Message Broker Service at: https://example.perspectium.net as user: badUser - *** FAILED ***
		Verify your firewall has port 443 open for outbound traffic.
		Verify your <message_connection> entry is properly configured.
		Attempting to connect to the database: psp_repl at host: localhost port: 3305 as user: badDBUser*** FAILED *** Unable to connect to database: Cannot create PoolableConnectionFactory (Communications link failure
		The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)
		Verify your database parameters are properly configured.
		Attempting to fetch schema from: https://example.service-now.com- SUCCESS
	
Validation Completed, results: *** FAILED ***

Double check your endpoint URL and credentials to avoid this issue. If all the credentials are correct, and you are still experiencing this issue, contact support@perspectium.com.

If you need to get a stack trace and/or heap dump for issues such as an unresponsive Agent or if the Agent is consuming too much memory, you will need the following commands: 

Heap Dump: 

jmap -dump:format=b,file=<filepath to write dump to>/$PID.hprof $PID 

 NOTE: $PID is the java.exe process taking up much of the memory

For example: 

jmap -dump:file=C:\PerspectiumLogs\12345.hprof 12345

For more information about memory map, click here.

Stack Trace:

jstack -l $PID > jstack_$PID.out

For example: 

jstack -l 12345 > jstack_12345.out

For more information about stack trace, click here.

When the Agent is ran, an additional check to validate the agent.xml file is ran. If the agent.xml file is invalid (i.e. bad file format, missing field characters, etc) then an error message is displayed on the log with a description of the error.

Configuration file validation has failed for: agent.xml. File is not a valid format.

If the application errors out in this manner, the application will gracefully stop to prevent potentially running the agent with bad/incorrect properties.

In order to correct this issue, fix the agent.xml and run the application again.

When successful, the following message will be in the log: 

Configuration file validation has passed for: agent.xml. File is a valid format.

When you perform a Java upgrade on your machine, this can cause temporary issues with the DataSync Agent. The main issue is the path that the DataSync Agent uses for Java is no longer valid.  When the DataSync Agent runs, it is actually running two processes. One for the Java Service Wrapper that allows the Agent to run as a service and another for the actual Agent Java application itself.  

When the Agent runs these processes, it will use the system level "java" command to execute this.  If the path of this is no longer valid, then the command may not work properly and the Agent will run into issues and not start properly. 

The following are suggested troubleshooting techniques for the issues upon upgrading the Java version in your Windows machine:

Validate that Java is installed and callable

java -version

C:\Users\example>java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181_b12)
Java Hotspot(TM) 64-Bit Server VM (build 25.181-b12, mixed mode)

Validate that the installed version matches the path 

where java

C:\Users\example>where java
C:\ProgramData\Oracle\Java\javapath\java.exe
C:\Program Files\Java\jdk1.8.0_181\java.exe

Validate that the Environment Variables show the right PATH value

Restart the system 

If all of the above suggestions are validated and no changes are needed, reboot the environment. Sometimes during the Java upgrades, the environment requires a restart.

Force a specific path

As a temporary measure, follow these steps in order to force the Agent to use a specific path for Java rather than the system path for Java:

 WARNING: Since you are forcing a Java path, you may run into errors on future Java upgrades.

uninstallService.bat

C:\Program Files\Java\jre1.8.0_181\bin\java.exe

wrapper.conf=C:\\Program Files\\Java\\jre1.8.0_181\\bin\\java.exe

installService.bat

C:\Program Files\Java\jre1.8.0_181\bin\java.exe" -classpath ...........