Troubleshooting Agent Installation and Configuration Errors

To gather the common files such as configurations and logs for troubleshooting, you will run the the createReport.bat file. Then, run the following command line in the root Perspectium directory:

# Windows Systems
bin\createReport.bat
Report successfully generated as file: ReplicatorReport.name.date.zip
 
# OS X and Linux Systems
bash bin/createReport

The tool will output a line telling you if it ran successfully and the name of the generated .zip file. The file will contain the name of your agent and the current date. 

The files included are:

• /conf/agent.xml

• /conf/config.xml

• /conf/log4j2.xml

• /conf/wrapper.conf

• /logs/* (all the logs)

• /bin/(instance schemas)

 NOTE:  The tool will package the files as they are, so if you have unencrypted sensitive data within them then you should edit the files appropriately before creating the report.

The agent.xml validation tool tests your access to the default and configured message bus connections. It will display other diagnostic information for your current agent name, version, and tasks. It will also test your connection to your database and outside instances such as ServiceNow as necessary. By default you will connect to our our monitor.perspectium.net server for monitoring data. So unless you have specified your Heartbeat and Reporting connection, you will see the attempts to connect here.

For Windows, go to the bin folder of the agent's installed directory and double click on the validateConfiguration.bat file. This will automatically run the configuration validation test.

The tool can be run at the command line of the root Perspectium directory, not bin, like so:

# Windows Systems
bin\validateConfiguration.bat
 
# OS X and Linux Systems
bash bin/validateConfiguration

The following example is a successful validation:

Configuration Validation Report: Fri Feb 02 09:13:40 PST 2018

Agent Information:
	Agent version: Argon_3270
	Agent name: 'example_agent'

Configured Tasks:
	Task: example_agent_subscribe Type: subscribe
		example_agent_subscribe instances: 4

		Attempting to connect to the Message Broker Service at: amqps://example.perspectium.net as user: myUser - SUCCESS
		Attempting to connect to the database: psp_repl at host: localhost port: 3306 as user: myUser - SUCCESS
		Attempting to fetch schema from: https://example.service-now.com- SUCCESS
	
Validation Completed, results: SUCCESS

Report has been saved in file: ConfigurationValidationReport.log

You should ensure that your firewall and or proxy allow outbound connectivity to the Perspectium Integration Mesh and your ServiceNow Instance.

Note: Running the configuration validation tool will perform connectivity tests for all elements defined within the agent.xml configuration file and will tell you if the test succeeded or failed. The following is recommended if additional trouble shooting is required.

Depending on the protocol you plan on using you should test the corresponding DNS resolved name and port number. If you plan on operating through a proxy then you should test it as well.

The connection to the destination host and port can be tested by calling telnet from the command line as necessary with the ports:

If you're running the Agent on a Windows server and need assistance installed the telnet client please reference this link: https://social.technet.microsoft.com/wiki/contents/articles/22715.how-to-enable-telnet-in-windows-server-2012.aspx

If you're running the Agent on a Linux distribution that supports yum then you should be able to install the client by issuing the following command which requires sudo privileges.

sudo yum install telnet

If your system does not leverage yum then ask your system administrator for assistance.

The following reflects a successful test to the host <your_url>.perspectium.net. The telnet command specifies the destination host's fully qualified domain name followed by the target port which in this case is 5672. The message starting with 'Connected to' states that the connection has been established which is the desired outcome.

# Good Connections (the IP address will vary per region)
telnet <your_url>.perspectium.net 5672
Trying 52.8.132.178 ...
Connected to <your_url>.perspectium.net.
Escape character is '^]'.

The next example is also successful at establishing a connection to the host <your_url>.perspectium.net which accepts connections directed to port 443 or HTTPS. 

telnet <your_url>.perspectium.net 443
Trying 52.8.203.198 ...
Connected to <your_url>.perspectium.net.
Escape character is '^]'.

The next example of a successful test is performed to ensure that the agent can connect to your ServiceNow instance which accepts HTTPS connections via port 443.

telnet <your_instance>.service-now.com 443
Trying 103.23.66.118 ...
Connected to <your_instance>.service-now.com.
Escape character is '^]'.

Now let's take a look at what failed connection attempts look like.

In the following example we attempt to connect to the host my_domaine.perspectium.net using port 443 or HTTPS. The error message could be due to a typo in the spelling of the fully qualified domain name. Perhaps the name you were assigned is my_domain but what was actually typed during the test or perhaps entered into the agent.xml configuration file was my_domaine.perspectium.net (Note the extraneous 'e' at the end of domain):

# Failed Connections (you may get a failed response or no response at all)
telnet my_domaine.perspectium.net 443
bad_domain.perspectium.net: nodename nor servname provided, or not known

Here are some additional examples or error responses that can happen when connection attempts fail:

# Failed Connections (you may get a failed response or no response at all)
telnet blocked_domain.perspectiun.net 443
telnet: connect to address XX.XX.XXX.XX: Connection refused
telnet: Unable to connect to remote host

It's also a good idea to see if you have general internet connectivity. For example, you should be able to connect to google via telnet:

telnet www.google.com 443

All my tests fail, what should I do? If all of your tests fail then it's likely that you have one or more of a few standard issues.

• Your company has a security policy in place such that outbound connections to certain destinations such as Perspectium must be configured. Discuss your issue with your network support team.

• Your company leverages an outbound proxy. A outbound proxy server is a server (a computer system or an application) that acts as an intermediary for requests from clients seeking resources from other servers, typically those in the public internet. Please contact your proxy support team, or possibly network support for assistance. 

If you see the following error with Connection refused:

ERROR AMQP - new connection attempt failed: Connection refused
ERROR SubscriberTask - Subscriber Error: Failed to get connection to URI: amqps://<your_url>.perspectium.net VHost: / User: your_instance
ERROR SubscriberTask - Can't report status - MessageBus is not set or not open

This is a result of an incorrect protocol/DNS relation. This is solved by correctly configuring the <message_connection> tag which depends on the assigned DNS given to you by Perspectium Support. Your connection will likely be either:

• amqps://<your_url>.perspectium.net

• https://<your_url>.perspectium.net

Incorrect Subscriber - Perspectium Credentials

ERROR SubscriberTask - Subscriber Error: Failed to authenticate to URI: https://<your_url>.perspectium.net VHost: / User: your_instance
ERROR SubscriberTask - Can't report status - MessageBus is not set or not open!

If your subscriber task has incorrect login credentials you cannot authenticate to our servers and you may see these error messages. Double check that your credentials are correct.

Incorrect Subscriber - ServiceNow Credentials

ERROR ServiceNowSchema - Error: Send Error: Unexpected response status: 401
WARN  SchemaDocument - com.perspectium.replicator.sql.subscriber.SchemaAccessException: Send Error: Unexpected response status: 401
WARN  SchemaDocument - retrying ... 0
ERROR ServiceNowSchema - Error: Send Error: Unexpected response status: 401
WARN  SchemaDocument - com.perspectium.replicator.sql.subscriber.SchemaAccessException: Send Error: Unexpected response status: 401
WARN  SchemaDocument - retrying ... 2
WARN  SchemaDocument - Failed to fetch schema for table: incident trying cached version
ERROR SQLTableSubscriber - Unable to establish table meta data.

If your subscriber task has incorrect credentials for the instance connection to ServiceNow you may see these errors upon pulling messages from the queue. Double check that your credentials are correct.

Incorrect Share - Perspectium Credentials

ERROR SharerTaskMessagePublisher - post error: Send Error: Send Error: Unexpected response status: 401
ERROR SharerTaskMessagePublisher - closeQueue Error: Send Error: Send Error: Unexpected response status: 401

If your share task has incorrect credentials than you may see this error upon connecting to our server.

Incorrect SQL Credentials

WARN  TaskDatabase - Access denied for user 'root'@'localhost' (using password: YES)
ERROR SQLDataSource - Error: Access denied for user 'root'@'localhost' (using password: YES)

If your SQL credentials are incorrect than you may see that your access is denied and receive the following error messages.

This is something that occurs more often during installation but can occur other times. When your agent is Subscribed it will store schemas from ServiceNow within the bin folder. When you receive a message the agent will check this schema and if this schema is corrupted, doesn't exist, is out of date, etc. the agent will check ServiceNow and update the schema. If your ServiceNow credentials are out of date/incorrect than you may see the following errors.

ERROR ServiceNowSchema - Error: Send Error: Unexpected response status: 403
WARN  SchemaDocument - com.perspectium.replicator.sql.subscriber.SchemaAccessException: Send Error: Unexpected response status: 403
WARN  SchemaDocument - retrying ... 0
...
WARN  SchemaDocument - Failed to fetch schema for table: //table_in_message// trying cached version
ERROR SubscriberTask - …….
    com.perspectium.replicator.SubscribeException: Error: SQLSubscriber processMessage error against table: //table_in_message// error: Unable to obtain or locate the schema for table: //table_in_message//

To fix this you can remove the stored schema for this table, do this by going to the bin/<your_instance>.service-now.com.schemas_directory folder and finding the tables' XML and delete it. Then double check that your ServiceNow credentials are correct and restart the agent. It should then re-acquire the correct schema from ServiceNow.

WARN  AMQP - com.perspectium.api.AMQPException: Failed close channel on URI 'amqp://<your_url>.perspectium.net' com.rabbitmq.client.AlreadyClosedException: channel is already closed due to channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'psp.out.replicator.<your_queue>' in vhost '/', class-id=60, method-id=70)

This is an issue of subscribing to a queue that cannot be found in the Perspectium Integration Mesh. Verify the queue you specified in the message connection tag is the queue you are sharing data to from your ServiceNow instance and that you have started sharing (the queue is automatically created when you first start sharing).

ERROR Encrypt - Error decrypting value [encrypted:vlOtU71Xu8N/EFQJJ5SS+o2+kRzuseHBK+]

This is generally a sign that at some point your encrypted field was edited post encryption causing it to fail to decrypt. This will also normally cascade causing some sort of authentication or 401 error. Check your if you have a Before Share Script that is modifying an encrypted field.

Encrypting sensitive password field values is supported in the config.xml and agent.xml files. The installer by default will tag any password field due for encryption with the prefix encrypt:. When the agent starts, any configuration values prefixed with encrypt: will be encrypted and the configuration file will be rewritten with the prefixes changed to encrypted: and the password encrypted.

If you change the password to any of your credentials then you will have to stop the agent, replace encrypted:password with encrypt:new_password where necessary, and restart the agent.

For example:

<share>
    <task>
        <task_name>My_Sharer</task_name>
        <handler>com.perspectium.replicator.sql.sharer.SQLSharer</handler>
        <message_connection user="user" password="encrypt:newpassword">https://<your_url>.perspectium.net</message_connection>      
 
        <database_type>mysql</database_type>
        <database_server>localhost</database_server>
        <database_port>3306</database_port>
        <database_user>root</database_user>
        <database_password>encrypt:newpassword</database_password>
 
        <database>repl_db</database>
        <table_name>incident</table_name>
        <topic>replicator</topic>
        <type>agent</type>
        <name>.bulk</name>			
    </task>
</share>

If you are trying to install the agent on Windows in a console window (such as using Putty to SSH into a different server) and get an error about the DISPLAY variable or initDisplay, this error is due to the Agent installer trying to create a GUI for installing the agent.

In this case, you will want to run the agent with the console switch:

java -jar perspectium-replicator-installer.jar -console

The following error displays the failure to create connections to both the Perspectium Integration Mesh and the database while the connection to the instance was fine. In this case, the failures were just due to bad credentials and using the wrong port to connect to my database (3305 instead of 3306).

Configuration Validation Report: Fri Feb 02 09:18:58 PST 2018

Agent Information:
	Agent version: Argon_3270
	Agent name: 'example_agent'

Configured Tasks:
	Task: example_agent_subscribe Type: subscribe
		example_agent_subscribe instances: 4

		Attempting to connect to the Message Broker Service at: https://example.perspectium.net as user: badUser - *** FAILED ***
		Verify your firewall has port 443 open for outbound traffic.
		Verify your <message_connection> entry is properly configured.
		Attempting to connect to the database: psp_repl at host: localhost port: 3305 as user: badDBUser*** FAILED *** Unable to connect to database: Cannot create PoolableConnectionFactory (Communications link failure
		The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)
		Verify your database parameters are properly configured.
		Attempting to fetch schema from: https://example.service-now.com- SUCCESS
	
Validation Completed, results: *** FAILED ***

Double check your endpoint URL and credentials to avoid this issue. If all the credentials are correct, and you are still experiencing this issue, contact support@perspectium.com.

When the agent application is ran, an additional check to validate the agent.xml file is ran. If the agent.xml file is invalid (i.e. bad file format, missing field characters, etc) then an error message is displayed on the log with a description of the error.

Configuration file validation has failed for: agent.xml. File is not a valid format.

If the application errors out in this manner, the application will gracefully stop to prevent potentially running the agent with bad/incorrect properties.

In order to correct this issue, fix the agent.xml and run the application again.

When successful, the following message will be in the log: 

Configuration file validation has passed for: agent.xml. File is a valid format.