John Alvord, IBM Corporation
jalvord@us.ibm.com
Overview
ITM 6 processing uses TCP/IP communications. In a z/OS system ITM 6 can also use System Network Architecture [SNA] communications. That is a separate topic and will not be addressed in this technote. A good starting point to understanding ITM communications technology is documented here:
ITM Port Usage and Limiting Port Usage
ITM 6 communications requires managing network interfaces. Network interfaces are identified by ip addresses. In many systems there is a Network Interface hardware device associated with ip address(es). There can also be software defined interfaces such as localhost [127.0.0.1] or those installed by virtualization software. Multiple ip addresses can be associated with a single Network Interface device.
The ITM 6 KDE component discovers the interfaces and creates a list of ip addresses that are suitable for connections. These addresses are registered into the Location Broker [think phone book]. ITM interface discovery logic ignores ip addresses like 127.0.0.1. It will pick an order or priority of registration based on the sequence seen during discovery.
Note that this only applies to ITM 6 server processes such as TEMS and WPA and KDE_Gateway. Agents use location broker information gathered at the services they connect to.
ITM 6 communications usually works very well using default processing. However in complex cases, you must control interface usage using these two environment variables:
KDEB_INTERFACELIST
KDEB_INTERFACELIST_IPV6
See Appendix 1 for adapting to the disruptive change that was temporarily introduced at ITM 623 FP1 and ITM 622 FP8 and was reverted at ITM 623 FP3.
This document ignores KDEB_INTERFACELIST_IPV6 – it works exactly the same but for IPV6 networks.
KDEB_INTERFACELIST Controls
The format of this environment variable format is a string having one or more segments separated by blanks. An individual segment looks like this: [control][ip_address]
control can be
absent meaning no operator
! meaning exclusive
– meaning subtractive
+ meaning additive
ip_address can be
absent
a name which resolves to an ip address
an asterisk which means the interface associated with hostname.
Here are specific usage cases.
KDEB_INTERFACELIST Simple Address Control (no operator)
KDEB_INTERFACELIST=192.168.1.1
After discovery this ip address will be placed first.
KDEB_INTERFACELIST=192.168.1.1 192.168.1.100
After discovery these two ip addresses will be placed first.
KDEB_INTERFACELIST Exclusive Bind (! Operator)
KDEB_INTERFACELIST=!192.168.1.1 and
In default configuration, ITM 6 communications will listen on all available interfaces for incoming work. That is called non-exclusive bind. When the exclamation mark prefixes the ip address, that single interface will be listened on and no other. That is called exclusive bind.
When exclusive bind is used, all ITM 6 processes must use exclusive bind. The usage must be coordinated. You must never mix exclusive and non-exclusive bind as the agents will overwrite each other’s connections constantly.
KDEB_INTERFACELIST=!*
In some platforms, one interface will be specially tagged as “hostname”. The above setting will listen exclusively to that interface.
It is quite normal and useful to have multiple ITM processes running with different exclusive binds. For example on Linux/Unix you can install multiple remote TEMS each using a different exclusive ip address [and different install directories]. So the requirement is that the exclusive binds must be coordinated.
KDEB_INTERFACELIST Subtraction Control (- operator)
KDEB_INTERFACELIST=-192.168.1.100
This means that a normal survey of all interfaces is performed, but the 192.168.1.100 interface is removed from the list after discovery.
KDEB_INTERFACELIST_IPV6=-
This means that a normal survey of all interfaces is performed, but all IPV6 interfaces are ignored. That can be useful when the IPV6 is not ready for production usage.
KDEB_INTERFACELIST Addition Control (+ operator)
KDEB_INTERFACELIST=+192.168.1.100
This means that a normal survey of all interfaces is performed, but the 192.168.1.100 interface is added to the survey.
Practical examples
Case 1: The system has three interfaces. The first two can be reached by agents but the third is dedicated to a backup function and should not be advertised in the
Location Broker (think phone book) nor listened on by the Tivoli process.
KDEB_INTERFACELIST=-192.168.1.100
This subtracts the ip address for the backup function
Case 2: The system has IPV6 interfaces, but there is no working IPV6 networking in the environment.
KDEB_INTERFACELIST_IPV6=-
That could be done to simplify processing and avoid publishing an interface that is not in face useful.
Case 3: The system has two interfaces which are valid for connection from agents.
Do not supply any controls unless required by other ITM processes on the system. The agents will connect to the location broker and then attempt a connection on each server interface. Whatever works, that will be the one the agent uses. There is no need to supply a priority order. For example one site had agents on two different sub-networks and each needed to connect to the TEMS via different ip addresses.
Case 4: The ITM process is an agent. That includes TEPS.
Do not supply any controls unless required by other ITM processes on the same server.
Case 5: Two remote TEMS on a single Linux/Unix system
One interesting example is on a Linux/Unix system where two different TEMSes are installed in different install directories.
This requires two separate dedicated interfaces. For each of them use the exclusive control:
KDEB_INTERFACELIST=!192.168.1.1
KDEB_INTERFACELIST=!192.168.1.100
In the same way you could set up two OS Agents, each using one interface and using a separate interface… perhaps each connecting to a different ITM hub TEMS.
Case 6: Specified Interfaces
There are cases where a specified interface must be discovered first. For agents, this will be the address the agent puts into the Node Status as the ip address and port number to contact. One good example is a cluster configuration where the virtual IP address must be registered – so the contact information remains the same no matter what the actual system the agent is running on.
This is still easily accomplished:
A) If there is a single usable interface specify nothing.
B) If there are two or more interfaces, list the specified interface first and then specify any other interface. In this case the specified interface will be discovered first and used for registration.
Case 6: Too Many Interfaces
If you see this error message
Status 1DE00046 KDE1_STC_INTERFACELIMITREACHED error
That means there are too many interfaces to continue.
Disable interface discovery by adding the following parameter to the Agent / Server configuration file:
KDEB_NOIFEXAM=1
In the same configuration file, specify the interface to be used to connect to the TEMS:
KDEB_INTERFACELIST=x.x.x.x
where x.x.x.x is the interface address which will be used.
Pre ITM control – KDCB0_HOSTNAME
A closely related control looks like this
KDCB0_HOSTNAME=xx.xx.xx.xx
This control has the identical meaning as
KDEB_INTERFACELIST=xx.xx.xx.xx
Most importantly KDCB0_HOSTNAME overrides any KDEB_INTERFACELIST setting. If you have
KDC0_HOSTNAME=xx.xx.xx.xx
KDEB_INTERFACELIST=!xx.xx.xx.xx
That is the exact equivalent to
KDEB_INTERFACELIST=xx.xx.xx.xx
Which can be extremely surprising, especially if you expect exclusive bind across many agents on some system.
You should remove KDCB0_HOSTNAME and replace with KDEB_INTERFACELIST if it is really needed.
Summary
This technote demonstrates how to configure ITM communications to avoid problems.
Sitworld: Table of Contents
Photo Note: Mud Creek Slide: 1 million tons of earth and rocks 2017/5/20
Appendix 1: KDEB_INTERFACELIST time of troubles.
At ITM 622 FP8 and ITM 623 FP1 the logic was changed in a problematic way.
KDEB_INTERFACELIST=192.168.1.1
Was interpreted to be the same as KDEB_INTERFACELIST=!192.168.1.1.
KDEB_INTERFACELIST=192.168.1.1 192.168.1.2
Was also interpreted to be the same as KDEB_INTERFACELIST=!192.168.1.1. The second interface was totally ignored ignored.
At ITM 622 FP9 and ITM 623 FP2 the logic was somewhat restored but was still a problem.
KDEB_INTERFACELIST=192.168.1.1 192.168.1.2
Was interpreted just as it is stated, a simple ordering of which interface comes first, just as before [and afterwards.
However
KDEB_INTERFACELIST=192.168.1.1
Was still interpreted to be the same as KDEB_INTERFACELIST=!192.168.1.1.
At ITM 623 FP3 all those changes were completely reverted.
This might seem to be an ancient history footnote, however the basic services component is installed along with the OS Agent and in many cases never updated. You may still need to take these limitations into account on agents or ITM 622 central services. That ITM 622 level is going End of Service on 28 April 2018 but history shows use it sometimes persist. Also ITM 623 GA/FP1/FP2 are still supported.
If you find yourself with a system at these problematic levels and you cannot upgrade – what do you do?
If you are not using KDEB_INTERFACELIST at all there is nothing to do.
If you are using exclusive binds or + or – nothing special is needed.
Managing ITM 622 FP8 and ITM 623 FP1
The easiest thing to do is switch the KDEB_INTERFACELIST to exclusive binds all the way across all agents on that system.
Managing ITM 622 FP9 and ITM 623 FP2
If you have a one element ordering statement
KDEB_INTERFACELIST=xx.xx.xx.xx
The best way it just to specify a following address
KDEB_INTERFACELIST=xx.xx.xx.xx 127.0.0.1
Which will preserve the ordering definition. For an Agent that second ip address will never be referenced and so does no harm.
We strongly suggest you keep the OS Agents upgraded. If there are 32 bit agents, you will also have to upgrade using tacmd updateFramework to get all the pieces upgraded. The best reference I have found is here:
The Wonderful World of Situations 