John Alvord, IBM Corporation
jalvord@us.ibm.com
Introduction
Recently I have encountered customers doing a lot of simultaneous SOAP activity. The ITM default communications default setup works very well indeed. I estimate a 98% success rate out of the box. However the default only permits 6 simultaneous SOAP connections and each SOAP connection will time out after 300 seconds. Exceeding these limits caused processing errors such as tacmd login failures.
In March 2010, I documented how and why to update the ITM communications string using Protocol Modifiers. I am re-publishing that here with new information to change the SOAP related configurations. I have also made the document more self contained. Here are two newly documented protocol modifiers.
SOAP Concurrency
HTTP_CONCURRENCY defaults to 8 connections. 2 connections are dedicated to the internal web server listening ports and that leaves 6 that are available for SOAP request. Many tacmd functions uses SOAP. However anyone can write a SOAP process and start processing ITM Data. In addition there are commercial products using SOAP. When the number of SOAP ports needed exceeds the available ports, everything slows down and even tacmd login can time out.
A warning note: I have worked with several cases where customers or some product attempted to get too much data for the hub TEMS to handle. One in particular would have required processing 20 gigabytes an hour of data through the hub TEMS and that just killed normal ITM processing. Please avoid that path by estimating and measuring usage and making sure the TEMS can handle the load.
To increase the number of SOAP to 30 ports use the following modifier: HTTP_CONCURRENCY:32
See below as how to implement this change.
SOAP Maximum Age
HTTP_MAXAGE defaults to 300 seconds. That means a SOAP request will time out after that amount of idle time. If you are doing anything that will take longer, you will lose control of the process.
To increase the time a SOAP request can remain inactive to 15 minutes use the following modifier: HTTP_MAXAGE:900
ITM Protocol Usage and Protocol Modifiers
Important
Much of the following material was incorporated into the ITM 623 Installation and Setup Guide here. There is additional material here which can be useful in specific circumstances.
Tivoli Monitoring protocol usage and protocol modifiers
Overview
ITM basic services communications are defined by the KDE_TRANSPORT environmental variable, and after this paragraph that will be the only variable mentioned. At an many stages of ITM development, KDC_FAMILIES was used, but KDE_TRANSPORT is intended to be the successor. At this moment, the two variables are processed identically. However many agent installers are only aware of the earlier variable. In every case examine the agent diagnostics and adopt whatever is being used.
ITM uses other communication protocols such as 1) Portal client to Portal server uses CORBA IIOP after startup, 2) TEPS/WPA/SPA can use ODBC or JDBC to communicate with the warehouse database, and 3) several other cases. Most of ITM to ITM communications uses TCP/IP or SNA [z/OS only] protocols. This document ignores the SNA case.
Modifiers to the protocols are of the form ‘attribute:value’. If they occur first, then they are global in effect if meaningful. If they occur after a protocol and before the next protocol, then the effect is only on that protocol. The protocol names and modifiers are case insensitive although they are presented in upper-case here.
In the Appendix there are references on how to implement the needed changes depending on platform.
KDE_TRANSPORT Structure
KDE_TRANSPORT is a string which lists protocols and modifiers. All protocols are assumed present. A protocol will be activated only if an interface is available for use.
The USE modifier activates a specific protocol if Y is specified, otherwise it deactivates it. The scanning starts with an implicit global USE:Y, meaning that all protocols are assumed to be activated by default. For example:
IP.PIPE PORT:1918 USE:Y IP.SPIPE USE:N
means that IP.PIPE will be available but IP.SPIPE will not. In addition all the unnamed protocols listed later are activated such as IPV6.
If a modifier is specified before any protocol, it generally applies globally. For example, a pool setting first will apply to all protocols.
KDE_TRANSPORT Transmission Control Protocol
TCP is a connection oriented protocol. Connection is made via a port number, which can be 0-65535. The connection continues until the application(s) tear it down.
Here are the protocol names:
IP.PIPE – tcp
IP.SPIPE – secure tcp
IP6.PIPE – ipv6 tcp
IP6.SPIPE – ipv6 secure tcp
The secure protocols are implemented with the Global Secure Toolkit (GSKIT) component.
Here are the tcp protocol modifiers:
PORT
PORT defaults to 1918 for tcp and 3660 for secure tcp. These numbers are registered with the Internet Assigned Numbers Authority, www.iana.org/assignments/port-numbers.
PORT defines the base port number. For a TEMS, the base number is used as the listening port, and is the port that Agents connect to. For agents, ITM default processing opens a listening port at number base+N*4096, where N is 1 to 15. If one is already in use, it tries the next higher iteration. The base port is reserved in case a TEMS gets started later.
For an Agent, the listening port is used for two main purposes: 1) TEMS requesting real-time data from the agent and 2) Agent receiving notifications from the TEMS, such as awareness of a WPA [Warehouse Proxy Agent] re-registration at a new IP address or port number.
Example: IP.PIPE PORT:1918 USE:Y
Note that in this mode of definition, there are a maximum of 15 agents on a server. If this is an issue see the EPHEMERAL modifier section further below.
SKIP and COUNT
SKIP and COUNT modifiers are used to control the port search algorithm. Default search is for baseport+N*4096 with N from 1 to 15. The SKIP modifier forces it to start with N equal to the SKIP value in the above calculation. The COUNT modifier means to only try that number of times, not all the way to 15.
The following is often used with Warehouse Proxy Agent:
IP.PIPE PORT:1918 SKIP:15 COUNT:1 USE:Y
It means that the only port checked will be 1918+15*4096 or 63358. This is useful because it means that Warehouse Proxy Agent (WPA) will have a fixed address even though a TEMS and other agents may be starting up. Having a fixed port number is vital if firewall rules are used. At some future ITM maintenance level, the WPA port will likely be fixed to a high port number and configuration will not be required,
EPHEMERAL
This modifier has three different values.
A value of Y means that the connection to the TEMS listening port will be used for all communications. The agent will not need a listening port. The advantage can be important in cases where firewall rules are used, since fewer ports are used. It is also a way to avoid the 15 agent limitation. The negative side is that if historical data collection is being done, then historical data must either be stored at the TEMS, or there must be a WPA process running on the same server as the TEMS the agent reports to if the historical data is being stored at the Agent. Having a WPA on the TEMS the agent connects to is considered a best practice in large environments so that is often already the case.
A value of OUTBOUND means the same thing as Y.
A value of INBOUND can be used at a TEMS and it means that every agent connecting to it is configured to ephemeral mode.
POOL
ITM processes use ports for communication within the server, never seen on the outside network. The temporary ports that are used can be controlled with this option.
Examples:
IP.PIPE POOL:50900-51923
IP.UDP POOL:01000-01023 POOL:01024-02048
Note that each pool modifier can specify a maximum of 1024, but you can have multiple such specifications.
Any pool range must fit within the local system defined ranges. On AIX the “no -a” will display current ranges. See this URL for a general overview:
http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html
The POOL control does not limit localhost port usage. See the next topic an environment variable to handle that,
When an ITM process loses contact with the target and then reconnects, a new listening port is created. That follows internet standards. The old port is retained by TCP itself to handle any late arriving, duplicate or fragmented packets. Default timing is to hold the older port for 120 seconds. The new listening port is from the ephemeral pool and is also controlled by the POOL protocol modifier.
KDE_LOOPBACK_POOL
This is not a protocol modifier exactly but it serves a parallel purpose. ITM makes use of a number of loopback [127.0.0.1] socket connections for intra-ITM process communication and awareness. By default the ports are supplied by TCP with no constraints. When this environment variable is present, the loopback ports are constrained just like the POOL protocol modifier. Here is an example usage:
KDE_LOOPBACK_POOL=50900-51923
The number of ports needed depends on a lot of aspects such has how many ITM processes are running, whether internal web server is running and other things. All ITM processes on that system should be defined identically. The function was introduced in ITM 630 FP6 and here is fuller documentation:
IV78573: TEBS: PORT POOLS FOR LOOPBACK DEVICE SOCKET ALLOCATIONS
KDE_TRANSPORT User Datagram protocol
For ITM, this protocol is not usually best overall. The advantage is a somewhat lower storage requirement. The negatives are 1) less reliability since applications are responsible for error recovery, 2) higher CPU resources are required, and 3) it cannot be used where firewall rules are in place.
Here are the protocol names:
IP.UDP – User Datagram Protocol
IP – Synonym for IP.UDP
IP6.UDP – IP V6 version of IP.UDP
The POOL and PORT modifiers can be used for this case The other modifiers are connection oriented.
KDE_TRANSPORT Hypertext Transfer Protocol
Each ITM process has an internal web server. These define what protocols are used to access the internal web server. The web server gives access to the ITM service console, the portal client, the SOAP server tryout page, and [starting with ITM 6.2.2] the Agent Service Index pages.
Here are the protocol names:
ip.tcp.http – http communications
ip.ssl.https – secure http communications
ip6.tcp.http – ipv6 http communications
ip6.ssl.https – ipv6 secure http communications
Here are the protocol modifiers
HTTP_CONCURRENCY
Defaults to 8 listen pipes. When SOAP is in heavy use this should be increased to a large number such as 16 or 32.
HTTP_MAXAGE
Defaults to 300 seconds. If a SOAP request is expected to be idle for a considerable period, increase this value accordingly.
HTTP_SERVER
Defaults to Y. If set to N then the internal web server is not started.
HTTP_CONSOLE
Defaults to Y. If set to N then the ITM service console is not started.
HTTP
Defaults to 1920. If set to 0, port number is set to a temporary or ephemeral number. That number will be will be controlled by the ip.tcp POOL number if present.
HTTPS
Defaults to 3661 . If set to 0, port number is set to a temporary or ephemeral number. That number will be will be controlled by the ip.ssl POOL number if present.
POOL
The TCP socket protocol also uses temporary ports and the usage is controlled by separate pool control settings. These are not protocols, but are required for the POOL setting:
ip.tcp – pool control for the ip.tcp.http protocol
ip.ssl – pool control for the ip.ssl.https protocol
ip6.tcp – pool control for the ip6.tcp.http protocol
ip6.ssl – pool control for the ip6.ssl.https protocol
It would be easier to have leading pool setting(s) which would apply to all protocols.
Interactions with other environment variables
KDEB_INTERFACELIST and KDEB_INTERFACELIST_IPV6
The dash “-“ option is used alone, these environment variables do not scan any of the related interfaces. In that case a protocol might go unused even though it is specified in KDE_TRANSPORT. You could also eliminate all the interfaces by name to achieve the same result.
Summary
This document explains how to understand and change the KDE_TRANSPORT environment variable to change ITM communication defaults.
Sitworld: Table of Contents
Photo Note: Three California Condors Looking For Trouble
Appendix: A Guide to Changing the KDE_TRANSPORT settings
Most of this document is a reference guide to constructing a KDE_TRANSPORT [or KDC_FAMILIES]. In practice what you do is highly platform dependent and KDC_FAMILIES is often what is needed.
If you decide to change KDE_TRANSPORT, make sure to change the settings on all ITM tasks when appropriate. For example, if you decide to turn off the ITM internal web server using the HTTP_SERVER:N modifier, you must make the same change for all ITM tasks. The task actually running the internal web server can switch depending on which one is up first and on ITM task recycles, so all ITM processes must have the same setting.
That is not true for agent specific operations such as using SKIP/COUNT to force a specific port number to be used by an ITM process.
The following change methods are focused on ITM 623 and earlier. If you need to work with ITM 622 and earlier, here are two references which can be followed.
Updating Linux/Unix agent KDC_FAMILIES configuration
Mass Configuration Change for Windows/Linux/Unix agents
Linux/Unix ITM process
Determine what the communications string is now. Review the diagnostic log [example: ibtems_lz_klzagent_53d40bd3-01.log] and search for KDE_TRANSPORT
… KDE_TRANSPORT=KDC_FAMILIES=”ip.spipe port:3660 ip use:n ip.pipe use:n sna use:n HTTP:1920“
The communications string is
ip.spipe port:3660 ip use:n ip.pipe use:n sna use:n HTTP:1920
In the logs directory you will also find lz.env and reviewing it you should see something like this
KDC_FAMILIES=ip.spipe port:3660 ip use:n ip.pipe use:n sna use:n HTTP:1920
Now you can see that the KDC_FAMILIES is being used in this case,
Next construct an lz.environment file in <installdir>/config. This file will have one line [to start with] and must have the same attributes/owner/group as the lz.ini file. Use the chmod/chown/chgrp commands to make that true. The contents of that line will be the same as the lz.env line with your added protocol modifiers. For example
KDC_FAMILIES=EPHEMERAL:Y ip.spipe port:3660 ip use:n ip.pipe use:n sna use:n HTTP:1920
When the ITM process is recycled, the new control will override what is there.
Note 1: This override will persist even if you do a manual reconfiguration of the agent… for example switching to ip.pipe. Therefore you must remember this permanent override and maintain it into the future.
Note 2: If a global change is needed, like turning off the internal web server
KDC_FAMILIES=HTTP_SERVER:N ip.spipe port:3660 ip use:n ip.pipe use:n sna use:n HTTP:1920
This must be repeated for every ITM process running on the system with separate environment files.
Note 3: The change for a TEMS is almost the same. There will be a leading export because that is required for a TEMS. It will be seen in the ms.env file. TEMS [and TEPS] require the internal web server and so cannot be disabled without crippling the product.
Windows Agents
The supported way to update KDC_FAMILIES uses the MTEMS graphical
interface. From that interface:
1) right-click on task line
2) Select Advanced
3) Select Edit Variables…
4) click on listbox arrow and find KDC_FAMILIES
5) click on Edit
6) You should see a value of @Protocol@. This value is kept in the kin.ini file. In a simple case, you would just insert a modifier before that text and hit OK. In a more complex case you would determine what the full value is and replace it all.
The Windows registry involved is actually
32-bit environment
HKEY_LOCAL_MACHINE\SOFTWARE\Candle\KNT\Ver610\Primary\Environment
64-bit environment
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Candle\KNT\Ver610\Primary\Environment
I worked with one customer who reported that altering the registry was easiest for him. Changing it for one agent changed usages for all agents on that system.
There is an technote example of doing the registry change via a .reg file here.
i/5
The environment variable is manually changed in
QAUTOTMP/KMSPARM(KBBENV)
z/OS
In z/OS these values are kept in the RKANPARU(KDSENV) member. for TEMS and in RKANPARU(KppENV) for Agents. Configuration changes here are manual.
The Wonderful World of Situations 