Overview

ITM uses TCP/IP [ see Note 1 for exceptions] to communicate between and within ITM processes. It uses port numbers to perform this activity. Customers who control port usage closely have questions about what ports are used and also what options are available to control and limit those ports. This document explains all these questions. If there is a topic you know well, please skip to the later sections which explain exactly how to control the ports. The presentation uses standard default port numbers but almost anything can be configured.

Terminology

In TCP/IP communications a port is a number from 0 to 65535. A process can ask to be notified if another process – on this or another server – wants to connect. A call to the TCP bind routine links the incoming communication to the listening process.

A network interface is a hardware and/or software construction that has one or more IP addresses. That can be IPV4 or IPV6 addresses. In simple systems there is typically a single hardware part which implements the network interface and usually a software network interface for localhost or 127.0.0.1, which is only known locally. Servers may have many network interfaces and a single hardware interface can even have multiple IP addresses.

When an external process connects to the listening port, that produces a socket connection. The connection can be read or written from either side and the TCP/IP software transmits the data.

ITM Usage of TCP/IP

ITM does not control outbound communication. It writes data to a target ip address and port number and lets TCP/IP calculate the best way to do that work. All TCP/IP environments have “route” commands which lets the system administrator control what network interface gets used for the communication.

By default, ITM listens on all interfaces using an anonymous BIND call. The KDEB_INTERFACELIST [and KDEB_INTERFACELIST_IPV6] environment variable can be used to force an exclusive bind. Remember you cannot mix anonymous and exclusive binds for ITM usage in a single system. See Sitworld: ITM Protocol Usage and Protocol Modifiers for all the details.

ITM and Location Broker

Each ITM process uses a communication environment variable KDE_TRANSPORT [mostly z/OS] or  KDC_FAMILIES [mostly Linux/Unix/Windows/i5] which names the protocols supported and uses protocol modifiers. See this above ITM Protocol document for details. Here is a relatively simple example connection string as shown in a TEMS diagnostic log

• KDE_TRANSPORT=KDC_FAMILIES=”ip.pipe port:1918 ip use:n ip.spipe use:n sna use:n HTTP:1920“

That text is composed from the data set by the user during an ITM configuration.

The incoming ports on hub and remote TEMSes are owned by a Location Broker. During that initial connection, information is provided which allows the connecting program to look up a service and the ip address, port number and other items needed for a connection. Location Brokers run at each TEMS and a hub TEMS maintains a master list – the Global Location Broker.

When a service registers, that information is added to the Location Broker data. For example a TEMS will register all the available network interfaces. If KDEB_INTERFACELIST is supplied, the ip addresses listed will be registered with those as first priority. In this way a service can tell a user what ip address and port to use.

The important thing to remember is that services register and users look up the data. The decentralized approach makes the process more resilient and high performance because there are fewer choke points.

ITM Port Usage – Agents

In a default configuration, agents use the following ports,

1) Connection to a TEMS base port – for example the ip.pipe protocol default port is 1918. The communications string defines the protocols used. The CT_CMSLIST environment variable names the servers where a TEMS may be running. The initial connection at port 1918 gives access to the Location Broker data and enough information to call TEMS routines. However from the standpoint of configuration, this is a socket to a 1918 listening port on the server running the TEMS.

2) The initial agent listening port at 1918+N*4096. If there is just one agent installed, the listening port will be 1918+4096 or 6014. If there are more than one agent installed, the agents contend for the listening ports. That means incidentally that there are a maximum of 15 agents using the default configuration. The listening port is used for several purposes including retrieval of real time data and receiving broadcasts about new WPA address.

3) The first working CT_CMSLIST entry that works is recorded as the primary TEMS for the agent. That is important later when Fallback to Primary logic is used.

4) The agent listening port follows that rule for first contact. BUT! BUT! BUT! if communications is lost to the TEMS and later reconnected, a new listening port is acquired from TCP. The fundamental Internet RFC documents require that – to handle late arriving packets or packet fragment smoothly.  That port gotten from TCP is known as a temporary or ephemeral port. [Ephemeral is from Greek and means “lives for just one day”]. TCP hands out temporary ports based on platform rules [example Linux temporary ports are always greater than 1024]. All TCP implementations also make use of a service file which states which ports must be reserved for expected processes. If every process followed those rules there could never be a conflict!! In any case, an agent listening port can be almost anything after it has been running for a while. ITM is a dynamic system which can managed 20,000 agents or more and dynamic environment means things change.

5) Most agents connect to a Warehouse Proxy Agent or WPA. This can be at any port but is now defaulted to 63358 = 1918+15*4096 [or 65100 = 3660+15*4096 for ip.spipe]. The agent determines the contact information found from the Location Broker. The WPA registers at the hub TEMS, the Global Location Broker information is propagated to the Local Location Broker at the remote TEMS and the agent looks it up as part of the Location Broker data.

6) By default all Agents and most ITM processes start an internal web server. The default ports used are 1920 and 3661 [http and https]. There are several purposes including the ability to start and stop diagnostic tracing dynamically. The first ITM process owns the 1920/3661 and the others register with it. If that process stops the 1920 ownership switches to another ITM process if possible.In addition each internal web server will have two additional ephemeral listening ports. These are used to implement a single web image view. When you connect to the internal web server, you get to a service index page which puts all the resources from all the internal web services in one spot. See Note 2 at the end on how that neat trick is performed. Many ITM installation prefer to turn the internal web server off – exactly how explained later.

7) Ephemeral ports. ITM makes use of ports which are received from TCP/IP as “the next free port”. These are used to communicate between ITM sub-systems. You can control what ephemeral ports are allocated using the POOL Protocol Modifier.

8) Localhost ports. These are on  127.0.0.1 which are not internet capable addresses. They are used to maintain awareness between ITM processes such as handling the internal web server switch process. At ITM 630 FP6 there is an environment variable to control what port number are used.

9) Each TCP socket connection uses two ports. One is the well known port like the TEMS 1918. The other is a temporary or ephemeral port which is never in listen mode. You can see it in a netstat -an output [local and foreign targets]. It is used to implement the dual-fifo asynchronous pipe logic which is TCP sockets.

Almost every single number you see in the above description is configurable except for 4096. So if you need to make a change you can always make that change. The rest of the technote shows how to limit or control port usage.

You can never firmly tie down what ports will be used by any agent long term. You can control initial configuration but then things can drift. In fact the TEMS Audit report uses the fact that an agent got multiple listening ports as a signal that the agent is having communication or configuration issues. Even within the TEMS, you can see that there is no long term socket connection to an agent. If there is a need for real time data for example, the ip address and port are looked up at the time needed. See Note 3 about what the TEMS is REALLY using to communicate… it isn’t always an ip address.

ITM Port Usage – Servers

In most ways servers like TEMS/TEPS/S&P/WPA are also agents. You use the same communications string to control.

Hub TEMS and TEPS must have the internal web server present for normal operations.

The remote TEMS have an added control named gbl_site.txt which identifies which TEMS the TEMS connects to. That is unrelated to this technote.

If Fault Tolerant Option is configured, each hub TEMS connects to the other one using the correct base port.

Changing the Agent Configuration

To implement changes you need to make a permanent change to the Agent configuration. If you make a communications string change to one ITM process it often has to be made to all ITM processes on that server. Here are two technotes that help explain that process. The first relates to just changing the communications string:

• Updating Linux/Unix agent KDC_FAMILIES configurationhttp://www.ibm.com/support/docview.wss?uid=swg21390561

The second is a more general technote which has been vetted by L3 and Development as valid long term.

• Linux/Unix Persistent Configuration Changes and Windows commenthttp://www.ibm.com/support/docview.wss?&uid=swg21502892

Limiting ITM Agent Internal Web Server ports.

The internal web server start can be suppressed. To do that just add

• http_server:n

to the start of the communications string.

This will eliminate the ports associated with the internal web server:  1920/3661 ports and also the two ephemeral ports which mirror 1920/3661.

Alternatively, you can also eliminate specific HTTP ports by adding HTTP:0 or HTTPS:0 protocol modifiers to the communications string.

NOTE: The internal web server is required on the TEPS and the hub TEMS, otherwise important ITM functions will fail.

Limiting ITM listening ports

If you add EPHEMERAL to the connection string like this

• ip.pipe port:1918 EPHEMERAL:Y use:y

then the agent will not use an open listening port or a WPA port. There is a limitation: if historical data is going to be collected either 1) a WPA is required on the TEMS the agent reports to or 2) historical data collection must be at the TEMS. This can actually be configured at the TEMS by adding EPHEMERAL:INBOUND in to the TEMS connection string.

This also makes setting up connections through firewalls easier since only the base port 1918 needs to be permitted. There is no significant performance impact with this choice. This modifier can be made to one agent but not another on the same server.

The diagnostic log will reference internal virtual ports [like 6015 for example] but these are invisible to TCP.

Controlling Ephemeral ports

These ports cannot be eliminated but you can configure them to certain number ranges using the POOL protocol modifier as described here:

Sitworld: ITM Protocol Usage and Protocol Modifiers

Universal Agent Ports

The Universal Agent is an ITM component that lets you extend ITM by writing your own agents.

UA uses all the same ports and by default will also use port 1919 to communicate with collectors [IANA registered]. Each data collector process will use an ephemeral port to form the socket is created.

KUMP_LOCAL_DATA=Y configures non-socket communication on a single server. In a very few cases that configuration causes collection issues.

Please consider use of Agent Builder instead which is being actively developed.
The tacmd createNode Function uses Ports

The tacmd createNode function is largely implemented by a java program running at the hub TEMS. It listens for work on [default] port 1978, a specific bind to the ip address of the system where the java program and TEMS are running. This port can be altered using the TEMS environment variable KDY_MANAGE_PORT and would usually be set in ms.ini or in ms.environment persistent configuration override file.

This function gets used for the first install of OS Agent on a system. Linux/Unix uses SSH/RSH/REXEC from the hub TEMS to the target agent. For example, SSH usually uses port 22. During agent createNode processing the service port and port 1918 from agent to hub will be used. Afterwards the agent will usually connect to a remote TEMS.

Summary

This document describes ITM port usage and shows ways you can eliminate or control port usage.

Sitworld: Table of Contents

Note 1

In z/OS the SNA communications can be used for communication. This document does not apply to that option. The Portal Client can use two communication techniques: http/https and then CORBA communications, which is also unmentioned in this document. The ITM EIF facility and the LDAP facility can use ports not described here.

Note 2 – Multiple Internal Web Servers

This discussion is only for http/https at port 1920/3661. In fact the logic  the IPV6 equivalents. Suppose there are three agents starting up at the same time. They all attempt to bind to the 1920 and 3661 listening ports. One succeeds and owns that listening ports. The failing two internal web servers make a connection to the winner and they also register the two ephemeral ports which parallel 1920 and 3661. The winner accumulates that data and generates the service index page. For example, if you click on a certain service on the index page, you might well be redirected to the the ephemeral parallel port on another internal web server associated with another agent.

Should the agent running the winner be recycled, the other two agents notice that their connection to the winner have failed. At that point they “start all over” and one of them becomes the winner and owner of port 1920/3661. When the original winner agent starts again, it attempt to get 1920/3661, fails, and then registers like all good losers.

In that way a single image is preserved. It all works just fine unless there are firewall rules that allow only specific ports. You can get to the 1920 port, lets say, but when you click on a service, you are directed to some ephemeral port and the firewall probably blocks it.

As usually there is considerable flexibility. As seen in the Protocols blog post you can use HTTP:nnnn and HTTPS:nnnn specify different listening ports. Incidentally, the ITM ports 1918/1919/1920/3660/3661 are registered with the IANA – Internet Assigned Numbers Authority. That tends to limit accidental conflicts.

Note 3 – The TEMS PIPE_ADDR control

In simple cases, TEMS appears to be using the agent ip address. But things are not always simple!

For example, an agent might be behind a Network Address Translation firewall. To connect to a TEMS is might have to use ip address 1.2.3.4 and port 4590. On the TEMS side access would be made to the TEMS ip address and port 1918. ITM Communications logic makes this happen with no configuration needed at all.

TCP socket logic is modeled and extended. Normal TCP socket is the functional equivalent of Unix dual-fifo asynchronous pipes. In effect either side can write to the other at any time. The TEMS PIPE_ADDR is a control that tracks how to communicate with an agent process. For the simplest of cases, it will be like an IP Address and all the rest of the information is default and mostly ignored. For the case of a incoming “beyond firewall” connection, an artificial address is used like 0.0.0.20 and some arbitrary port. In that case the TEMS side firewall address and port is recorded, as well as the beyond firewall agent side address and port… and of course some controls indicating what sort of translation is needed.

These PIPE_ADDR are specific to a TEMS. So a hub TEMS would have no idea how to contact an agent connected via a remote TEMS. The hub TEMS passes the work off and the remote TEMS has thy PIPE_ADDR information and knows what to do.

This is also used to handle agents configured with EPHEMERAL:Y – where an Agent=>TEMS socket connection multiplexes three different virtual tcp socket connections.

This is also used to handle agents connecting via KDE_Gateway logic – an more complex socket multiplexing solution. KDE_Gatway is needed with there is more that one NATing firewall router between the Agent and the TEMS. TEMS can handle just a single address translation link.

The last area I have seen this used is when a link on z/OS utilizes SNA communications. It translates between SNA and TCP transparently. There may be others.

This is complex but communications is often complex and ITM6 was built to work in that environment with minimal configuration.