Version 1.64000 31 May 2017
John Alvord, IBM Corporation
jalvord@us.ibm.com
Following is the information on tracing for TEMS Audit
There was a document size limit but is logically a part of it the document.
Appendix 1
Some of the advisories and report sections require diagnostic tracing. We will use a standard workload tracing for these examples. Independent of the implementation the diagnostic trace string looks like this:
error (unit:kpxrpcrq,Entry=”IRA_NCS_Sample” state er)(unit:kshdhtp,Entry=”getHeaderValue” all) (unit:kshreq,Entry=”buildSQL” all)(unit:kfastpst,Entry=”KFA_PostEvent” all er)(unit:kdssqprs in metrics er)(unit:kdsstc1,Entry=”ProcessTable” all er)(unit:kraafira,Entry=”runAutomationCommand” all)(unit:kglhc1c all)
You always need error. The rest are additions to trace more.
There are multiple ways to set this tracing:
Static Trace Definitions – requires a process recycle
Linux/Unix
The best way is to add a file ms.environment to the <installdir>/config directory which has the same attributes/owner/group as ms.ini. Use touch/chmod/chown/chgrp to create such a file. If one already exists use it. Add the following to that file – one long line.
KBB_RAS1= error (unit:kpxrpcrq,Entry=”IRA_NCS_Sample” state er)(unit:kshdhtp,Entry=”getHeaderValue” all) (unit:kshreq,Entry=”buildSQL” all)(unit:kfastpst,Entry=”KFA_PostEvent” all er)(unit:kdssqprs in metrics er)(unit:kdsstc1,Entry=”ProcessTable” all er)(unit:kraafira,Entry=”runAutomationCommand” all)(unit:kglhc1c all)
Windows
Using MTEMS, right click on the TEMS line, click Advanced, click Edit Trace Parms…
In the Ras1 Filter enter
error (unit:kpxrpcrq,Entry=”IRA_NCS_Sample” state er)(unit:kshdhtp,Entry=”getHeaderValue” all) (unit:kshreq,Entry=”buildSQL” all)(unit:kfastpst,Entry=”KFA_PostEvent” all er)(unit:kdssqprs in metrics er)(unit:kdsstc1,Entry=”ProcessTable” all er)(unit:kraafira,Entry=”runAutomationCommand” all)(unit:kglhc1c all)
Note there is no KBB_RAS1= in this context.
z/OS
Add the following to the RKANPARU(KDSENV) file
KBB_RAS1= error (unit:kpxrpcrq,Entry=”IRA_NCS_Sample” state er)(unit:kshdhtp,Entry=”getHeaderValue” all) (unit:kshreq,Entry=”buildSQL” all)(unit:kfastpst,Entry=”KFA_PostEvent” all er)(unit:kdssqprs in metrics er)(unit:kdsstc1,Entry=”ProcessTable” all er)(unit:kraafira,Entry=”runAutomationCommand” all)(unit:kglhc1c all)
At that point recycle the TEMS and collect the data.
The ms.environment technique works from ITM 623 GA onward. Before that you can achieve the same goal by updating the TEMS config file
hostname_ms_temsnode.config
with the configuration string added in single quotes like this:
KBB_RAS1= ‘error (unit:kpxrpcrq,Entry=”IRA_NCS_Sample” state er)(unit:kshdhtp,Entry=”getHeaderValue” all) (unit:kshreq,Entry=”buildSQL” all)(unit:kfastpst,Entry=”KFA_PostEvent” all er)(unit:kdssqprs in metrics er)(unit:kdsstc1,Entry=”ProcessTable” all er)(unit:kraafira,Entry=”runAutomationCommand” all)(unit:kglhc1c all)’
Such a temporary update will be lost during a TEMS configuration, so it is fine for cases like this.
Dynamic – tacmd settrace
The best modern way to run such a command is via tacmd settrace. The sequence looks like this when tacmd is run from Linux/Unix. Note that the <temsnodeid> is the TEMS nodeid and not the hostname running the TEMS. This what would be seen in a tacmd listsystems output.
Linux/Unix
cd <installdir>/bin
./tacmd login -s …. [login to hub TEMS]
./tacmd settrace -m <temsnodeid> -p KBB_RAS1 -o ‘error (unit:kpxrpcrq,Entry=”IRA_NCS_Sample” state er)(unit:kshdhtp,Entry=”getHeaderValue” all) (unit:kshreq,Entry=”buildSQL” all)(unit:kfastpst,Entry=”KFA_PostEvent” all er)(unit:kdssqprs in metrics er)(unit:kdsstc1,Entry=”ProcessTable” all er)(unit:kraafira,Entry=”runAutomationCommand” all)(unit:kglhc1c all)’
Note the single quotes around the diagnostic trace parameters.
After some time – usually a couple hours – you can disable the trace like this:
cd <installdir>/bin
./tacmd login -s …. [login to hub TEMS]
./tacmd settrace -m <temsnodeid> -p KBB_RAS1 -r
Windows
The commands are largely the same but quoting is different. External quotes are double quotes and embedded double quotes are tripled.
cd <installdir>bbin
tacmd login -s …. [login to hub TEMS]
tacmd settrace -m <temsnodeid> -p KBB_RAS1 -o “error (unit:kpxrpcrq,Entry=”””IRA_NCS_Sample””” state er)(unit:kshdhtp,Entry=”””getHeaderValue””” all) (unit:kshreq,Entry=”””buildSQL””” all)(unit:kfastpst,Entry=”””KFA_PostEvent””” all er)(unit:kdssqprs in metrics er)(unit:kdsstc1,Entry=”””ProcessTable””” all er)(unit:kraafira,Entry=”””runAutomationCommand””” all)(unit:kglhc1c all)”Note the single quotes around the diagnostic trace parameters.
After some time – usually a couple hours – you can disable the trace like this:
cd <installdir>\bin
tacmd login -s …. [login to hub TEMS]
tacmd settrace -m <temsnodeid> -p KBB_RAS1 -r
Example Trace Strings
Hub TEMS basic workload:
./tacmd settrace -m <temsnodeid> -p KBB_RAS1 -o “error (unit:kpxrpcrq,Entry=”””IRA_NCS_Sample””” state er)(unit:kshdhtp,Entry=”””getHeaderValue””” all) (unit:kshreq,Entry=”””buildSQL””” all)(unit:kfastpst,Entry=”””KFA_PostEvent””” all er)(unit:kdssqprs in metrics er)(unit:kdsstc1,Entry=”””ProcessTable””” all er)(unit:kraafira,Entry=”””runAutomationCommand””” all)(unit:kglhc1c all)”
Remote TEMS basic workload:
error (unit:kpxrpcrq,Entry=”IRA_NCS_Sample” state er) (UNIT:kfaprpst ER ST) (UNIT:kfastinh,ENTRY:”KFA_InsertNodests” ALL)(unit:kdssqprs metrics in er)(unit:kdsstc1,Entry=”ProcessTable” all er)(unit:kraafira,Entry=”runAutomationCommand” all)(unit:kglhc1c all)
Hub TEMS Workload plus Heartbeat:
error (unit:kpxrpcrq,Entry=”IRA_NCS_Sample” state er) (UNIT:kfaprpst ER ST) (UNIT:kfastinh,ENTRY:”KFA_InsertNodests” ALL)(unit:kdssqprs metrics in er)(unit:kdsstc1,Entry=”ProcessTable” all er)(unit:kraafira,Entry=”runAutomationCommand” all)(unit:kglhc1c all)
Remote TEMS Workload plus Heartbeat:
error (unit:kpxrpcrq,Entry=”IRA_NCS_Sample” state er) (UNIT:kfaprpst ER ST) (UNIT:kfastinh,ENTRY:”KFA_InsertNodests” ALL)(unit:kdssqprs metrics in er)(unit:kdsstc1,Entry=”ProcessTable” all er)(unit:kraafira,Entry=”runAutomationCommand” all)(unit:kglhc1c all)(UNIT:kfaprpst ST ER)
Hub TEMS plus heartbeat plus KPX traces to watch input in detail:
error (unit:kpxrpcrq,Entry=”IRA_NCS_Sample” state er)(unit:kshdhtp,Entry=”getHeaderValue” all) (unit:kshreq,Entry=”buildSQL” all)(unit:kfastpst,Entry=”KFA_PostEvent” all er)(unit:kdssqprs in metrics er)(unit:kdsstc1,Entry=”ProcessTable” all er)(unit:kraafira,Entry=”runAutomationCommand” all)(unit:kglhc1c all)(UNIT:kfaprps ST ER)
Remote TEMS plus heartbeat plus KPX traces to watch input in detail:
error (unit:kpxrpcrq,Entry=”IRA_NCS_Sample” state er) (UNIT:kfaprpst ER ST) (UNIT:kfastinh,ENTRY:”KFA_InsertNodests” ALL)(unit:kdssqprs metrics in er)(unit:kdsstc1,Entry=”ProcessTable” all er)(unit:kraafira,Entry=”runAutomationCommand” all)(unit:kglhc1c all)(UNIT:kfaprps ST ER)(UNIT:kfastinh,ENTRY:”KFA_InsertNodests” ALL)(UNIT:kpxreq ALL)(UNIT:kpxreqds ALL)
Rarer dynamic options
There was an earlier way to make dynamic tracing changes documented here:
Dynamically modify trace settings for an IBM Tivoli Monitoring component
http://www-1.ibm.com/support/docview.wss?rs=0&uid=swg21266129
It is sometimes blocked by firewall restrictions and lack of login credentials to the system running the TEMS.
There is also a z/OS TEMS option which looks a bit like
CTDS TRACE ADD FILTER ID=001 UNIT=KOCACHE CLASS(ALL)
If must use that please contact the author for details.
Versions:
Here are recently published versions, In case there is a problem at one level you can always back up.
1.64000 – first publication of the trace appendix
Sitworld: Table of Contents
Note: Art Deco Cat sculpture
The Wonderful World of Situations 