setupCmdLine.sh

The WebSphere Application Server uses the setupCmdLine.sh file, which defines the required environment needed for WebSphere Application Server. The file is sourced in all the start, stop, and monitoring scripts used by the WebSphere Application Server agent.

bash-3.00# cat /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/startServer.sh

#!/bin/sh

binDir='dirname ${0}'

. ${binDir}/setupCmdLine.sh

${WAS_HOME}/bin/startServer.sh "$@"

By design, the WebSphere Application Server agent does not source the user's profile. If you need to export any additional environment variables, create an environment file with the required variables and source it in following scripts:

startManager.sh, startNode.sh, startServer.sh, stopManager.sh, stopNode.sh, stopServer.sh, serverStatus.sh

For example, append a line in startServer.sh in the following manner:

# Call User Environment here.

. <Path to env file>/setUserEnv.sh




We run setupCmdLine.sh because this file contains the paths of all the commands that have to run. Running this command makes sure that the path and classpath is updated with the most recent path.

/opt/was/AppServer/profiles/<<profilename>>/bin/setupCmdLine.sh

             Once you run setupCmdLine.sh, Switch back to your login credentials and run startManager.sh to start the deployment manager or corresponding nodeagent/server.

In order to run setupCmdLine.sh, the user must be logged on as was. To do this, do a $su – was. Enter the password when prompted.
setupCmdLine.sh

sample setupCmdLine.sh file

#!/bin/sh

# @(#) 1.21 CFG/ws/code/profile.templates/src/bin/setupCmdLine.sh, WAS.config.base, WAS61.CFG, cf250912.19 5/8/06 13:29:35 [6/13/09 11:33:16]

WAS_USER_SCRIPT="/opt/was/AppServer/profiles/<<profilename>>/bin/setupCmdLine.sh"

export WAS_USER_SCRIPT

USER_INSTALL_ROOT="/opt/was/AppServer/profiles/<<profilename>>"

WAS_HOME="/opt/was/AppServer"

JAVA_HOME="/opt/was/AppServer/java"

WAS_CELL=<<cell name>>

WAS_NODE=<<node name>>

#ulimit -n 10000

OSGI_INSTALL="-Dosgi.install.area=$WAS_HOME"

OSGI_CFG="-Dosgi.configuration.area=$USER_INSTALL_ROOT/configuration"

ITP_LOC="$WAS_HOME"/deploytool/itp

CONFIG_ROOT="$USER_INSTALL_ROOT"/config

CLIENTSAS=-Dcom.ibm.CORBA.ConfigURL=file:"$USER_INSTALL_ROOT"/properties/sas.client.props

STDINCLIENTSAS=-Dcom.ibm.CORBA.ConfigURL=file:"$USER_INSTALL_ROOT"/properties/sas.stdinclient.props

SERVERSAS=-Dcom.ibm.CORBA.ConfigURL=file:"$USER_INSTALL_ROOT"/properties/sas.server.props

CLIENTSOAP=-Dcom.ibm.SOAP.ConfigURL=file:"$USER_INSTALL_ROOT"/properties/soap.client.props

JAASSOAP=-Djava.security.auth.login.config="$USER_INSTALL_ROOT"/properties/wsjaas_client.conf

CLIENT_CONNECTOR_INSTALL_ROOT="$USER_INSTALL_ROOT"/installedConnectors

CLIENTSSL=-Dcom.ibm.SSL.ConfigURL=file:"$USER_INSTALL_ROOT"/properties/ssl.client.props

WAS_LOGGING="-Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager -Djava.util.logging.configureByServer=true"

QUALIFYNAMES=-qualifyHomeName

PATH="$JAVA_HOME"/ibm_bin:"$JAVA_HOME"/bin/:"$JAVA_HOME"/jre/bin:$PATH

WAS_EXT_DIRS="$JAVA_HOME"/lib:"$WAS_HOME"/classes:"$WAS_HOME"/lib:"$WAS_HOME"/installedChannels:"$WAS_HOME"/lib/ext:"$WAS_HOME"/web/help:"$ITP_LOC"/plugins/com.ibm.etoo

ls.ejbdeploy/runtime

WAS_CLASSPATH="$WAS_HOME"/properties:"$WAS_HOME"/lib/startup.jar:"$WAS_HOME"/lib/bootstrap.jar:"$WAS_HOME"/lib/j2ee.jar:"$WAS_HOME"/lib/lmproxy.jar:"$WAS_HOME"/lib/urlp

rotocols.jar:"$JAVA_HOME"/lib/tools.jar

PLATFORM=`/bin/uname`

case $PLATFORM in

  AIX)

    WAS_LIBPATH="$WAS_HOME"/bin

    NLSPATH=/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/en_US/%N:${NLSPATH:=}

#    WAS_BOOTCLASSPATH=

    ;;

  Linux)

    WAS_LIBPATH="$WAS_HOME"/bin

    NLSPATH=/usr/lib/locale/%L/LC_MESSAGES/%N:${NLSPATH:=}

    JAVA_HIGH_ZIPFDS=200

#    WAS_BOOTCLASSPATH=

    ;;

  SunOS)

    if [ "$LANG" = "" ]

    then

       LANG=C

       export LANG

    fi

    WAS_LIBPATH="$WAS_HOME"/bin

    NLSPATH=/usr/lib/locale/%L/LC_MESSAGES/%N:${NLSPATH:=}

#    WAS_BOOTCLASSPATH=

    ;;

  HP-UX)

    WAS_LIBPATH="$WAS_HOME"/bin

    NLSPATH=/usr/lib/nls/msg/%L/%N:${NLSPATH:=}

#    WAS_BOOTCLASSPATH=

    ;;

  *)

    WAS_LIBPATH="$WAS_HOME"/bin

    NLSPATH=/usr/lib/locale/%L/LC_MESSAGES/%N:${NLSPATH:=}

#    WAS_BOOTCLASSPATH=

    ;;

esac

export PATH WAS_HOME WAS_CELL WAS_NODE JAVA_HOME ITP_LOC CLIENTSAS CLIENTSSL STDINCLIENTSAS SERVERSAS CLIENTSOAP CLIENT_CONNECTOR_INSTALL_ROOT WAS_LOGGING QUALIFYNAMES

WAS_EXT_DIRS WAS_CLASSPATH CONFIG_ROOT NLSPATH JAVA_HIGH_ZIPFDS WAS_LIBPATH OSGI_INSTALL OSGI_CFG

Sources:

https://sort.symantec.com/public/documents/sfha/6.1/linux/productguides/html/applicationha_webas_agent_61_kvm_lin/ch03s03.htm

close wait Issue

Close Wait problem

CLOSE_WAITS occur when the connection is open at one end and closed at one end.

CLOSE_WAITS occur due to a lot of reasons and the actual reason has to be figured out.

Login to the Webserver that shows the CLOSE_WAIT problem and run the below 

netstat -an | egrep 'CLOSE_WAIT' | awk '{print $5}'| sort | uniq -c | sort -nr

or place it as a shell script in /usr/local/adm/bin/count_closewait and run count_closewait going forth

//content of count_closewait.sh script is stated below

**begin

clear

echo "Here is the current Snapshot of CLOSE_WAIT ... "

echo

echo "Count Applicaiton (IP+Port) "

echo "===== ===================== "

netstat -an | egrep 'CLOSE_WAIT' | awk '{print $5}'| sort | uniq -c | sort -nr

echo

**end

below is a snapshot of the output 

Count Applicaiton (IP+Port)

===== =====================

  120 <<ipaddress1>>.9081

  110 <<ipaddress1>>.9087

   90 <<ipaddress1>>.9086

   85 <<ipaddress1>>.9088

   83 <<ipaddress1>>.9083

   88 <<ipaddress2>>.9082

   73 <<ipaddress2>>.9087

   71 <<ipaddress2>>.9084

   60 <<ipaddress2>>.9083

From the above output we infer the count of closewaits to a particular Server on a particular port. This displays the IP+Port with the maximum closewaits, at top.

1. Out of all the outgoing connections from that server, check the IP and the corresponding port no. with the most number of CLOSE_WAIT’s associated with it.

1. After identifying the ip and port no, do an nslookup to find the name of that server.

    nslookup <ipaddress>

Now login to the appropriate server as found from the above step.

1. Now find the process id of the process running on that port number of the server on which the CLOSE_WAIT was detected. This can be done by using the lsof command

  lsof  | grep  <port no>

The corresponding Java/JVM instance will be process ID will be listed 

java    38863002  was  .....                0t0    TCP *:9083 (LISTEN)

ps -ef |grep <<pid>> in the above case 

ps -ef |grep 38863002 and the JVM instance will be found.

NOTE : Alternately you can also check the webserver plugin-cfg.xml for the port number above which will give you the application server instance being referred by it.

Now check the systemout logs of that JVM for the reason of CLOSE_WAIT.

1. Some times the CLOSE_WAIT’s may be a result of network connectivity issues. Contact the Networks team to resolve the issue.

1. If there is no problem in network connectivity, check the applications that are running on that JVM. This can be found by using the info_app command

      info_app | egrep ‘<JVM>|Server’

1. If any application has a problem, then the CLOSE_WAIT’s may be due to the non responsiveness of that application. Contact the application support team for that and meanwhile perform the recycle of the web server to prevent any other applications having problems.

          If all the applications are running fine, then go ahead and recycle the Web Server.

      Notes

§  At any point of time, if you feel that the number of CLOSE_WAITS has gone too high, you’d better recycle the Web server. That should clear the issue

§  Usually, after performing the recycle of the web server on one side, the number of CLOSE_WAIT sockets on the other side also increases due to the increased load on the other server. Hence it is advised to recycle the other web server after the first one.