Geekcorp Software

A TCL based SNMP
Scotty Daemon
v1.0.4 Release
Sveinar Rasmussen
April 2000 

Abstract. This document describes the SNMP daemon programmed in TCL and running under the Scotty environment. Today's business networking environments are very complex and different from another. The SNMP is today's networking standard for monitoring networks. It's simple and that makes it consistent. RFC 1514 on Host Resources MIB explains a lot of portable information for different hosts, and most of the non-commercial SNMP daemons available today do not support this information in the MIB II tree. The SNMP daemon presented in this document, will try to provide the host resources information consistently in a portable way.

1.0 Introduction

    Network management is a requirement for anyone who wants to control and monitor their LANs and WANs. This vast new empire of products, designed to act as cohesive and well-organized networking systems, can quickly turn into a disorganized mass of independently operating devices. To alleviate these problems, SNMP based network management applications must be employed. Example, the BigBrother system uses an own protocol instead of the SNMP to send out information to other hosts. Because of this, one cannot view the variables in one of the most common network monitoring tools. The Snmp Monitor Ex (a Scotty extension) does not speak the non-standard rother language and cannot monitor these variables.

    Network management systems should also be able to work with a large variety of operating systems and workstation platforms. The SNMP daemon, abbreviated snmpd from now on, is written in the portable script language Tcl. Non-heterogeneous operating systems have totally different ways of providing information to the user. By using the appropriate routines depending on the running OS, the snmpd fills in the appropriate variables in the MIB-2 resources tree.

    Snmpd was not designed to be an independent SNMP daemon. Being more of an extension providing the Host Resources MIB (RFC 1514), it is recommended that the system administrator has an additional SNMP daemon running on port 161 (the original SNMP port). See below for ways to multiplex two agents on the snmp port.

    Snmpd supports the following platforms: HP-Unix (v10.20), IRIX (v6.2), NetBSD (v1.2), SunOS (v5.5) and Linux (v2.0.30). The numbers in parentheses show under what version the snmpd was tested under. Snmpd is meant to be run as a normal user (i.e. you don't have to run this program with root properties). The port multiplexor does, however, need root privileges to bind to the port 161.

1.1 Capabilities of the system management agent (Host Resources)

2.0 Usage

    1) First of all, you need a working Scotty installation. Of course, you will need Tcl/Tk as required by Scotty.

    2) Run ./configure and let it do everything for you. The output should be something like this (output is grabbed running the configure script on HP box):
     

             TCL based SNMP Scotty Daemon Setup
                    _____  ____  ______  ___
                   / __/ |/ /  |/  / _ \/ _ \
                  _\ \/    / /|_/ / ___/ // /
                 /___/_/|_/_/  /_/_/  /____/

      Checking standard external programs for all platforms...
        Checking who...                                      ok!
        Checking lpstat...                                   ok!
        Checking uname...                                    ok!
      Checking hpux platform dependent external programs...
        Checking ioscan...                                   ok!
        Checking ioscan for its set-group ID...              ok!
        Checking vmstat...                                   ok!
      Searching for scotty...
        Guessed newest scotty at /store/bin/scotty2.1.6
        Updating Tcl script 'snmpd' with information...      ok!
      Building snmpd-sys which is a low-level C program...
              gcc -O3 -s   -o snmpd-sys snmpd-sys.c    -DOS_UNIX -DSPEC_HPUX
        Compilation was a success.
              cp snmpd-sys ..
              make clean
              rm -f snmpd-sys core *~ *.o
        Copying snmpd-sys to snmpd dir...
        Cleaning up...
      Building snmpd-mux which is a SNMP port 161 2-multiplexor...
              cc -O -s -Ae  -o snmpd-mux snmpd-mux.c
        Compilation of snmpd-mux was a success.
              cp snmpd-mux ../lib/
              make clean
              rm -f snmpd-mux snmpd-sys core *~ *.o
        Copying snmpd-mux to snmpd/lib dir...
        Cleaned up...

      Setup program finished.
      Thank you!
       

    3) Edit the snmpd.conf file. Among other things, this file tells the daemon what port numbers it will use, community and the values of static variables upon startup. You should also specify the update intervals. Comments in the file are marked with a "#" in the beginning of each line.

    Now, a few words about the configuration file. The UpdateHr*Int settings tell snmpd how often it is supposed to update the variables in the specified class. Formally, it is the number of seconds between each variable update upon user request.

    The TotalUpdateInt is the interval between a complete update of all the variables.

    To avoid memory leaks or any problems, snmpd has been designed to automatically shutdown and restart itself. The RestartAgentInt setting tells the number of seconds between each agent restarting procedure. After removing a lot of memory leaks from this program, the recommended value should be at least more than a week. This could of course vary under severe system load (a lot of SNMP queries). The whole restarting procedure takes less than 10 seconds. All the set variables are restored on restart.

    The RebuildVarsListCount tells snmpd to regenerate the list of set operations after a number of set requests. In true spirit of making the common case fast, the restructuring of the data file is only done after a given number of set requests towards snmpd. Thus, this speeds up the set operation since it only concatenates changes to a file when a set operation comes along. This list is used to make the agent remember and override static variable settings by the user the next time it is fired up.

    Next, there are three settings for the logging system of the snmpd. You can specify a valid directory and an arbitrary name where you want the log file of snmpd to be placed. Use the intuitive LogfileLocation for this. The LogEventLevel variable tells snmpd what events you would like to have an entry in the log file. There are four levels here: nothing, min, normal (recommended) and max. "nothing" will not place anything in the log file - no matter what goes wrong, while "max" will place a lot of information (example: snmp get requests, set requests and other various operations performed by the agent). The "normal" setting puts the most common stuff in the log file, while "min" only puts the most crucial information in the log.

    The last of the three variables, LogToSyslogLevel, is quite similar to the one just mentioned except that it specifies which of the events should go into the system's log (syslog). The level granularity is the same as with LogEventLevel.

    The observant reader then notices that there are two port settings for this agent - logically there should only be one. In reality, that's how it works, too:  ThisAgentSnmpdPort specifies the port number snmpd expects requests on. However, the  OtherAgentSnmpdPort is used by the port multiplexor. The latter is a program specifically written for this snmpd in order to concatenate the Host Resources snmp tree together with the standard enterprise snmp agent shipped with your machine (normally). In this way, the snmpd becomes more of an snmpd extension in itself rather than a standalone snmp agent. As mentioned earlier, the first port variable, ThisAgentSnmpdPort, tells the port number of this agent. The latter, OtherAgentSnmpdPort, is for the other standard snmpd agent normally running on port 161 on every machine (enterprise version). You need to configure your original snmp agent to run on a different secure port than port 161, if you decide to use the multiplexor (see below how).

    Be aware that the static variable settings given in the configuration file are standard default values and can be overridden by the user's SNMP set requests.

     
    A sample config file can be found in etc/snmpd.conf as viewed below:
    #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#
    #                    SNMPD Default Configuration                    #
    #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#
    #    Official Web Site: http://www.cc.uit.no/sveinar/Snmpd/doc/     #
    #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#

    #/////////////////////////////////#
    # Common settings                 #
    #/////////////////////////////////#
    Community                    public

    #/////////////////////////////////#
    # Intervals                       #
    #/////////////////////////////////#
    UpdateHrSystemInt                60
    UpdateHrStorageInt               60
    UpdateHrDeviceInt                60
    UpdateHrSWRunAndPerfInt          60
    UpdateHrSWInstalledInt        86400
    TotalUpdateInt                86400
    RestartAgentInt              432000
    RebuildVarsListCount             20

    #/////////////////////////////////#
    # Logging                         #
    #   Level: nothing min normal max #
    #/////////////////////////////////#
    LogfileLocation       etc/snmpd.log
    LogEventLevel                normal
    LogToSyslogLevel                min

    #/////////////////////////////////#
    # Ports (multiplexor)             #
    #/////////////////////////////////#
    ThisAgentSnmpdPort             1701
    OtherAgentSnmpdPort            1702

    #/////////////////////////////////#
    # Static variable settings        #
    #/////////////////////////////////#
    <StaticVars>
     sysContact.0  SysContact  "FMS@uit.no, phone: +47 776 45999"
     sysName.0     sysName     "slibo.cc.uit.no"
     sysLocation.0 SysLocation "EDB-Sentret, University of Tromso, Breivika,N-9037 Tromso"
     sysServices.0 sysServices "2"
    </StaticVars>

    4) Run snmpd. Voila!

    Optional:

    5) Run snmpd-mux port multiplexor to make the commercial standard agent work together with this agent. You need to stop any agents occupying port 161, configure your newly stopped agent to run on a different secure port (e.g. port 1702), restart the original agent with the new settings and lastly, run lib/snmpd-mux :)

    Hot tip: You are advised to set the snmp time-out in your snmp browser to 10 seconds if you're running an old out-ranged HP Unix 700 series machine with approximately 30 percent load. This way, you will never get annoying time-outs.

2.1 Files and directory structure

As you can see from the setup program, the snmpd uses a lot of external programs to figure out a few variables. The table below will explain:
 
Table showing external program name, what it is for and regarding platform
snmpd-sys   used for a lot of internal stuff. See 2)   All
who   used to figure out number of users logged on   All
ps   used to find number of procs and names of procs   All except HPUX
ioscan   used to scan devices for HPUX   HPUX
hinv   used to scan devices and memSize for IRIX   IRIX
vmstat   used to get CPU load for unix's (except IRIX)   All except IRIX
lpstat   used to check printer jobs   All
uname   used to determine operating system name   All
systune   used to get MaxProcesses on IRIX   IRIX
The ioscan needs to have its SetUID bit set, otherwise you must run the snmpd as root.
 
Table showing the directory structure of snmpd
 File 
 Description 
 snmpd    the main program. 
 configure    the setup program which is completely platform 
  independent.
 doc/   the HTML documentation is found here.
 etc/snmpd.conf    the snmpd configuration file
 etc/snmpd.log   the default snmpd log file (optional).
 etc/snmpd.vars   the set variables file (optional). When a variable 
  is set by the user, the change is saved here. This 
  way, the setting is permanent when agent restarts. 
 lib/snmpd-sys   the low-level C program needed to get certain 
  system specific information.
lib/snmpd-mux   the C program that does the port multiplexing of 
  standard snmp port 161 (optional).
 src/Makefile   the Makefile for the low-level C program.
 src/snmpd-sys.c    the C source for the low-level program.
 src/snmpd-mux.c   the C source for the port multiplexor program.
 

3.0 Implementation issues

    The source code for this project is about 53 kB. Languages are Bourne Shell, C and Tcl - the configuration program is Bourne Shell based, the low-level info retrieval program is written in C and the main program is in Tcl.

    I will now discuss some of the Tcl implementation. The parseCfgVar is a useful procedure simplifying process of reading variable settings stored in the snmpd configuration file. It has two arguments: the name of the variable to retrieve and its default value if it isn't present in the configuration file. The return value is either the value found in the configuration file or the default value.

    updateVars is a procedure to run a complete update of all the variables in the snmpd engine on regular occasions (very seldom). This is to avoid having the users wait for a while when requesting variables about e.g. installed software - if the snmpd has to parse through store to figure this out (because of old variables), it could become quite time consuming :)

    As part of the assurance of having a stable agent, I have written the restartAgent procedure. To avoid using the execl extended Tcl command in order to maintain a strict Tcl requirement only, this procedure creates a Bourne Shell script to launch itself by.

    The CreateSafe doesn't create a safe Tcl agent at all. Recommendations from Scotty developers: "Tcl8.0 will be quite different anyway and hence the time to make the old Scotty code to work, is not very well spent." Conclusion: Wait till next Scotty version and the Tcl 8.0 release to fix this.

    Note that all the global variables binded to a snmp instance are stored in an array called  mibVals. This two-dimentional array has indices on snmp variable name and level indexes.

    The current operating system is also listed as a running process in the process table. Since process IDs are usually in the range of 0 to 65535, the ID of the OS is hardwired at PID 9999999.

    File used to get information about store is /store/store/Adm/reports/report.out. Edit the Tcl script to change this.

3.1 Extending the agent
    This agent can be extended with new functionality. If you want to extend this agent, there are three sources you might have to modify. A configuration setup for extending the agent would be valuable to make things simpler, but time didn't allow this in this release. Therefore, I will say a few words about each of the sources:

    File: snmpd                                                            

    Add a line to run the new procedure handling the actual update of the new variables. This is done by adding the procedure call in updateVars. This way, your new variables are updated as the system decides to do a full value update.

    Implement the procedure for the new branch of the MIB tree to be supported. The procedure retrieves the values from the system and these are reflected to the snmp engine.

    Add an update interval for that particular new set of variables. This step includes adding a variable setting in the configuration file and a new global variable UpdateHr[proc name]Int.

    Add a line in Main in order get a variable update upon agent startup.

    The last step is to bind the new procedure to the appropriate MIB OID. That is done in the if regexp structure found at the of the Main procedure. Hot tip: You can figure out the OID of the new variable tree by inserting  puts "Request received: OID = [lindex [lindex %V 0] 0]" in the bind code block.

  File: src/snmpd-sys.c                                                

      In order to add a new low-level functionality, you firstly have to define a new opcode for new function intended in the #define section at the beginning of the C source.

      Add the new opcode to the dispatch routine (switch-case statement) in main.

      Write the C function doing all the actual work printing out the information to standard output.

     
File: src/snmpd-mux.c                                                

      Extending the multiplexor to handle more than two agents, you should at first get the OID for the new branch of the MIB to be supported (see last bullet on notes for snmpd). Add this OID to the if structure in main. The variable client controls the destination agent after receiving a user request.

      In the beginning of main: add a client entry using setclientsock. If the second parameter is NULL, then you indicate the default agent running on the current machine. On the other hand, if the second parameter is an IP address, you are setting the socket for the extra client. The third argument of this procedure is the port number in ASCII to the agent specified. Note: you can actually place the extra agent on a different machine than the localhost (127.0.0.1) - just write the correct internet address in the second argument. The first agent points to the client structure. You have to adjust the array allocations to compensate for your agent addings.

      Adding another agent implies another port number. Adjust the configuration file to include this new port number and update this C code to read the new configuration variable, as well.

4.0 Conclusion

    The idea was to present a SNMP based network management program to alleviate certain problems of heterogeneous systems. An SNMP daemon has been successfully implemented supporting five different operating systems; HP-Unix, SunOS, IRIX, Linux and NetBSD.

    A product like this makes it possible for e.g. Snmp Monitor Ex to monitor resources on remote machines handling disk space, cpu utilization, device list, memory, file system and process information across the net. Most importantly, this program uses the SNMP to communicate - SNMP is today's networking standard for monitoring networks.

    This package is also shipped with a port multiplexor for port 161, the standard SNMP port. Such a tool makes it possible to simply add the MIB Host Resources tree into your favorite snmp daemon program (commercial or non-commercial) already running on your machine. Very few of these standard snmp daemons support this host resources branch of variables. With this program, you can have the best of both worlds - the resources tree and the enterprise specific ones.

    Although the development of this product has only been less than a month, I believe that the software has great potential. I have made new friends in the SNMP / Scotty community - a pity this software can't be released as freeware (property of University of Tromsoe).

    So, what we have here is an agent. Being an agent, it's very dumb by definition. It only answers questions raised. Listening to the thirty-nineth IETF meeting held in Munich, Germany, 11-15 August 1997 (MBone broadcast), they briefly discuss the SNMP v3 - they are talking about restructuring the communication architecture in SNMP.
    When the v3 goes into general use in network monitoring programs, putting more "smarts" into the agent remains to be seen. Also, moving this snmp from being just an extension to a full-blown snmp agent could be one interesting thing to do in the future.

    Finally, as a last minute note, please read the ChangeLog below. It contains a lot of information about Snmpd.

5.0 ChangeLog

    version 0.0.1-0.0.17:
      • Initial internal versions.
       
    version 0.0.18:
      • Removed terrible memory leak. The program crashed with a 60 MB core after 12 hours. The problem was in fact a memory leak in Scotty itself (v2.1.6). After a 2-day discussion with the developers of Scotty, we came to a workaround solution. The bug will officially go away in Scotty v2.2.0 in the future. Warm greetings to my friends developing Scotty: Juergen Schoenwaelder, Martin Knoll and Buz Owen.
      • Added support for IRIX max processes.
      • Restructured agent engine with smart updates to avoid CPU waste. This also includes a cleanup system when e.g. processes are removed from the process list.
       
    version 0.0.19:

        Bugfix: no variables could be accessed while the system was updating.

        To ensure reliability of the program, a restart option timer has been added. This way, all forthcoming memory leaks etc. are flushed e.g. every 5 days. The agent kills itself and restarts after 5 seconds.

      • Automated installation procedure with self-made Bourne shell script. It checks for necessary files, appropriate chmods, finds the scotty executable in order to update the Tcl script with its path and compiles the low-level C program. Platform dependent stuff are taken into consideration by the script.
     
    version 0.0.20:
      • Bugfix: Agent didn't update the list of installed programs when things where updated in the Store system.
      • Bugfix: MaxProcesses for the IRIX version needs a new program called systune (in order to give away the correct number :). Setup program adjusted to check for systune's existence and correct attributes.
      • Bugfix: Process memory shown was only physical memory usage. Fixed now under IRIX and HPUX.

        It is no longer necessary to use the external program ps under HP-UX. I wrote my own little ps program :)

       
    version 0.0.21:
      • Added support for SNMP Set operation.
      • All Set operations are remembered as the agents restarts by saving changes to file.
      • The list of variables which is set is regenerated after a configuration specific number of set operations.
      • Distribution structured into folders (doc, lib, src, etc etc.).
      • Configuration script prepped for the upcoming Linux support.
      • Better error handling if e.g. config file is missing.
       
    version 0.0.22:
      • Linux platform support introduced. Adjusted configure script, makefile and snmpd. Device scanning is not supported yet in Linux, though.
      • First step towards using a safe interpreter in Tcl has been made.
       
    version 0.0.23:
      • Snmpd now complains to a log file (etc/snmpd.log) and syslog. The latter entry is dependent of the type of error - a serious one is definitely placed in the syslog ;-) The advantage with this is that snmpd is silent and thus appropriate to be a background process.
      • Bugaware: the safe interpreter broke the snmp engine. It is disabled till further notice.
      • Bugfix: the safe interpreter has been given a background error (bgerror) checking. This way, crashes aren't silent anylonger.
       
    version 0.0.24:
      • Bugfix: Updated process performance (memory and cpu usage) only when requesting process overview (not the intended process performance table).
      • Removed annoying "unsafe Tcl interpreter" warning message.
       
    version 0.0.25:
      • Added support for hrSWRunName (process table).
      • Added support for hrSWRunParams (process table).
      • Added support for hrInstalledLastChange (installed table). This variable contains the time since a new program was installed on your machine (i.e. store was last updated).
      • Small optimizations on the processes info retrieval system.
      • Added variable in config: specify where snmpd logfile should be placed.
      • Added variable in config: specify the level of snmpd logging (3 levels).
      • Added variable in config: specify the level of syslog logging (3 levels).
       
    version 1.0.1 PR1:
      • Implemented port multiplexor program for port 161 (the standard snmp port). This way, you can easily incorporate this snmp agent with the one shipped with your computer (enterprise version). One just has to configure the standard snmp agent to run on a different port than 161, specify the port settings in this programs configuration file and run lib/snmpd-mux program with root privileges. The multiplexor is utterly simple: it receives requests (set or get-next etc.) on port 161, studies the package, if the related variable in the request is for the Host Resources tree then the request is routed to this agent (otherwise, it routes the stuff directly to the standard snmp agent), it receives the answer from the agent receiving the routed message and ships it off to the originator. The program uses the snmpd configuration file, snmpd.conf. I programmed this in C to gain high performance.
      • Bugfix: there was a bug in GetProcesses. Removed.
      • Updated configure program to deal with the new C program, snmpd-mux.
      • Removed old port value in the configuration file. Added two new port settings for this agent and the other (potential) original snmp agent.
      • Added detection of the xfs file system (enhanced 64 bit guaranteed constant stream file system).
      • Unsafe scotty interpreter problem has been discussed with Juergen (Scotty developer). The conclusion was that there will be a much easier scheme in Tcl v8.0 when it's released. The next Scotty series, v2.2.x, will have a completely different way of handling safe Tcl interpreters. Since I'm only using only one Tcl script which is trusted, patching the stuff to work with the v2.1.x series of Scotty is, according to Juergen and I, hours not very well spent. Conclusion: code in a holding pattern will Tcl 8.0 is here.
      • Documentation is now updated.

    version 1.0.1 PR2:

      • Rewrote the logging procedure to handle 4 message classification levels (one new added). Removed one bug that make it impossible to only log to the syslog file.
      • Bugfix: When the snmpd received a get-request instead of get-next-request, the engine didn't update the variables. Fixed.
      • Bugfix: The SNMP port multiplexor (snmpd-mux.c) didn't work with two of the SNMP operations (get-request and set). A better analysis of incoming requests has been provided solving this problem.
      • Bugfix: When the agent restarted itself, there was a tiny intermediate file forgotten in the cleanup process. Fixed.

    version 1.0.1 Release:

      • First official release.

    version 1.0.2 Release:

      • Annual update to work with RedHat and new Scotty.

    version 1.0.3/1.0.4 Release:

      • Annual update to work with RedHat, Solaris and new Scotty.

6.0 TODO

      Include support for more variables. Especially the partition tables, hard drives etc.

      Processors load support is not implemented for IRIX. Unfortunately, it would appear to me that SGI hasn't made this information accessible using pfStats ("The PFSTATSHW_CPU pfStats statistics class for accumulation of statistics on CPU usage are not yet implemented for IRIX6.x operation (64bit or 32bit)." as it says in one of their release notes.)

      MaxProcesses, MemorySize and device scanning hasn't been implemented for SunOS or NetBSD.

      Incorporate safe Tcl interpreter with Tcl v8.0 when released.

      Add traps to the agent.

7.0 Source code

    This software is property of the University of Tromsoe. Luckily, the software has been made available for download through the GNU license. And best of all, it's for free!

    Go to the download section to start downloading.



Sveinar Rasmussen (web)