Introduction into universal monitoring systems

Table of content

I. FreeBSD Operating system, specifics and administration. 1

FreeBSD 4.x introduction. 1

FreeBSD 4.x administration – important tips and advices. 2

II. SNMPSTAT software. 4

Introduction. 4

APPENDIX I. File’s location, configurations, specifics. 10

I. FreeBSD Operating system, specifics and administration

This document discuss ‘Universal monitoring and management’ system, built on FreeBSD OS. Such system can be built on any other opensource Posix (Unix like) OS, such as Linux or Solaris.

FreeBSD 4.x introduction.

FreeBSD 4.x (4.7 – 4.9) OS is used as a basic OS for universal monitoring / management servers. This selection was based on both, historical and technical reasons. Technical reasons for such choice are:

- Stability (system is a very stable server platform);

- Easy to maintain open source software – system have excellent ‘/usr/ports’ system which automate software extracting, patching, compiling and installation for 95% of all open source products.

- Security (with or without secure level).

- FreeBSD 4 is most reliable branch in FreeBSD family (new FreeBSD 5 became release-grade only about 1 year ago).

FreeBSD is classical BSD OS, enhanced to satisfy modern requirements. It is ahead of Linux in some things (ports system, jail, secure-level) and behind in others (set of hardware, software raid, YaST2 for SuSe Linux, thread system is not so debugged as in Linux or Solaris, commercial support is not common for FreeBSD). FreeBSD uses it’s own binary file format, but have linux compatibility module, which allows running most Linux software.

FreeBSD differ from classical unixes in a few BASIC things:

- It used db files for users (never edit /etc/passwd);

- It does not have classical /etc/rc.d system, but use /usr/local/etc/rc.d for addition of the services;

- It has 2 installation systems – packages (binary) which can be installed thru sysinstall, and port system, which uses make to extract, patch, compile and install software.

- It has a very wide set of installation FTP servers over the world (to install FreeBSD, you can bring 2 FD and have a network card – it is enough);

- It has boot-time configuration facility (boot –c), and require kernel recompilation (simple ‘make’) to add/delete some drivers (or multi-CPU mode, for example).

- It has both, ‘crontab’ based cron tables, and root table ‘/etc/crontab’.

In other ways, it is classical BSD Unix for i386:

- Generic kernel can run on wide variety of servers;

- System have /proc file system;

- System have both row and block devices;

- System use classical (not stream type_ device drivers;

- Command set is BSD derived, some commands have BSD syntax instead of System-V syntax (ps; netstat; df).

- File system – standard bsd 4.2 file system, with standard fsck. In most cases, recover itself after power failure.

FreeBSD 4.x administration – important tips and advices.

Universal monitoring systems are configured to use WEBMIN (web based GUI) for standard administrative tasks. SSH and command line tools can be used for other administrative tasks. Many things are doing by system installation tool.

Webmin is located at port 8101, protocol https. To reach it, just open

https://server:8101/ . Use ‘root’ account to have full privilege.

System installation tool - /stand/sysinstall – can be used for network changes, package installation, standard components installation. It can create users and groups, but is not optimal for this.

Tasks and tools:

• User and group administration – use webmin. If you cannot, run ‘vipw’ as a root. If you cannot, edit /etc/master.passwd and run pwd_mkdb /etc/master.passwd.
• System configuration, network interfaces, standard services – use ‘/stand/sysinstall’. Edit /etc/rc.conf if you want to change IP address, interface parameters, configure named or other services (but try /stand/sysinstall -> Configure first). You can use webmin for some of these tasks, but it is not recommended.
• Standard services - /etc/inetd.conf (if you did not installed xinetd). You can use /stand/sysinstall or just edit this file and restart inetd.
• Package lists – pkg_info. It reports both, packages and ports.
• Binary packages – install only for very standard services or for the services which are not primary for your system (X11 for example). Use /stand/sysinstall -> Configure -> Install packages .
• Source based ports – use this system for all software, which is intended to be used in production, except few systems which was installed from very beginning (ssh , sendmail, ntpd). This is a file tree of the ports, located at /usr/ports. To install port (_FOLDER_ and SYSTEM are subdirectory and directory names):
  cd /usr/ports/_FOLDER_/_system_
  make
  make install
  
  You can find full description of ports on www.freebsd.org. In port directory, files pkg_descr contains short description of the system. Port system depends of the sources, so sometimes you will be asked to register and download software before installation (example – SUN Java for Linux).
• Add on startup scripts are located at /usr/local/etc/rc.d directory and must have a suffix .sh . They have start | stop argument, and always are called using full name (‘sh /usr/local/etc/rc.d/samba.sh’, not ‘cd /usr/local/etc/rc.d; ./samba.sh’). Most ports and many packages install sample scripts into this directory, which must be copied without ‘sample’ suffix and edited if necessary.
• Duplex configuration – in most cases, configured by interface options. To configure them, use /etc/rc.conf variables, for example:
  ifconfig_xl0="inet 192.168.11.21 netmask 255.255.255.0 media 100baseTX mediaopt full-duplex up"
• Port system install software into /usr/local ; standard software (including packages) are installed into /usr. Sources (including kernel) are in /usr/src for standard software, and in …/_port_/work in port system (_port_ is port directory).
• System documentation is available as:
• In webmin;
• On the basic web page (localhost:80) as both, handbook and set of port documentation;
• As ‘man command’;
• On the http://www.freebsd.org.

Other files:

• Apache configuration (port 80): /usr/local/etc/apache
• Home page, port 80: /usr/local/www/data
• Installed ports documentation: /usr/local/share/doc

II. SNMPSTAT software

SNMPSTAT software is integrated by the web system, port 8100 (https). It uses it’s own authentication, administrative system, set of groups. Users are not synchronized with system users (and we expect that system users will include only those who require direct access to the system).

Introduction.

Home page:

Subsystems:

• Snmpstat system – Routers/Switches/Firewalls snmp monitoring. Configured by conf file, using web interface. Includes:
• Visualization system, which shows all network, or part of network on the screen, including routers / links status and traffic. Can show about 300 objects on the standard screen in ‘FULL’ mode, can easily work with 500 – 800 objects (routers, switches, firewalls, interfaces, ports).
• Detailed router statistics - includes performance charts, reliability reports and so on.
• Detailed link/port statistics – includes performance charts, reliability and utilization reports and so on;
• Alert system, which sends alerts (by e-mail); it interacts with ticket system;
• Ticket system – allows masquerading failures temporary or permanently, changing status (for example, we set up encrypted tunnels to show failure if system detects warning state), write comments and so on.
• Log system interacts with tickets and alerts and allows to see all events on per-object basis or on per-day basis, write notifications, make comments and so on.
• Report subsystem, which automatically maintains usage and reliability, reports for your system.
• Expiration subsystem, which generalize information so that you can always check link utilization in history, but lost details in time (for example, have only per – day average for 3 years ago data).
• Example:
  
• CCR – Cisco Configuration Repository. System provides tftp service for Cisco devices, collects configurations (manually or automatically), save them in CVS, sends change reports, and allows reviewing changes, simplifying OS upgrades and configuration changes. Example:
  
  
  AND
  
• ProBIND DNS control system – allows controlling few Unix DNS servers, for a few different DNS name spaces. Simplify IP address allocation. Exclude DNS name space inconsistency. Allows building mixture DNS zones (DNS by bind, Windows domain by active directory). Just example of the screen below:
  
• Private documentation (mirrored from amur);
• System inventory – database system, which allows to view and edit server placements, rack views and so on. Below is an example – view of our production racks, front view, second page:
  

APPENDIX I. File’s location, configurations, specifics.

Operating system:

Users: /etc/master.passwd, cmd: vipw, pwd_mkdb

Startup configuration: /etc/rc.conf

/usr/local/etc/rc.d/*.sh

Kernel: /usr/src/sys, see ‘man config’ and handbook.

Cron tables:

/etc/crontab – classical crontab

crontab - controls per user cron tables.

Daily, weekly, monthly jobs, including sanity checks:

/etc/periodic/

/etc/defaults/periodic.conf

/etc/periodic.conf

man periodic

Standard web (port 80):

Home: /usr/local/www/data

Configs: /usr/local/etc/apache

Mail system:

Type: sendmail

Installed from: system (not ports)

Directory: /etc/mail

Database:

MySQL data base;

Use ‘webmin’ to administrate .

MySQL differ ‘locahost’ access and ‘All’ access. You can install and use Windows MySQL client instead (very good system).

Files: /var/db/mysql

Snmpstatd:

Directory: /p/stat

Conf. Files: /p/stat/Poll.conf, /p/stat/WWW-local.conf

Global config: /p/stat/WWW/bin/build_lib.conf

Start cmd: /p/stat/bin/START

Docs: /p/stat/WWW/DOC (available by web).

Snmpstat users and groups:

Directory: /p/stat/httpd/PWD

Users: /p/stat/httpd/PWD/pwd

Groups: /p/stat/httpd/PWD/group

Snmpstat web (port 8100):

Dir /p/stat/httpd

Home: /p/stat/httpd/home

Main menu: /p/stat/httpd/home/index2.html

Conf: /p/stat/httpd/conf/*

Commands (restart etc): /p/stat/httpd/bin

CCR (Cisco Configuration Repository):

Directory: /var/CISCO

Repository: /var/CISCO/tftpboot/hosts

CVS home: /var/CISCO/cvs

Commands: /var/CISCO/bin

Inventory:

Home: /p/stat/DB

Config: /p/stat/DB/config.inc

ProBIND:

Home: /var/PROBIND/extdns

Conf: /var/PROBIND/extdns/inc/config.inc

Tacacs plus:

Templates: /p/stat/tac_plus/tac_plus.tmpl

Group: tacacs

Directory: /usr/local/etc/tac_plus

Service ports:

sshd: 22

www: 80 (open documents)

webmin: 8101 https

snmpstat system: 8100 https

Installation requirements:

- For snmpstatd system: SNMP access required to all network devices;

- For CCR system: ssh / telnet access required to all network devices; tftp access required FROM all network devices;

- Recommended: SYSLOG access FROM all network devices.

- Recommended: static NAT translation for this host, WITHOUT any service available from outside. tftp is harmless, because it is used to fill in CVS repository only so any frauded data (even if possible) will be detected in a few minutes by cvsdiff.

- For operators: access to port 8100 required.

- For system administrators: access to ports 22 and 8101 required. We recommend access to port 8100 from all operator’s desktops and dialin pptp (you can not make active operations, using this port, except ProBIND), and access to ports 22 and 8100 limited to your sysadmins.

- Configure /root/.forward so that you receive root’s mail. It contains daily, weekly and monthly sanity checks and security reports (including reports about user list changes etc).