Exigen monitoring system configuration, section II.
Cisco repository (CCR)  Configuration.

 

Stage I  (system configuration) must be completed, system have correct name, IP, dns resolvers are configured, e-mail is configured; you have user account and account in ‘snmpstat’ system.

 

Cisco repository system (CCR) provides tftp service (both Upload and Download) for Cisco routers (switches, firewalls and so on), automate configuration downloading, saves configurations into the CVS, track configuration changes, provides read access to configurations by the WEB.

 

It supports cisco routers with IOS, Catalist switches with IOS and CatSys, PIX firewalls, Cisco VPN servers. In some cases, small adjustment may be required for the system scripts, or you should set up correct prompts on the router.

 

For simplicity, we will use ‘router’ below, which can be replaced by switch or firewall.

 

Table of content:

 

1. Prerequisites: 1

2. Selecting login policy and configuring access login and password. 2

3. Adding password information into the CCR. 2

4. Edit default access configuration. 4

5. Adding new router into CCR system. 4

6. Saving  router configuration in CCR: 6

7. Using CCR for configuration tracking and making updates. 8

APPENDIX I. Internal CCR structure. 8

APPENDIX II. Using CCR for router/switch/firewall maintanance. 10

 

 

1. Prerequisites:

  1. Select router, switch or firewall for pilot configuration;
  2. Login onto monitoring system and verify, that you can:
    1. login onto this router, using telnet or slogin.
    2. Can login into the router, using user’s login and password (if you do not use user name, it’s OK);
    3. Can run ‘enable’ (know password).
  3. Be sure, that you allow tftp FROM router to the server. (Notice – you can configure SRC address for TFTP sessions on the router, if required). Be sure, that you know, how (which IP) to access server from the router (in case of NAT, address may be different from internal IP address).
  4. Be sure, that you can resolve router by its name (ping router-name). Add all necessary domains into the /etc/resolv.conf. If you do not control your DNS system, you can describe routers in ‘/etc/hosts’ file (not recommended).

    We recommend creating additional domain and putting all router, firewall and switch names there. Such as ‘net.company.com’.

 

 

See an example of test session below:

 

alex@amur.amc.portera.com 101>telnet SJCTRAIN-SW-1

Trying 10.23.4.5...

Connected to SJCTRAIN-SW-1.net.exigengroup.com.

Escape character is '^]'.

 

 

Cisco Systems Console

 

 

 

 

Username: monitor

 

Password:

SJCTRAIN-SW-1> enable

 

Enter password:

SJCTRAIN-SW-1> (enable) ping 10.200.6.56

10.200.6.56 is alive

SJCTRAIN-SW-1> (enable)

 

Notice: current version of CCR (1.1) does not allow configuring IP address for the router – name must resolve into the address. It should be fixed in next releases.

 

2. Selecting login policy and configuring access login and password.

CCR requires user login, user password, and enable password to get access to the router and save configuration (and to acquire information in informational commands). Keeping passwords in clear text is not a good idea. So, CCR offer a few ways to get this information.

 

  1. Logins and Passwords can be encrypted, using public key, and CCR uses private key to decrypt them. This key is available to ‘root’ only, and stored in slightly separate place. This is Public Key method or RSA method (because it uses RSA public/private keys). It allows to read configurations automatically, do not require any password to be entered when you save configurations,  but allows ‘root’ to retrieve passwords from the file.  Recommended for environment with a limited number of server users and high usage of automated saving and scripting.

 

  1. Logins and Passwords can be encrypted by 3-DES, using passphrase as a password. This is PassPhrase method. It required to know passphrase for saving configurations, but it allows to have different logins and passwords for different routers, do not require knowing of the individual logins and passwords for saving configuration (which improve security by reducing number of people, knowing real enable password), and still allow running automated scripts (but to do it, password must be saved in the root-access-only file). Method is recommended in most cases, when PublicKey cannot be used (if you do not configure daily SAVE_ALL scripting, it is 100% safe- root user can not retrieve passwords.

 

  1. Operator enters login and passwords every time, when he want to save configurations (on the ‘Save’ form). It is a ‘back-up’ to other 2 methods, and can be used as the only access method in some environment (but it will require everyone, who can initiate configuration saving, to know all passwords). This is manual login configuration method.

 

You must select one of these access methods. For testing, we recommend to start with method 1 – Public Key.

Notice: no matter, which method do you use, skilled root user can always learn passwords by changing CCR scripts and waiting for the next system usage by operator. So, it is much more important to restrict access to the management server (esp. root access) and run good IDS system, than to choose Manual or Passphrase method.

 

PeblicKey(RSA) and PassPhrase logins are stored in passwd files in encrypted form. Default is usually  ‘DEFAULT.pwd’. When you add / change a router, you select one of this files and assign it to this router. The same files can be used for both (1) and (2) methods, CCR distinguish PublicKey and PassPhrase methods by file format.

 

There is not any way to see content of this file, using web. Network operators can use password files or change them, but can not see user names and passwords. CCR never shows such information.

Local unix users:  System root can decrypt password file, only if he knows passphrase (for method 2) or public key location (for method 1). Non root users can not read password files at all.

3. Adding password information into the CCR.

Before adding a router, it is better to create authernication (Password) file for it. Many routers can share one password file, if they have the same authentication source (the same login names and passwords). On the other hand, you can create and use as much this files as you wish. Skip this step, if you wish to use method (3) /manual login configuration/ for the routers. You can do this step AFTER creating router, using method (3) for the first tests.

 

Open CCR window. You will see something like this (with 1 – 2 test objects instead of a real list of routers from my example):

 

 

To create new password file (or edit existing one), click on Edit password file button (top of CCR screen):

 

 

 

Now, fill in input fields:

-        Check Create checkbox;

-        Enter file name (without suffix);

-        Enter passphrase (2 times, must match) , if you wish to use PassPhrase method (it unchecks RSA key checkbox automatically); do not change this line for PublicKey (RSA)  method;

-        Enter user name. Can be skipped, if your router do not ask user name;

-        Enter user password (2 times, must match);

-        Enter enable password (2 times, must match).

-        Click APPLY.

-        Verify results (see below) and click RETURN.

 

 

To change existing file, select it in selection list (and do not check Create).

 

To delete existing file, select it, check Confirm delete checkbox, and click DELETE.

Remember – you cannot see content of password file, you can only replace it or delete it.

 

4. Edit default access configuration.

You must edit default settings to adjust server’s IP address, default access protocol and so on. Open CCR window and click on EDIT button in the Default Setting line:

 

Change TFTP server IP to IP address of your server. Set up default password file (you can change it when configure individual routers). Set up default access protocol. Click on APPLY. Do not delete .default object.

5. Adding new router into CCR system.

 

You can add devices in batch mode, using unix script on the server. It can be used to make initial configuration in the big network. This method is not described here.

 

Now, you can add new router. Open CCR screen, and click on Add new device into depository (to add new router) or on EDIT (to edit / delete router):

 

 

Fill in, at least, fields marked by ‘*’:

-        Device Name – device name (1 word), which should be resolvable into IP address (this limitation will be withdrawn in next release);

-        IP Address – this is SRC address of TFTP packets FROM device to the server (in future, one of this addresses should be used for device access);

 

Other fields:

-        protocol – telnet or slogin. CCR can try both, so you can keep it empty;

-        TFTP server IP – destination IP for this server, relative to the device (router). In most cases, it is server’s IP; but, in case of NAT translation, it can be translated server’s address. Keep it empty, if it is server’s address.

-        Password file – reference to the file, configured in previous step (on of such files).

-        Additional options – used to enter non-standard options (for example, NO-SAVE=”1” means ‘do not save configuration for this device – for example, it is saved manually).

 

Now, click ‘APPLY’. You will see a new router on main CCR window.

 

To edit router parameters, clock on EDIT. For example:

 

Change parameters (to delete IP address, check Del box) and APPLY;  to delete it, check Confirm delete and click on APPLY.

 

6. Saving  router configuration in CCR:

To check configuration, try to save configuration into CCR. Open CCR window, check Select box for new device, and click on Read Configuration… button. You will see next window:

-        Enter comment, it is used for CVS and is mandatory (CCR 1.1 does not place this comment into the CVS due to the bug);

-        If you use PublicKey method, click Start Download;

-        If you use passphrase or mixed mode (some devices use PublicKey and others uses passphrase), enter passphrase and click Start Download;

-        If you use manual login configuration, enter user, password and enable information and click Start Download.

 

If everything is correct, you will see a protocol like this (here it is for 1 device):

 

Saving configuration from SJCTRAIN-SW-2 (reason: xxx)







 
|/var/CISCO/bin/SAVE_CONFIG  -m 'xxx' -p SJCTRAIN-SW-2
 
spawn telnet SJCTRAIN-SW-2
Trying 10.23.1.6...
Connected to SJCTRAIN-SW-2.net.exigengroup.com.
Escape character is '^]'.
 SJCTRAIN-SW-2 
 
User Access Verification
 
Username: Kerberos:     No default realm defined for Kerberos!
monitor
Password: 
 
SJCTRAIN-SW-2>enable
Password: 
SJCTRAIN-SW-2#
SJCTRAIN-SW-2#write net
Remote host []? 10.200.6.56
Name of configuration file to write [sjctrain-sw-2-confg]? .running.SJCTRAIN-SW-2.conf
Write file .running.SJCTRAIN-SW-2.conf on host 10.200.6.56? [confirm]
Building configuration...
 
Writing .running.SJCTRAIN-SW-2.conf !! [OK]
SJCTRAIN-SW-2#
 
*** SUCCESS
Saving CISCO / PIX / SWITCH configuration






 
Saving config for SJCTRAIN-SW-2
FIle name to open=/var/CISCO/tftpboot/hosts/.default/.conf/options.conf
FIle name to open=/var/CISCO/tftpboot/hosts/SJCTRAIN-SW-2/.conf/options.conf
Using the following options:
TFTP_SERVER_IP=10.200.6.56
LOGIN_PROTO=telnet
PWD_FILE=DEFAULT
cmd=|/usr/local/bin/sudo /var/CISCO/etc/PROD-SAVE.cwd -host SJCTRAIN-SW-2 -ip 10.200.6.56 -enc_pass /var/CISCO/etc/.conf/DEFAULT.inf -proto telnet  -p
ph=*********
chmod 0640 SJCTRAIN-SW-2/SJCTRAIN-SW-2.conf
chown monitor,monitor SJCTRAIN-SW-2/SJCTRAIN-SW-2.conf
unlink SJCTRAIN-SW-2/.running.SJCTRAIN-SW-2.conf
       
       

Return

CVS:

SJCTRAIN-SW-2 

 
 
 

Unfortunately, it is unlikely that everything works well from very first attempt, so you will see, more likely, something like this (passphrase was not correct):

 

var/CISCO/bin/SAVE_CONFIG  -m 'xx' SJCTRAIN-SW-2
spawn telnet SJCTRAIN-SW-2
Trying 10.23.1.6...
Connected to SJCTRAIN-SW-2.net.exigengroup.com.
Escape character is '^]'.
 SJCTRAIN-SW-2 
 
User Access Verification
 
Username: 
 
 
*** FALURE - USER name was not defined, exitting...
Saving CISCO / PIX / SWITCH configuration






 
Saving config for SJCTRAIN-SW-2
FIle name to open=/var/CISCO/tftpboot/hosts/.default/.conf/options.conf
FIle name to open=/var/CISCO/tftpboot/hosts/SJCTRAIN-SW-2/.conf/options.conf
Using the following options:
TFTP_SERVER_IP=10.200.6.56
LOGIN_PROTO=telnet
cmd=|/usr/local/bin/sudo /var/CISCO/etc/PROD-SAVE.cwd -host SJCTRAIN-SW-2 -ip 10.200.6.56 -enc_pass /var/CISCO/etc/.conf/PWD.IMAX.inf -proto telnet  
ph=*********
chmod 0640 SJCTRAIN-SW-2/SJCTRAIN-SW-2.conf
chown monitor,monitor SJCTRAIN-SW-2/SJCTRAIN-SW-2.conf
unlink SJCTRAIN-SW-2/.running.SJCTRAIN-SW-2.conf
       
       

Return

CVS:

SJCTRAIN-SW-2 

 
 

In this case, you should verify configuration download manually. Step by step:

-        Login onto the server;

-        Telnet or slogin into router, using its name as you configured in CCR;

-        Enter user name and password as you configured;

-        Enter enable as you configured;

-        Save configuration onto the server, using it’s IP address as you configured it. Use file name SERVER_NAME.conf (for example, SJCTRAIN-SW-2.conf for SJCTRAIN-SW-2 switch).

 

Possible reasons of failure:

-        You cannot login into router (it is easy to troubleshoot). Try method (3) to verify, is a problem in your password file or somewhere else.

-        You configured very non-standard prompt on the router – each prompt must end up with ‘$’ or ‘#’ or ‘>’ symbol.

-        Router have not TFTP access to the server (check firewall logs);

-        You used wrong IP address, when configured access (do not forget – you can configure few IP addresses for 1 router). To verify it, look into tftp logs, or connect back from router to the server (using, for example, telnet server 22) and determine IP address, running netstat –n command.

-        Something wrong with permissions or configuration:

a.      Go to /var/CISCO/tftpboot / directory;

b.     See, if directory exists with name = IP address of your router;

c.      Go to this directory (it will be symlink) and verify, that it has configuration file (empty on this stage) with name router_name.conf (where router_name is name of router), and this file is open for writing.

 

 

 

7. Using CCR for configuration tracking and making updates.

CCR allows both, configuration saving and browsing, and router updates. It provides rad and write TFTP service for the routers. Every router has it’s own directory (which is determined by IP addresses). Directory /var/CISCO/tftpboots contains symlinks (with names equal to IP addresses) to real directories, located at /var/CISCO/tftpbot/hosts. Every directory contains configuration file, with name the same as router_name.conf (it prevents possible errors in IP addresses – if IP address is wrong, tftp will not find such file and will not write out configuration), change history directory, and placeholder for update file. Configurations can be seen by CVSWEB – just open CCR window and click on cvs link:

 

To upload anything to the router, place it into the same directory and use tftp with a simple file name (without any directories and without / prefix). We recommend to use SAMBA for access to this directories – just create a symlink in your home directory, and configure SAMBA.

 

APPENDIX I. Internal CCR structure.

CCR is located in directory /var/CISCO, and uses standard tftp server of FreeBSD, running it as:

 
tftp   dgram   udp     wait    root    /usr/libexec/tftpd      tftpd –s        /var/CISCO/tftpboot -c -l -u monitor
 
 

In this mode, tftp change directory to /var/CISCO/tftpboot/SRC_IP every time, when it receives packet frim SRC_IP source, so limiting every router to it’s own directory.

 

On the other hand, CCR interacts with HTTP server on port 8100, which is running as monitor, so most files are owned by monitor:monitor user.

 

File structure:

 

-        /var/CISCO/bin :

-        SAVE_ALL – script, which saves all configurations daily and sends change report;

-        SAVE_CONFIG list_of_routers – saves configuration for this routers;

-        create_host.sh – allows to create hosts in batch mode;

 

All other scripts are used to collect different information and are not described here (they are experimental for now).

-        /var/CISCO/etc

a.      PROD-SAVE.cwd – main send/expect script;

b.     RUN-cmd – read only send/expect script (it uses DEFAULT password file always);

c.      .conf – password configurations (directory).

-        /var/CISCO/cvs – cvs repository for the system;

-        /var/CISCO/tftpboot – base directory for tftp.

 

 

/var/CISCO/tftpboot/hosts have a few different objects inside:

-        ALL-logs – directory for daily logs;

-        .default – pseudo – router which keeps defaults for other hosts;

-        router_name – depository for every router.

 

For every router, system creates in /var/CISCO/tftpboot:

-        hosts/router_name – home directory for tftp, which contains all configurations for this router and is a root’ when you makes tftp transactions;

-        IP_ADDRESS – symlinks to the home directory, which maps IP address and router directory together.

-        hosts/router_name/.conf/options.conf  - router access options;

-        hosts/router_name/.status – status of last configuration download.

-        hosts/router_name/router_name.conf – configuration fil, used for downloading and containing last saved configuration.

 

It is very important, that all directories under tftpboot , etc/.conf and cvs are owned by monitor:monitor, to allow CGI scripts work as a monitor.

 

There are 2 files, which contain critical security information and must be root only readable:

-        /var/CISCO/etc/.conf/rsa_priv – contain private key for password-less access in case, if you use PublicKey password storage method.

-        /var/CISCO/etc/.conf/.pf – this file can be created and contain passphrase, if you want to use passphrase password encryption but run automated daily scripts (SAVE_ALL).

 

System uses sudo to run send-expect script as a root:

Cmnd_Alias SAVE_CMD=/var/CISCO/etc/PROD-SAVE.cwd

User_Alias SAVE_OPER=monitor

SAVE_OPER               ALL_PROD=(root) NOPASSWD: SAVE_CMD

 

And this script (and ONLY this script) have access to .pf file or to private key.

 

Main command is SAVE_CONFIG [options] [list_of_routers] and Web operation Save configurations into repository.

 

How it works (all names are relative to /var/CISCO):

-        For every router, read default options from tftpboot/hosts/.default/.conf

-        Then, read router specific options from tftpboot/hosts/router_name/.conf/options.conf

-        Run Send/Expect script, which decrypts password file (if necessary), logins to the router and request saving configuration onto the network;

-        TFTP daemon allows saving configuration ONLY into the router specific directory (due to IP_ADDRESS symlink) and only into the file router_name.conf.

-        After configuration is saved, system remove some variable lines from it (such as timer clocks, which changes during NTP adjustment) and open passwords (for Cisco VPN server, for example);

-        Resulting file is compared with previous configuration. If configuration changed, diff is added into the log file (into both ‘local’ logs for this router and daily logs for all routers) and saved into the CVS.
Notice: only cvs/CVS, tftpboot/CVS, tftpboot/hosts/CVS, and tftpboot/router_name/CVS directories rely to the local CVS repository; other CVS directories refer to the sources of the system and can be absent (depending on installation method).

 

 

APPENDIX II. Using CCR for router/switch/firewall maintanance.

 

1) Save configurations after any change:

-        open CCR;

-        select devices;

-        click on Save configurations… button;

-        Review save protocol and check status (on the first page).

 

2) review configurations and configuration changes:

-        open CCR;

-        Click on cvs link for reviewed device;

-        Select colored diff and review changes, for example:

 

 

3) Set up daily change reports:

-        set up PublicKey access method or write out etc/.conf/.pf (root access only) file with passphrase;

-        Add /var/CISCO/bin/SAVE_ALL into the cron for user monitor (recommended time: 23:30);

-        Create local alias, net-change, on your system. Script sends change repirts to local user net-change.

 

4) Update IOS on device:

-        copy (link, or symlink) new IOS , Pix-Os or CatalistOs onto the device directory;

-        Login into device (router, firewall…) Download new OS onto device, using it’s name without any prefixes, following standard upgrade procedure for this device.

 

5) Apply changes to device configuration from the file:

-        Create file with update (we recommend to name it update.conf);

-        Login to device and load this file (using conf net, for example);

-        After completing, rename update.conf file or delete it (to exclude possible mistakes with this file).

 

 

6) Restoring old configuration in case of emergency.

-        Login onto the server; change user to ‘monitor’ (using ‘sudo’);

-        cd to device directory (for example, /var/CISCO/tftpboot/hosts/EQX-1a);

-        Using WEB interface, find configuration version you’d like to restore;

-        extact this version, using ‘cvs update –r VERSION_NUMBER’;

-        Download this configuration into the device. Remember, that the only way to make exact restoration is to conf over and then reload (reboot device), simple conf net add new configuration to the old one.

-        After restoring configuration, update file to delete sticked version:
cvs update –r 1

-        Now, save configuration again using WEB interface or
SAVE_CONFIG router_name

 

It is little tricky, because, when you restore old version, cvs stick version number to the file and forbid file changes until you update it back to the main branch (-r 1).