README for ISconf Version 2i Localized for: acme.net See LICENSE in this directory See http://www.infrastructures.org for some background information about the files and scripts in this directory tree. If you found this README in /var/isconf on any of the machines in your infrastructure, then read this entire document before you try to administer any of those machines. Found this README in a tarball or on the net? Not running this code within your infrastructure yet? Then you're spending way too much time doing systems administration -- probably 2 to 4 times too much. Skip down to "How to install ISconf". DO NOT EDIT the contents of /var/isconf on any machine. These files are mastered from the CVS respository at cvs:/cvsroot, and checked out into images:/images/acme.net to make them available to the anonymous rsync server. All other images and install packages are NFS mounted from isconf:/images. See "Description of key hosts" below for more detail. Note that the entire /var/isconf directory tree is refreshed via rsync from the repository at each rc.isconf run -- all changes need to be made there. The whole process is kicked off by '/etc/rc.isconf start'. This calls /var/isconf/bin/isconf, which then calls 'make', using the makefiles in /var/isconf/conf/*.mk. See 'Description of key files" below for more detail. If you need to make a change to any host, read this entire document several times, go study Infrastructures.Org, join the infrastructures mailing list, and *then* follow the instructions in the "How to make a change" section below. Description of key terms: NOTE: See Infrastructures.Org for more discussion of each of these terms. Enterprise Cluster (EC) The collection of hosts that makes up a distinct function, and which you want to manage as a single unit. You might have a separate EC for dev, one for test, one for stage, and one for prod, for instance. You might put all the hosts for one department or division into an EC. It's possible, but rare, to put all the hosts for an entire company in a single EC. The key thing to ensure is that you can give each EC its own DNS subdomain -- see below. You can nest EC's inside of each other, using N-level subdomains. Note that an EC is *not* the same thing as an HA or parallel computing cluster, but you can put one or more HA or parallel clusters inside of an EC. Gold Server The main administrative machine for an EC. Usually the isconf, rsync, images, and cvs server all rolled into one. See "Description of key hosts". Description of key hosts: NOTE: All hostnames below are relative to the EC's subdomain. Think of the subdomain as the "EC name". For instance, if the EC name is "prod.acme.net", then the isconf server for the prod.acme.net EC is going to have an FQDN of "isconf.prod.acme.net". Each of the following hostnames *must* exist in each EC's subdomain, even though a host might be shared between EC's. I.E. use CNAMEs. isconf This is the "gold server" described at Infrastructures.Org. It should be the only hand-maintained machine in the EC (though you should still list it in hosts.conf and use the code to maintain this machine as much as possible). rsync The anonymous rsync server for all hosts in the EC. This is almost always a CNAME pointing at the isconf server. images This host is where install packages, network install images, and other large blobs are NFS-mounted from; specifically, images:/export/images. The DNS entry for this host is almost always a CNAME pointing at the isconf server. If you have more than one nearly-identical EC (i.e. dev/test/stage/prod), you will probably want one master images server for all of your EC's, and a slave images server in each EC, with selected contents of images:/export/images being replicated to each slave from the master using rsync or equivalent. cvs This is where the CVS repository lives, at cvs:/cvsroot. This hostname is usually a CNAME pointing at the master isconf server, though you could put the CVS repository anywhere you choose. You want only one CVS repository for all of your EC's though -- use branch tags if you need to keep EC's versioned separately from each other. Description of key files/scripts: /etc/rc.isconf This is the only script you'd normally run on a host to cause it to change. Runs at boot as "rc.isconf start", or hourly from cron as "rc.isconf cron". Can also be (carefully) run manually for testing or if you're in a hurry and/or don't want to reboot. The script first updates itself via anon rsync, then updates the /var/isconf tree, then runs /var/isconf/bin/isconf. Can also be manually run to do specific tasks, as in "rc.isconf clstart" to start an HACMP cluster manager, or "rc.isconf clapps" to install/update cluster applications after a cluster is all the way up. It in fact can be used to call "rc.isconf target" where 'target' is any valid 'make' target (see below). The latter usage is at your own risk -- you can do great damage if you don't put some thought into this. /var/isconf/bin/isconf Reads /var/isconf/conf/hosts.conf (below), uses the configuration information found there to set up the environment, targets, and command line arguments for 'make', then runs make. /var/isconf/conf/hosts.conf This is the top-level configuration file for your entire site. In it you will find the names of hosts, followed by the attributes and make targets which describe each host. See the comments in the file for more info. /var/isconf/conf/main.mk This is the top-level makefile which is passed to 'make' by the /var/isconf/bin/isconf script. It usually doesn't contain much; its primary purpose is to "include" other makefiles, such as /var/isconf/conf/$(OS).mk. (The OS environment variable is set by the /var/isconf/bin/isconf script, based on the output of 'uname'.) For instance, in an EC containing both Solaris and AIX machines, you will find both aix.mk and sunos.mk in the /var/isconf/conf/ directory. The makefiles can in turn call other scripts, other iterations of make itself, or do anything else needed to perform the actions needed to build, configure, and upgrade machines, per the configuration specified in /var/isconf/conf/hosts.conf. /var/isconf/stamps Where /var/isconf/bin/isconf and make store their state. These are (usually) zero-length files, each named after a makefile stanza. If it's stamped, the stanza's been run, and the change won't be made on this host again. The stamps are (usually) made using a 'touch $@' command as the last line of the make stanza. If you want a stanza to repeat, don't include the 'touch'... How to make a change: WARNING: It is possible to use this procedure to destroy every host in your infrastructure within minutes. WARNING: It is possible to render this code incapable of updating itself, leaving you with little option other than to log into every machine in your infrastructure and fix it. - Join the infrastructures mailing list at Infrastructures.Org. - Read this entire document several times. If there is anything you don't understand or don't agree with, do not proceed until you join the infrastructures mailing list and gain an understanding of the issue. - Discuss the change you want to make on the infrastructures list, comparing notes with others until you are satisfied that the change you want to make is both right and safe. DISCLAIMER: Only you are responsible for making this decision. Only you are responsible for ensuring that you understand the risks and issues involved. - Learn how to use CVS. Do not proceed until you do. See www.cvshome.org for more information. - In your home directory, working as an ordinary user, check out the correct copy of the isconf code from CVS. See /var/isconf/CVS/Repository, /var/isconf/CVS/Root, and /var/isconf/CVS/Tag for hints. - In the CVS sandbox you just checked out, edit the acme.net/isconf/conf/hosts.conf file if you want to add a host. See the contents of the file for more information. - If you want to make a change to a host, then find the stanza in the makefiles and/or in hosts.conf which describes that change. (This assumes the change is one which has already been coded for an existing host.) Add the make target name to the BOOT='...' variable for that host in hosts.conf. It's a very good idea to always add targets to the end of BOOT -- see the discussion of 'Ordering' at Infrastructures.Org. - If you want your change to happen during cron, or to be repeated every hour (or whenever cron runs rc.isconf), then you'll be editing the CRON='...' variable instead of the BOOT variable. Otherwise the same rules apply. - If you want to code a new change, one that's never been done on any host in the EC before, then you'll need to add one or more stanzas to the makefile. Learn 'make'. Do not proceed until you do. If want your newly written change code to only run once per host, make sure you add a 'touch $@' line to the end of your make stanza(s). See the description of the /var/isconf/stamps directory above. - If your change is to a file that is simply replicated to all hosts in the EC, see the acme.net/system tree. - After you are happy with your change and have checked it back in, TEST IT! Do not proceed until you do. Read the warnings above. To test your change, log into the rsync server for your test EC (you do have a test EC, don't you?), cd to /images/acme.net, do a 'cvs up -d', then log into a test host and run 'rc.isconf start' (or 'rc.isconf cron'). Ensure the script exits with a return code of zero. Test the new functionality you just added. Redo until happy. - To deploy the change to production or other EC's, log into their gold servers and do a cvs update (WARNING: Take care to observe any branch tags that might be in use). If your change included new blobs in /images, then you'll need to do whatever you do to replicate those -- look under /images/acme.net/bin for replication scripts; that's usually where they are. How to install ISconf: ISconf version 2, while publicly available, is not optimized for an easy install at a new site -- every site is way too different. We don't recommend you try this alone. Join the infrastructures list at Infrastructures.Org to converse with people working on similar efforts. TerraLuna, LLC is a consulting firm run by the same people who created Infrastructures.Org, who maintain ISconf, and who make a living out of using this code to automate systems administration for corporate clients. See the TerraLuna.com or stevegt.com websites for contact information. $Id: README,v 1.1 2002/01/25 10:38:24 stevegt Exp $