Debian Cluster Tools
A cluster, of the type considered in these pages, is a set of diskless machines that function based on a central disk server. The diskless machines are called nodes and the server is called a cluster server. These nodes may be either compute nodes in a parallel processing cluster or, more interestingly, X11-capable terminals, which may be in fact full-featured workstations. The nodes are diskless only in the sense that any disks existing in them are not used for any cluster-related system functions. These remote-boot nodes may be combined with boot alternatives for systems installed in the local disk as well.
The system image of the nodes on the server works essentially as a single image for all the nodes. This involve the use of filesystems organized in a special way, which we call hard-linked cluster filesystems. These are just ext2 or ext3 filesystems, the only thing which is different is the way in which the data is organized in them. With this method one is able to achieve an economy of disk space in the ratio of approximately N to 2, where N is the number of nodes.
At the same time, each node will function and can be managed essentially as if it had its own system image. The manager will be able to use the Debian package administration tools in the system of the nodes in the usual way. Some of the tools distributed here will access all the nodes and use in them, in a very transparent way, the usual Debian tools. In this way one can manage all the nodes at once. It is also very easy to change files simultaneously in all the nodes.
A cluster is easier to manage and maintain if all the nodes are identical, but it is by no means necessary that it be so. Not only one can have a considerable amount of inhomogeneity in the hardware, but it is also possible to have different sets of packages in each node. Of course, the more inhomogeneity there is in the software, the more disk space will be needed in the cluster server, and the more complex the administration becomes.
If all the nodes became completely different from each other, then the usefulness of this scheme would disappear. However, all systems tend to have a large amount of common core software even if they are used for very different things. If one is in a position of having to install, maintain and manage large number of similar machines, more often than not one will find this scheme useful and economical, both in terms of the hardware and in terms of the human resources involved in maintenance and management.
A solid network infrastructure is, of course, essential for a system which is strongly network-based such as a cluster. A 100 Mbps connectivity level for nodes and a 1 Gbps standard for the server and the backbone will be sufficient. It is important that all machines be connected to switches rather than hubs, for good network performance. The nodes should boot remotely from the server, through the network, using the PXE protocol and the PXELinux boot loader. The cluster architecture can be implemented in any hardware architecture where PXE and PXELinux will work, but so far it has been used in the i386 and amd64 hardware architectures only.
The maximum number of nodes that you can have in a cluster is limited both by the performance of the network and by the I/O capability of the disks on the cluster server. For compute nodes on a parallel-processing cluster the network is the main concern, while for remote-boot X11 terminals the I/O on the disks in the server is the limiting factor, mainly because of the system logging activity of the nodes. One can have 24 terminals based on a single 10000 RPM disk on a simple server quite comfortably, and one may go to double that number by improving the server. One can quite easily go to over a hundred compute nodes.
Anyone in charge of installing, maintaining and managing collections of identical or at least similar machines may find use for this cluster architecture. There are many common examples, such as departments than must provide X11 terminals for many members, public terminal rooms, public-access terminals such as in libraries, informatics laboratories and informatized classrooms. It is a very easy way to set up collections of identical public X11 terminals. There is the classic example of a parallel processing cluster too, of course. It should be noted that some special security precautions are necessary in the case of machines in public places.
A set of tools is available that will enable the manager of a cluster of this type to maintain the structure of the hard-linked filesystems, to manage Debian packages in all the nodes at once, to create new nodes, and to monitor the operation of the nodes of the cluster. Since all the tools are implemented as shell scripts, they are not really architecture-dependent, but the packages were made as i386 packages only. It is a trivial matter to make corresponding packages for other architectures, so long as one has access to machines of these architectures. A pair of "all" architecture packages is available as a temporary measure. These packages can be used in the case of the amd64 architecture, for example.
There are two Debian packages containing this software, they are called cluster-tools-client and cluster-tools-server. The first one must be installed on all the nodes of the cluster, the second one on the cluster server. You can get these Debian packages, as well as the sources, from this site:
http://sft.if.usp.br/debian-cluster
You can configure your Debian system to automatically get and update these packages by adding to the /etc/apt/sources.list file the line
deb http://sft.if.usp.br/debian-cluster stable main
An interim pair of packages for the "all" architecture, which can be used in any Debian architecture (but note that not all such architectures will support remote booting via PXE), can be found in the site
http://sft.if.usp.br/debian-leonel
You can configure your Debian system to automatically get and update these packages by adding to the /etc/apt/sources.list file the line
deb http://sft.if.usp.br/debian-leonel stable main
You can also mirror these sites using rsync, if you want to. You can get an example of a shell script to do this at the URL
http://sft.if.usp.br/sbin/mirror-debian-cluster
The changelog of the package cluster-tools-client is available here.
The changelog of the package cluster-tools-server is available here.
Package Installation: some hints about how to install or upgrade these packages.
You will be able to read the manual pages that come with the software using the link below. Start with the page cluster(5), which gives an explanation of the structure of a remote-boot cluster and points you to the pages of each one of the available tools.
This is the first half of a book about remote-boot clusters, called Linux for Workgroups. The second half was planned, but so far not completed. This first half covers the general design of a cluster, the choice of parts, the hardware assembly, the system installation and the basic aspects of the operation and management of such a cluster. A few details which are already a bit outdated are clearly marked as such. The chapters contained in this first half have been read and used by a few people involved with building clusters, and that was very helpful as a means of debugging them.
You can download the first half of the book, either in PDF (size: 1868 kB) format or in PS (size: 2063 kB) format. If you have the appropriate PDF plug-in in your browser, you should be able to read the PDF version online. If you have your browser configured to use a PS viewer, you should be able to read the PS version in your screen.
You will find here copies of the README files from the documentation sub-directories. You will get the full contents of these sub-directories with the cluster-tools-server package, the sub-directories will be in the package's documentation directory.
Configuration Library: a set of example configuration files for software elements common in clusters.
Disk Sizes: a description of the necessary sizes of disks and partitions for clusters.
First Node: a micro-howto describing a few ways to get the first node in place.
Kernel Configurations: kernel configuration files for remotely-booted compute nodes and X11 terminals.
Kernel patches: a security patch needed when using PXELinux, specially for machines in public rooms.
Package Lists: lists of packages usually needed on a cluster, in each type of machine.
You can also get a set of compressed tar files containing a stripped-down version the base system of the Sarge distribution, for setting up the first node:
There are also some kernel-image Debian packages available, for various types of clusters nodes, including both compute nodes and terminal nodes. These kernels support a few major-brand network cards:
Kernel-Image Packages for Cluster Nodes
In addition to this, the corresponding kernel-source packages are also available, and you can use them to compile your own custom kernels, for example in order to add support for additional network cards:
Kernel-Source Packages for Cluster Nodes
You can get these Debian packages from the debian-cluster site. You can install and update them in the usual way when you configure your /etc/apt/sources.list to use that site, as explained above.
Please send me mail about any bugs and problems you find in either the software or the Debian packages. For the time being there is no dedicated mailing list, but I will answer any questions I get by mail, so long as I can afford the time.
Jorge L. deLyra <delyra@fma.if.usp.br>, Department of Mathematical Physics, Physics Institute, the University of Sao Paulo, Sao Paulo, SP, Brazil.
However, do not try to send me mail in HTML format or from MS-Windows based mailers, for they will not pass my mail filters. If you have trouble reaching me at the address above, try the PMC mailing list at the address below.
PMC mailing list <pmc@fma.if.usp.br>.
Note that you must subscribe to the list first. It is managed by ezmlm, so all you have to do is to send an empty mail message to the subscription address below.
PMC mailing list subscription <pmc-subscribe@fma.if.usp.br>,
You can also get help about the mailing list sending mail to the help address below.
PMC mailing list help <pmc-help@fma.if.usp.br>,
|
|
|
|