beowulf-config

Name

/etc/beowulf/config -- Configuration file for BProc and beoboot

Description

This manual page describes the format of the Beowulf-config file /etc/beowulf/config. This file defines the structure of a Scyld ClusterWare cluster, and provides a central location for many of the operational parameters.

The Beowulf-config file contains the settings for beoboot, node initialization, BProc communication parameters, and other aspects of cluster operation.

The syntax of the Scyld configuration files is standardized, and is intended for human editing with embedded comments. Tools are provided for reading and writing from common programming and scripting languages, with writing retaining comments and formatting.

Tip

While /etc/beowulf/config may be edited with any text editor, it is best changed using the beonetconf or beosetup programs. However, some parameters may not be editable using those programs, and must be set carefully when editing the file, as incorrect editing may leave the cluster unusable.

Caution

Regardless of which option you use to edit the Beowulf-config file, the changes don't take effect until Beowulf services are restarted. You can restart Beowulf services by rebooting the master node, or by executing either /ect/rc.d/init.d/beowulf restart or service beowulf restart.

Scyld Config File Format

The config file format is a line-oriented sequence of configuration entries. Each configuration entry starts with a keyword followed by parameters. The line is terminated by a newline or '#'. The latter character starts a comment.

The keyword and following parameters have the same syntax rules: they may be preceded by whitespace and continue to the next whitespace or the end of the line.

Keywords and following parameters may include whitespace by quoting between a matching pair of '"' (double quote) or ''' (single quote) characters. A '\' (backslash) removes the special meaning of the following quote character.

Note that comments and newlines take precedence over any other processing, thus a '#' may not be used in a keyword or embedded in a parameter, and a backslash followed by a newline does not join lines.

Each configuration option is contained on a single line, with keyword and optional parameters. Blanks lines are ignored. Comments begin with an unquoted '#' and continue to the end of the line.

The Beowulf-config file is intended to be parsed using the beoconfig utility.

Keywords

interface interfacename

The interface directive is used to specify the name of the interface that connects the master node to the compute nodes. This is used by the cluster services and management tools such as the BProc master daemon and the Beoserv daemon. Common values are eth1 (the second ethernet device) and myri0 (for Myrinet). If present, entries after the interface name specify the IP address and netmask that the interface should be configured to.

iprange [nodenumber] IPaddress1 IPaddress2

The iprange directive specifies the range of IP addresses to be assigned to nodes. If the optional nodenumber is given, the first address in the range will be assigned to that node, the second address to the next node, etc. If no node number is given, the address assignment will begin with the node following the node that was last assigned. If no nodes have been assigned, the assignment will begin with node 0.

ip [nodenumber] IPaddress

The ip directive assigns one or more IP addresses to a single node. The addresses must be in dotted notation (eg. 192.168.1.100). If the optional nodenumber argument is given, the specified IP address will be used for that node. If no node number is given, the addresses will be assigned beginning with the node number following whatever node was assigned last. If no nodes have been assigned, the IP address assignment will start with node 0.

bootfile [nodenumber] phase2-pathname

The bootfile directive is used to specify the name of the phase 2 image that should be used. If a node number is given, then the specified phase 2 image will be used for that node. If no node number is given, then the specified phase 2 image will be used for all compute nodes that have not been specified in any other bootfile entries. The nodenumber can also be a list of comma separated node ranges (eg. 1-5, 10-20).

bprocport port

The bprocport directive is used to specify the TCP port that the compute node connects to on the master for BProc.

fsck fsck-policy

The fsck directive is used to specify the file system checking policy to be used at boot time. The valid policies are never, safe, or full.

never

The file system on the compute nodes will not be checked on boot.

safe

The file system on the compute nodes will go through a safe check everytime the compute node boots.

full

The file system on the compute nodes will go through a full check everytime the compute node boots. The full check might possibly remove files from the filesystem if they cannot be repaired.

kernelcommandline options

The kernelcommandline directive is used to specify any options you wish to have passed to the kernel on the compute nodes. These are the same options that are normally passed with 'append=' in lilo, or on the lilo prompt while the machine is booting (eg. 'kernelcommandline apm=power-off'.)

kernelimage imagename

The kernelimage directive is used to specify the full path to the kernel that should be used when creating the phase 2 boot images for the compute nodes.

ignore MACaddress

The ignore directive is used to specify a MAC address (eg. 00:11:22:AA:BB:CC) that beoserv should ignore RARP requests from. Multiple ignore directives are allowed.

insmod module-name [options]

The insmod directive is used to specify a kernel module to be loaded (usually a network driver). Options for the module may be specified as well.

libraries librarypath1 [, librarypath2, ...]

The libraries directive is used to specify a list of libraries that should be cached on the slave nodes when they boot. The library path can be a directory or file. If a directory name is specified, everything in that directory will be copied over, but none of the sub-directories under that directory will be copied over.

logfacility facility

The logfacility directive specifies the log facility that the BProc master daemon should use. Some example log facility names are daemon, syslog, and local0 (see the syslog documentation for more information.) The default log facility is daemon.

mcastbcast interface

The mcastbcast directive is used to tell beoserv to use broadcast instead of multicast when transmitting files over the interface. This is useful when network equipment has trouble with heavy multicast traffic.

mcastthrottleinterface rate

The mcastthrottle directive is used to control the rate at which data is transmitted over the specified interface. The rate is given in megabits per second. This is useful when the compute node interfaces cannot keep up with the master interface when sending large files.

mkfs mkfs-policy

The mkfs directive specifies the policy to use when building a Linux file system on the compute nodes. The valid policies are never, if_needed, or always.

never

The filesystem on the compute nodes will never be recreated on boot.

if_needed

The filesystem on the compute nodes will only be recreated if the filesystem check fails.

always

The filesystem on the compute nodes will be recreated on every boot. fsck will be assumed to be set to never when this is set.

modprobe module-name [options ]

The modprobe directive is used to specify the name of the kernel module to be loaded with dependency checking along with any specified module options.

modarg options

The modarg directive is used to specify options to be used for modules that are loaded during the boot process without options. This is useful for specifying options to modules that get loaded during the PCI scan.

moddep module-list

The moddep directive is used to specify module dependencies. The first module listed is dependent on the remaining modules in the space separated list. The first module will be loaded after all other listed modules. Module dependency information is normally automatically generated by the beoboot script.

node [nodenumber] MACaddress [user] [group]

The node directive is used to assign MAC addresses to node numbers. The user is the username of the user who owns the node (this can also be "any"), and the group is the name of the group who owns the node (this can also be "any"). There should be one of these lines for each node in your cluster. Note the following:

  • If a vaule is not provided for the nodenumber argument, the first node entry is node 0, the second is node 1, the third is node 2, etc.

  • The value off can be used for the MACaddress argument to leave a place holder for that node number.

  • To skip a node number, use the value node or node off for the MACaddress argument.

  • To skip a node number and make sure it will never be automatically filled in by something later in the future, use the value node reserved for the MACaddress argument.

nodes numnodes

The nodes directive is used to specify the total possible number of nodes in the cluster. This should normally be set to match the iprange. However, if multiple ipranges are specified, then this value should represent the total number of nodes in all the iprange entries.

pci vendorid deviceid drivername

The pci directive is used to specify what driver should be used in support of the specified PCI device. A device is identified by a unique vendor ID and device ID pair. The vendor and device ID's can be either in decimal or hexadecimal with the "0x" notation. You should have one of these lines for each PCI ID (a vender ID combined with a device ID) for each device on your compute nodes that is not already recognized. Any module dependencies or arguments should be specified with moddep and modarg.

server transport-protocol port

The server directive is used to specify the port number that beoserv uses for the specified transport protocol (see the transportmode directive.) A different port number must be used for each protocol type. The allowable values for the protocol are mc (multicast) or tcp (unicast). The default port numbers are 10001 for multicast and 1556 for unicast.

transportmode protocol

The transportmode directive is used to specify the transport protocol to be used by beoserv for serving IP addresses, boot files, and libraries to the compute nodes. The valid transport protocols are mc (multicast) or tcp (unicast). The default value is tcp.

Examples

# Beowulf Configuration file

# Network interface used for Beowulf
# Only first argument to interface is important
interface eth1 192.168.1.1 255.255.255.1

# These two should probably agree for most users
iprange 192.168.1.100 192.168.1.107
nodes 8

# Default location of boot images
bootfile /var/beowulf/boot.img
kernelimage /boot/vmlinuz-2.4.17-0.18.12.beo
kernelcommandline apm=power-off

# Default libraries
libraries /lib /usr/lib

# Default file system policies.
fsck full
mkfs if_needed

# beoserv settings
server mc 10001
server tcp 1556
transportmode mc

# Default Modules
bootmodule 3c59x 8139too dmfe eepro100 epic100 hp100 natsemi
bootmodule ne2k-pci pcnet32 sis900 starfire sundance tlan
bootmodule tulip via-rhine winbond-840 yellowfin

# Non-kernel integrated drivers
bootmodule e100 bcm5700 # gm

# PCI Gigabit Ethernet.
#  * AceNIC and SysKonnect firmwares are very large.
#  * Some of these are distributed separate from the kernel
bootmodule dl2k hamachi e1000 ns83820 # acenic sk98lin

node 00:50:8B:D3:25:4D
node 00:50:8B:D3:07:8B
node 00:50:8B:D3:31:FB
node 00:50:8B:D3:62:A0
node 00:50:8B:D3:00:66
node 00:50:8B:D3:30:42
node 00:50:8B:D3:98:EA

See Also

beoconfig(1), beonetconf(1), and beosetup(1).