Required Packages¶

To compile and run KFS, you need to have the following software packages installed on your machine:

• Boost (preferably, version 1.34 or higher)
• cmake (preferably, version 2.4.6 or higher)
• log4cpp (preferably, version 1.0)
• gcc version 4.1 (or higher)
• xfs devel RPMs on Linux

This document assumes that you have downloaded the source to directory: ~/code/kfs. We assume that you want to build the source in ~/code/kfs/build.

There are few parts to compiling the source:

1. Build the C++ side to get the metaserver/chunkserver binaries, tools, and the C++ client library.
2. Build the Java side to get a kfs.jar file which contains the wrapper calls to native C++ via JNI; this allows Java apps to access files stored in KFS.
3. Building Python extension module for Python support (optional)

Building C++ Components¶

It is preferable to build KFS binaries with debugging symbols. The sequence of steps are:

cd ~/code/kfs
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RelWithDebInfo ~/code/kfs/
gmake
gmake install

The binaries will be installed in:

1. Executables will be in: ~/code/kfs/build/bin
2. Libraries will be in: ~/code/kfs/build/lib

Building Java Components¶

To build Java support setup:

cd ~/code/kfs
ant jar

This will produce the following set of files:

• ~/code/kfs/build/classes --- This will contain the Java class files
• ~/code/kfs/build/kfs-{version}.jar --- The jar file containing the Java classes

Add .jar file to your CLASSPATH environment variable

export CLASSPATH=${CLASSPATH}:~/code/kfs/build/kfs-[version].jar

Building Python Support¶

Build the KFS client library (see above). Let the path to the shared libraries is ~/code/kfs/build/lib. Now, to build the Python extension module:

cd to ~/code/kfs/src/cc/access
Edit kfs_setup.py and setup the include path. Specifically,
       kfsext = Extension('kfs', include_dirs ['kfs/src/cc/', '<path to boost>'])
python kfs_setup.py ~/code/kfs/build/lib/ build

This will build a shared object library _kfs.so_. The kfs.so library needs to be installed either in the site-packages for python or in an alternate location. To install in site-packages for python:

python kfs_setup.py ~/code/kfs/build/lib/ install

To install in alternate location such as, ~/code/kfs/build/lib

python kfs_setup.py ~/code/kfs/build/lib install --home=~/code/kfs/build/lib

If you installed in alternate location, update PYTHONPATH environment variables:

export PYTHONPATH=${PYTHONPATH}:~/code/kfs/build/lib/lib64/python

Also, update the LD_LIBRARY_PATH environment variable:

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:~/code/kfs/build/lib

Using Python Client on the Mac (Leopard)¶

For the Mac, update the DYLD_LIBRARY_PATH environment variable (so that libkfsClient.dylib will be found):

export DYLD_LIBRARY_PATH=${DYLD_LIBRARY_PATH}:~/code/kfs/build/lib

  

   
Requirements¶
   
The KFS package comes with scripts for installing/starting/stopping KFS servers on a set of machines. The scripts use ssh to login and execute commands (either on a single node or in a cluster). The scripts require that the user have "no password" ssh access to every affected server. For every host affected, be sure you can "ssh host true" without being prompted for a password. 
   
Defining Machine Configuration for KFS-0.2.3 (or higher)¶
   
With KFS-0.2.3. (or higher), the system configuration can be defined using two files. One that defines the enviroment (such as, paths, port numbers to etc) for all the chunkservers and another file that lists the nodes on which the chunkservers should be started. The set of machines on which KFS servers have to be deployed is defined in a configuration file that follows the Python config file format. For example, the config file machines.cfg: 
   
[metaserver]
node: machine1
clusterkey: kfs-test-cluster
rundir: /mnt/kfs/meta
baseport: 20000
loglevel: INFO
numservers: 2
[chunkserver_defaults]
rundir: /mnt/kfs/chunk
chunkDir: /mnt/kfs/chunk/bin/kfschunk
baseport: 30000
space: 3400 G
loglevel: INFO


   
Next, assuming that there are 3 nodes in the cluster, their names should be listed in a separate file, machines.txt. The format of this file is one entry per line. For example, 

   
10.2.3.1
10.2.3.2
10.2.3.3


   
Defining Machine Configuration (Old style format)¶

   
The set of machines on which KFS servers have to be deployed is defined in a configuration file that follows the Python config file format. For example, 

   
[metaserver]
node: machine1
clusterkey: kfs-test-cluster
rundir: /mnt/kfs/meta
baseport: 20000
loglevel: INFO
numservers: 2
[chunkserver1]
node: machine2
rundir: /mnt/kfs/chunk
chunkDir: /mnt/kfs/chunk/bin/kfschunk
baseport: 30000
space: 3400 G
loglevel: INFO
[chunkserver2]
node: machine2
rundir: /mnt/kfs/chunk
chunkDir: /mnt/kfs/chunk/bin/kfschunk
baseport: 30000
space: 3400 G
loglevel: INFO


   
This file defines a configuration for a deployment comprising of a metaserver and two chunkservers. The various fields: 

   
node: This defines the machine name where the binary should run 
rundir: This defines the directory on the machine where KFS binaries will be installed. 
baseport: This port at which the metaserver/chunkserver process will listen for connection from clients 
loglevel: The level for outputting messages. Since KFS uses log4cpp, the values are INFO/DEBUG 
space: The storage space exported by a chunkserver for storing chunks (units are 'G' for GigaBytes and 'M' for MegaBytes) 
chunkDir : The list of directories used to store chunk files on the chunkserver nodes. For a JBOD configuration, this would be a space separated list of directory names. 
clusterkey : A key that is shared between metaserver/chunkserver. On startup, each chunkserver announces itself to the metaserver and provides the cluster key. This key should match the value that the metaserver; only then is the chunkserver part of the KFS deployment. 

   
Script for install/launching processes¶

   
Two sets of scripts are checked into the repository: 

   
Linux: Use ~/code/kfs/scripts 
Solaris, Mac OSX: Use ~/code/kfs/scripts.solaris 

   
The instructions in this section are for Linux platform. Use the corresponding script in scripts.solaris for Solaris/Mac platforms. When you use any script which will transfer files (particularly kfssetup) on the Macintosh, you will need the additional option --tar=tar; the scripts assume that GNU tar is named "gtar" otherwise. 

   
Installing KFS Binaries¶

   
Define the configuration in machines.cfg. Then, run the install script. The install script runs a set of processes concurrently. This works well when the servers need to be configured for a distributed setting. For a single node setup, the processes need to be launched sequentially: 

   
When all the servers are on a single host
cd ~/code/kfs/scripts
python kfssetup.py -f machines.cfg -m machines.txt -b ../build -w ../webui -s

MacOSX:
cd ~/code/kfs/scripts.solaris
python kfssetup.py -f machines.cfg -m machines.txt -b ../build -w ../webui --tar=tar -s

When the servers are on multiple hosts
cd ~/code/kfs/scripts
python kfssetup.py -f machines.cfg -m machines.txt -b ../build -w ../webui



   
NOTE: The "-m" option is supported only with KFS-0.2.3 or higher. 

   
Upgrading from prior KFS 0.2.x releases¶

   
Stop the servers. Upgrade the binaries and relaunch: 

   
Stopping the servers:
cd ~/code/kfs/scripts
python kfslaunch.py -f machines.cfg -S



   
When all the servers are on a single host
cd ~/code/kfs/scripts
python kfssetup.py -f machines.cfg -b ../build -w ../webui -u -s



   
When the servers are on multiple hosts
cd ~/code/kfs/scripts
python kfssetup.py -f machines.cfg -b ../build -w ../webui -u



   
Starting KFS servers¶

   
To launch the KFS servers, run the launch script: 

   
cd ~/code/kfs/scripts
python kfslaunch.py -f machines.cfg -m machines.txt -s


   
Checking System Status¶

   
To verify that the servers started up and are connected to the metaserver, use the kfsping tool: 

   
cd ~/code/kfs/build/bin/tools
kfsping -m -s <metaserver host> -p <metaserver port>


   
If you using KFS-0.2.2 or higher, you can use a Web browser to monitor the servers. The KFS package now includes a simple python-based web server that shows the set of servers that are currently connected to the metaserver. The KFS web server runs on the same machine as the KFS metaserver. The web server's port is metaserver's baseport + 50. For example, 

   
metaserver runs on machine node1 and uses port 20000
the web server runs on machine node1 at port 20050.  Point the browser at http://node1:20050


   
The KFS web server uses a file "all-machines.txt" to track where the chunkservers should be running. This file should be manually created and placed in ~/code/kfs/webui before deploying KFS binaries. The format of this file is the name of the chunkserver machines, one entry per line. For example. 

   
10.2.3.1
10.2.3.2
10.2.3.3


   
When you open the page, http://node1:20050/cluster-view, the web server will return a web page that lists: 

   
where the chunkservers are currently running 
where the chunkservers could not be started (due to ssh failures, etc.) 
where the chunkservers have failed 

   
Stopping KFS servers¶

   
To stop the KFS servers, run the launch script: 

   
cd ~/code/kfs/scripts
python kfslaunch.py -f machines.cfg -m machines.txt -S


   
Adding new chunkservers¶

   
It is not necessary to stop the KFS servers to add new chunkservers. Simply update the machines.cfg file and start the servers (see Starting KFS servers). 

   
Un-installing KFS Binaries¶

   
To format the filesystem/uninstall the binaries, run the install script: 

   
cd ~/code/kfs/scripts
python kfssetup.py -f machines.cfg -m machines.txt -b ../build/bin -U