DAOS Software Installation

DAOS runs on both Intel 64 and ARM64 platforms and has been successfully tested on CentOS 7, OpenSUSE 42.2 and Ubuntu 18.04 distributions.

Software Dependencies

DAOS requires a C99-capable compiler, a Go compiler, and the scons build tool. Moreover, the DAOS stack leverages the following open source projects:

  • CaRT for high-performance communication leveraging advanced network capabilities.

  • gRPC provides an secured out-of-band channel for DAOS administration.

  • PMDK for persistent memory programming.

  • SPDK for userspace NVMe device access and management.

  • FIO for flexible testing of Linux I/O subsystems, specifically enabling validation of userspace NVMe device performance through fio-spdk plugin.

  • ISA-L for checksum and erasure code computation.

  • Argobots for thread management.

The DAOS build system can be configured to download and build any missing dependencies automatically.

Distribution Packages

DAOS RPM and deb packaging is under development and will be available for DAOS v1.0. Integration with the Spack package manager is also under consideration.

DAOS from Scratch

The following instructions have been verified with CentOS. Installations on other Linux distributions might be similar with some variations. Developers of DAOS may want to review the additional sections below before beginning, for suggestions related specifically to development. Contact us in our forum for further help with any issues.

Build Prerequisites

To build DAOS and its dependencies, several software packages must be installed on the system. This includes scons, libuuid, cmocka, ipmctl, and several other packages usually available on all the Linux distributions. Moreover, a Go version of at least 1.10 is required.

A exhaustive list of packages for each supported Linux distribution is maintained in the Docker files:

The command lines to install the required packages can be extracted from the Docker files by removing the "RUN" command which is specific to Docker. Check the docker directory for different Linux distribution versions.

DAOS Source Code

To check out the DAOS source code, run the following command:

$ git clone https://github.com/daos-stack/daos.git

This command clones the DAOS git repository (path referred as ${daospath} below). Then initialize the submodules with:

$ cd ${daospath}
$ git submodule init
$ git submodule update

Building DAOS & Dependencies

If all the software dependencies listed previously are already satisfied, then type the following command in the top source directory to build the DAOS stack:

$ scons --config=force install

If you are a developer of DAOS, we recommend following the instructions in the DAOS for Development section.

Otherwise, the missing dependencies can be built automatically by invoking scons with the following parameters:

$ scons --config=force --build-deps=yes install

By default, DAOS and its dependencies are installed under ${daospath}/install. The installation path can be modified by adding the PREFIX= option to the above command line (e.g., PREFIX=/usr/local).

Environment setup

Once built, the environment must be modified to search for binaries and header files in the installation path. This step is not required if standard locations (e.g. /bin, /sbin, /usr/lib, ...) are used.

CPATH=${daospath}/install/include/:$CPATH
PATH=${daospath}/install/bin/:${daospath}/install/sbin:$PATH
export CPATH PATH

If using bash, PATH can be set up for you after a build by sourcing the script scons_local/utils/setup_local.sh from the daos root. This script utilizes a file generated by the build to determine the location of daos and its dependencies.

If required, ${daospath}/install must be replaced with the alternative path specified through PREFIX.

DAOS in Docker

This section describes how to build and run the DAOS service in a Docker container. A minimum of 5GB of DRAM and 16GB of disk space will be required. On Mac, please make sure that the Docker settings under "Preferences/{Disk,Memory}" are configured accordingly.

Building from GitHub

To build the Docker image directly from GitHub, run the following command:

$ docker build -t daos -f Dockerfile.centos.7 github.com/daos-stack/daos#:utils/docker

This creates a CentOS 7 image, fetches the latest DAOS version from GitHub, builds it, and installs it in the image. For Ubuntu and other Linux distributions, replace Dockerfile.centos.7 with Dockerfile.ubuntu.18.04 and the appropriate version of interest.

Once the image created, one can start start a container that will eventually run the DAOS service:

$ docker run -it -d --privileged --name server \
        -v /tmp/uri:/tmp/uri \
        -v /dev/hugepages:/dev/hugepages \
        daos

If Docker is being run on a non-Linux system (e.g., OSX), the export of /dev/hugepages should be removed since it is not supported.

Building from a Local Tree

To build from a local tree stored on the host, a volume must be created to share the source tree with the Docker container. To do so, execute the following command to create a docker image without checking out the DAOS source tree:

$ docker build -t daos -f utils/docker/Dockerfile.centos.7 --build-arg NOBUILD=1 .

Then create a container that can access the local DAOS source tree:

$ docker run -it -d --privileged --name server \
        -v ${daospath}:/home/daos/daos:Z \
        -v /tmp/uri:/tmp/uri \
        -v /dev/hugepages:/dev/hugepages \
        daos

${daospath} should be replaced with the full path to your DAOS source tree. As mentioned above, the export of /dev/hugepages should be removed if the host is not a Linux system.

Then execute the following command to build and install DAOS in the container:

$ docker exec server scons --build-deps=yes install PREFIX=/usr

Running DAOS Service in Docker

Please first make sure that the uio_pci_generic module is loaded:

$ ls /sys/bus/pci/drivers/uio_pci_generic
ls: cannot access /sys/bus/pci/drivers/uio_pci_generic: No such file or director

$ sudo modprobe uio_pci_generic

$ ls /sys/bus/pci/drivers/uio_pci_generic
0000:00:04.0  0000:00:04.3  0000:00:04.6  0000:5f:00.0  0000:80:04.2  0000:80:04.5  0000:81:00.0  module     uevent
0000:00:04.1  0000:00:04.4  0000:00:04.7  0000:80:04.0  0000:80:04.3  0000:80:04.6  0000:da:00.0  new_id     unbind
0000:00:04.2  0000:00:04.5  0000:5e:00.0  0000:80:04.1  0000:80:04.4  0000:80:04.7  bind          remove_id

SCM and NVMe storage can then be configured by running the follow command:

$ docker exec server daos_server storage prepare -n -f

Note that this command reports that /dev/hugepages is not accessible on OSX. This still allows running the DAOS service despite the error.

The DAOS service can then be started as follows:

$ docker exec server mkdir /var/run/daos_server
$ docker exec server orterun -allow-run-as-root -H localhost -np 1 \
        daos_server start \
        -a /tmp/uri \
        -o /home/daos/daos/utils/config/examples/daos_server_local.yml

The daos_server_local.yml configuration file sets up a simple local DAOS system with a single server instance running in the container. By default, it uses 4GB of DRAM to emulate persistent memory and 16GB of bulk storage under /tmp. The storage size can be changed in the yaml file if necessariy.

Once started, the DAOS server waits for the administrator to format the system. This can be triggered in a different shell, using the following command:

$ docker exec server daos_shell -i storage format -f

Upon successful completion of the format, the storage engine is started, and pools can be created using the daos admin tool (see next section).

DAOS for Development

This section covers specific instructions to create a developer-friendly environment to contribute to the DAOS development. This includes how to regenerate the protobuf files or add new Go package dependencies, which is only required for development purposes.

Building DAOS for Development

For development, it is recommended to build and install each dependency in a unique subdirectory. The DAOS build system supports this through the TARGET_PREFIX variable. Once the submodules have been initialized and updated, run the following commands:

$ scons PREFIX=${daos_prefix_path}
      TARGET_PREFIX=${daos_prefix_path}/opt install
      --build-deps=yes
      --config=force

Installing the components into seperate directories allow upgrading the components individually by replacing --build-deps=yes with --update-prereq={component_name}. This requires a change to the environment configuration from before. For automated environment setup, source scons_local/utils/setup_local.sh.

ARGOBOTS=${daos_prefix_path}/opt/argobots
CART=${daos_prefix_path}/opt/cart
HWLOC=${daos_prefix_path}/opt/hwloc
MERCURY=${daos_prefix_path}/opt/mercury
PMDK=${daos_prefix_path}/opt/pmdk
OMPI=${daos_prefix_path}/opt/ompi
OPA=${daos_prefix_path}/opt/openpa
PMIX=${daos_prefix_path}/opt/pmix
FIO=${daos_prefix_path}/opt/fio
SPDK=${daos_prefix_path}/opt/spdk

PATH=$CART/bin/:$OMPI/bin/:${daos_prefix_path}/bin/:$PATH

With this approach, DAOS would get built using the prebuilt dependencies in ${daos_prefix_path}/opt, and required options are saved for future compilations. So, after the first time, during development, only "scons --config=force" and "scons --config=force install" would suffice for compiling changes to DAOS source code.

If you wish to compile DAOS with clang rather than gcc, set COMPILER=clang on the scons command line. This option is also saved for future compilations.

Go dependencies

Developers contributing Go code may need to change the external dependencies located in the src/control/vendor directory. The DAOS codebase uses dep to manage these dependencies.

On EL7 and later:

$ yum install yum-plugin-copr
$ yum copr enable hnakamur/golang-dep
$ yum install golang-dep

On Fedora 27 and later:

$ dnf install dep

On Ubuntu 18.04 and later:

$ apt-get install go-dep

For OSes that don't supply a package:

  • Ensure that you have a personal GOPATH (see "go env GOPATH", referred to as "$GOPATH" in this document) and a GOBIN ($GOPATH/bin) set up and included in your PATH:
$ mkdir -p $GOPATH/bin
$ export PATH=$GOPATH/bin:$PATH

To update the vendor directory using dep after changing Gopkg.toml, make sure DAOS is cloned into "$GOPATH/src/github.com/daos-stack/daos"

Then:

$ cd $GOPATH/src/github.com/daos-stack/daos/src/control
$ dep ensure

Protobuf Compiler

The DAOS control plane infrastructure uses protobuf as the data serialization format for its RPC requests. The DAOS proto files use protobuf 3 syntax, which is not supported by the platform protobuf compiler in all cases. Not all developers will need to build the proto files into the various source files. However, if changes are made to the proto files, the corresponding C and Go source files will need to be regenerated with a protobuf 3.* or higher compiler.

The recommended installation method is to clone the git repositories, check out the tagged releases noted below, and install from source. Later versions may work, but are not guaranteed.

Generate the Go file using the gRPC plugin. You can designate the directory location:

$ protoc myfile.proto --go_out=plugins=grpc:<go_file_dir>

Generate the C files using Protobuf-C. As the header and source files in DAOS are typically kept in separate locations, you will need to move them manually to their destination directories:

$ protoc-c myfile.proto --c_out=.
$ mv myfile.pb-c.h <c_file_include_dir>
$ mv myfile.pb-c.c <c_file_src_dir>