DAOS Errors

DAOS error numbering starts at 1000. The most common errors are documented in the table below.

DAOS Error Value Description
DER_NO_PERM 1001 No permission
DER_NO_HDL 1002 Invalid handle
DER_INVAL 1003 Invalid parameters
DER_NOSPACE 1007 No space left on storage target
DER_NOSYS 1010 Function not implemented
DER_IO 2001 Generic I/O error
DER_ENOENT 2003 Entry not found
DER_KEY2BIG 2012 Key is too large
DER_IO_INVAL 2014 IO buffers can't match object extents

When an operation fails, DAOS returns a negative DER error. For a full list of errors, please check (DER_ERR_GURT_BASE is equal to 1000 and DER_ERR_DAOS_BASE is equal to 2000).

The function d_errstr() is provided in the API to convert an error number to an error message.

Log Files

On the server side, there are three log files created as part of normal server operations:

Component Config Parameter Example Config Value
Control Plane control_log_file /tmp/daos_control.log
Data Plane log_file /tmp/daos_server.log
Privileged Helper helper_log_file /tmp/daos_admin.log

Control Plane Log

The default log level for the control plane is INFO. The following levels may be set using the control_log_mask config parameter:

Data Plane Log

Data Plane (daos_io_server) logging is configured on a per-instance basis. In other words, each section under the servers: section must have its own logging configuration. The log_file config parameter is converted to a D_LOG_FILE environment variable value. For more detail, please see the Debugging System section of this document.

Privileged Helper Log

By default, the privileged helper only emits ERROR-level logging which is captured by the control plane and included in that log. If the helper_log_file parameter is set in the server config, then DEBUG-level logging will be sent to the specified file.

Debugging System

DAOS uses the debug system defined in CaRT but more specifically the GURT library. Log files for both client and server are written to "/tmp/daos.log" unless otherwise set by D_LOG_FILE.

Registered Subsystems/Facilities

The debug logging system includes a series of subsystems or facilities which define groups for related log messages (defined per source file). There are common facilities which are defined in GURT, as well as other facilities that can be defined on a per-project basis (such as those for CaRT and DAOS). DD_SUBSYS can be used to set which subsystems to enable logging. By default all subsystems are enabled ("DD_SUBSYS=all").

Priority Logging

All macros that output logs have a priority level, shown in descending order below.

The priority level that outputs to stderr is set with DD_STDERR. By default in DAOS (specific to the project), this is set to CRIT ("DD_STDERR=CRIT") meaning that all CRIT and more severe log messages will dump to stderr. However, this is separate from the priority of logging to "/tmp/daos.log". The priority level of logging can be set with D_LOG_MASK, which by default is set to INFO ("D_LOG_MASK=INFO"), which will result in all messages excluding DEBUG messages being logged. D_LOG_MASK can also be used to specify the level of logging on a per-subsystem basis as well ("D_LOG_MASK=DEBUG,MEM=ERR").

Debug Masks/Streams:

DEBUG messages account for a majority of the log messages, and finer-granularity might be desired. Mask bits are set as the first argument passed in D_DEBUG(mask, ...). To accomplish this, DD_MASK can be set to enable different debug streams. Similar to facilities, there are common debug streams defined in GURT, as well as other streams that can be defined on a per-project basis (CaRT and DAOS). All debug streams are enabled by default ("DD_MASK=all").

Common Use Cases

Refer to the DAOS Environment Variables document for more information about the debug system environment.

Common DAOS Problems

This section to be updated in a future revision.

Bug Report

Bugs should be reported through our issue tracker^1 with a test case to reproduce the issue (when applicable) and debug logs.

After creating a ticket, logs should be gathered from the locations described in the Log Files section of this document and attached to the ticket.

To avoid problems with attaching large files, please attach the logs in a compressed container format, such as .zip or .tar.bz2.