SAP HANA Runtime Dumps
You are interested in details about SAP HANA runtime dumps.
Environment
SAP HANA
Cause
1. What is a SAP HANA runtime dump?
2. Where do I find more information related to the creation of runtime dumps?
3. What is the purpose of runtime dumps?
4. What flavors of runtime dumps exist?
5. Is a runtime dump generated on service, host, tenant or system level?
6. What is the meaning of trace files containing “rtedump_tmp” in their names?
7. Which parameters influence the generation of a runtime dump?
8. What are the main parts of a runtime dump?
9. How long does the creation of a runtime dump typically take and how large will it be?
10. Is the information collected by a runtime dump always consistent?
11. Which advanced options exist for runtime dump generation?
Resolution
1. What is a SAP HANA runtime dump?
A SAP HANA runtime dump is a text file that provides various information about the current state of the SAP HANA database. Runtime dumps are essential for SAP support to understand the SAP HANA behavior in a problem situation.
A runtime dump can be created manually or automatically and per default it is created in the SAP HANA trace directory /usr/sap/<sid>/HDB<inst>/<host>/trace.
2. Where do I find more information related to the creation of runtime dumps?
SAP Note 2462851 provides guided answers for runtime dump creation.
3. What is the purpose of runtime dumps?
Runtime dumps can be used for various purposes, e.g.:
4. What flavors of runtime dumps exist?
Contents of runtime dumps can vary and the following flavors exist:
| Flavor | SAP Note | File name | Details |
| Classic | 1813020 | <service>_<host>.<port>.rtedump.<timestamp>.trc <service>_<host>_<port>_runtimedump_<timestamp>.trc |
Standard runtime dump that can be created in different situations:
Per default the file name contains ‘rtedump’, but the runtime dumps generated implicitly as part of the full system info dump may follow other naming conventions (including upper / lower case variations, other timestamp format and other mixture of “.” and “_”) and include “runtimedump” instead. |
| Memory | 1999997 | <service>_<host>.<port>.rtedump.<timestamp>.oom.trc <service>_<host>.<port>.rtedump.<timestamp>.after_oom_cleanup.trc <service>_<host>.<port>.rtedump.<timestamp>.compositelimit_oom.trc <service>_<host>.<port>.rtedump.<timestamp>.oom_memory_release.trc |
These runtime dump variations are linked to memory issues like an overall out-of-memory situation or a database request having reached the statement memory limit. |
| Savepoint | 2100009 | <service>_<host>.<port>.rtedump.<timestamp>.savepoint_blocked.trc | This runtime dump is generated if a blocking savepoint phase exceeds the configured blocked savepoint threshold:
indexserver.ini -> [persistence] -> runtimedump_for_blocked_savepoint_timeout = <seconds> |
| Index handle waits | 1999998 | <service>_<host>.<port>.rtedump.<timestamp>.indexhandle_blocked.trc | This type of runtime dump is generated if a wait for an index handle takes longer than defined with the following parameter (SAP HANA >= 2.00.010):
indexserver.ini -> [indexmgr] -> dump_wait_seconds = <seconds> |
| Page corruption | 2116157 | <service>_<host>.<port>.rtedump.<timestamp>.page.trc | This runtime dump variation is generated when a persistence inconsistency is recognized. |
5. Is a runtime dump generated on service, host, tenant or system level?
A runtime dump is always created on service level. If you want to generate a runtime dump for multiple services on the same host (e.g. indexserver, nameserver, xsengine) or for services on different hosts, you have to trigger them individually.
The following tools can be used to automatically generate runtime dumps for multiple services:
6. What is the meaning of trace files containing “rtedump_tmp” in their names?
While a runtime dump is created, the file is named with “rtedump_tmp” rather than “rtedump”. If you see a file with this substring, you can assume that the runtime dump is currently being created or that the creation of a previous runtime dump has intermittently failed and wasn’t properly finished. Once the runtime dump is completed, the file is renamed from “…rtedump_tmp…” to “…rtedump…”.
7. Which parameters influence the generation of a runtime dump?
The following SAP HANA parameters (SAP Note 2186744) have an influence on generation and content of runtime dumps (“indexserver.ini” can be adjusted to other services or “global.ini” depending on the desired context):
| Parameter | Unit | Default | SAP Note | Details |
indexserver.ini -> [persistence] -> runtimedump_for_blocked_savepoint_timeout |
s | 0 (no timeout) | 2100009 | When the blocking phase of a savepoint exceeds the configured timeout, a runtime dump of type “savepoint_blocked” is created. |
indexserver.ini -> [runtimedump] -> default_sections |
* | 2400007 | Define the sections that should be printed with a runtime dump per default, see “What are the main parts of a runtime dump?” below for the most important sections
Available starting with SAP HANA 1.00.122.07 and 2.00 |
|
indexserver.ini -> [sql] -> display_sql_statement_parameters_in_dump |
false | 2288661 | This configuration parameter can be used to activate the display of bind values (“statement parameters”) in SAP HANA dumps. |
8. What are the main parts of a runtime dump?
The runtime dump consists of several sections that are described in the table of contents at the beginning of the dump. Particularly important sections are:
| Section | Description | Available details |
| [BUILD] | Build information | SAP HANA version |
| [SYSTEMINFO] | System information | Instance name, operating system version |
| [STACK_SHORT] | Short call stacks | Short call stacks for all threads, important to understand current thread activies (SAP Note 2313619)
You can use the SAP HANADumpViewer (SAP Note 2491748) in order to graphically display the call stack hierarchies of the threads recorded in a runtime dump. |
| [LIMITS] | Limits | Operating system related limitations related to memory, files and processes |
| [STACK_FULL] | Full call stacks | Full call stacks for all threads, provides further details on top of [STACK_SHORT] that may sometimes be helpful for a better understanding |
| [STATISTICS] | Statistics data | Content of various SAP HANA monitoring views like DATABASE_STATISTICS, M_CONDITIONAL_VARIABLES, M_PROCESS_MEMORY, M_VOLUME_IO_DETAILED_STATISTICS, M_VOLUME_IO_TOTAL_STATISTICS, M_EVENTS, M_HEAP_MEMORY, M_JOBEXECUTORS, M_MEMORY_OBJECTS, M_MUTEXES, M_READWRITELOCKS, M_SEMAPHORES, M_LOG_BUFFERS, M_LOG_SEGMENTS, M_SAVEPOINTS, M_CONNECTIONS, M_PREPARED_STATEMENTS, M_TRANSACTIONS, M_SESSION_CONTEXT, M_MVCC_OVERVIEW, M_BLOCKED_TRANSACTIONS, M_SQL_PLAN_CACHE_OVERVIEW or their SAP HANA internal pendants |
| [PROCESS_INFO] | Process information | Overview of processes running on operating system level |
| [INDEXMANAGER_WAITGRAPH] | Wait graph for index handles and other internal locks | Internal SAP HANA lock dependencies (SAP Note 1999998) can be found in this section (in dot format). |
| [SAVEPOINT_SHAREDLOCK_OWNERS] | Owners of shared ConsistentChangeLock locks | In case a savepoint is blocked in the waitForLock phase (SAP Note 2100009), the blocking activities can be found in this section. |
| [OS_MEMORY] | Operating system information about memory | Overview of operating system memory information, including the configuration of critical aspects like transparent huge pages or ulimit |
| [IPMM_MEMORY] | Inter process memory management information | In this section you can find SAP HANA IPMM information (SAP Note 1999997) including memory allocated by different SAP HANA processes, allocation limit, used and allocated memory and a memory flight recorder. |
| [THREADS] | Running threads | Information about running threads including:
|
9. How long does the creation of a runtime dump typically take and how large will it be?
The actual duration and the size of a runtime dump significantly depends on the size of the system, actual load and resource consumption. Sizes can vary between a few MB and several 100 MB, and the runtime can vary between less than 1 minute and many hours.
10. Is the information collected by a runtime dump always consistent?
A runtime dump isn’t based on a consistent read and so information can be retrieved at different points in time, thus resulting in potentially inconsistent results. For example, the closely connected sections [STACK_SHORT] and [THREADS] may not match in all details and so the call stack of one thread may not match the thread state and thread method. Also within a single section like [STACK_SHORT] the information is not necessarily consistent.
11. Which advanced options exist for runtime dump generation?
the runtimedump option of hdbcons allows – among others – the following advanced options:
| Option | Details |
runtimedump l |
List available runtime dump sections |
runtimedump dump -c |
Write runtime dump to console (stdout) |
runtimedump dump -f <file_name> |
Write runtime dump to a file with name <file_name> Be aware that <file_name> must reside within one of the following folders:
Otherwise the error “write access for file with name … is prohibited” is issued. |
runtimedump dump -i |
Generate call stacks also for threads with inactive context |
runtimedump dump -s <section1>,...,<sectionN> |
Generate runtime dump only with the specified subset of available sections
Example: (runtime dump with sections [STACK_SHORT] and [THREADS]) runtimedump dump -s STACK_SHORT,THREADS |
Keywords
SAP HANA runtime dump analysis monitoring call stacks threads stuck hang
