SAP HANA Runtime Dumps

You are interested in details about SAP HANA runtime dumps.

Environment

SAP HANA

Cause

1. What is a SAP HANA runtime dump?
2. Where do I find more information related to the creation of runtime dumps?
3. What is the purpose of runtime dumps?
4. What flavors of runtime dumps exist?
5. Is a runtime dump generated on service, host, tenant or system level?
6. What is the meaning of trace files containing “rtedump_tmp” in their names?
7. Which parameters influence the generation of a runtime dump?
8. What are the main parts of a runtime dump?
9. How long does the creation of a runtime dump typically take and how large will it be?
10. Is the information collected by a runtime dump always consistent?
11. Which advanced options exist for runtime dump generation?

Resolution

1. What is a SAP HANA runtime dump?

A SAP HANA runtime dump is a text file that provides various information about the current state of the SAP HANA database. Runtime dumps are essential for SAP support to understand the SAP HANA behavior in a problem situation.

A runtime dump can be created manually or automatically and per default it is created in the SAP HANA trace directory /usr/sap/<sid>/HDB<inst>/<host>/trace.

2. Where do I find more information related to the creation of runtime dumps?

SAP Note 2462851 provides guided answers for runtime dump creation.

3. What is the purpose of runtime dumps?

Runtime dumps can be used for various purposes, e.g.:

Purpose SAP Note
Analysis of high memory consumption 1999997
Analysis of lock contention 1999998
Analysis of blocking savepoint phases 2100009
Analysis of high CPU consumption 2100040

4. What flavors of runtime dumps exist?

Contents of runtime dumps can vary and the following flavors exist:

Flavor SAP Note File name Details
Classic 1813020 <service>_<host>.<port>.rtedump.<timestamp>.trc
<service>_<host>_<port>_runtimedump_<timestamp>.trc
Standard runtime dump that can be created in different situations:

  • Manually via “runtimedump dump” in hdbcons (SAP Note 2222218)
  • Manually via SQL (MANAGEMENT_CONSOLE_PROC)
  • Manually via SAP HANA Studio -> “Administration” -> “Diagnosis Files” -> “Diagnosis Information” -> “Collect”
  • Implicitly as part of a full system info dump (SAP Note 1732157)
  • Automatically via SAP HANASitter (SAP Note 2399979)

Per default the file name contains ‘rtedump’, but the runtime dumps generated implicitly as part of the full system info dump may follow other naming conventions (including upper / lower case variations, other timestamp format and other mixture of “.” and “_”) and include “runtimedump” instead.

Memory 1999997 <service>_<host>.<port>.rtedump.<timestamp>.oom.trc
<service>_<host>.<port>.rtedump.<timestamp>.after_oom_cleanup.trc
<service>_<host>.<port>.rtedump.<timestamp>.compositelimit_oom.trc
<service>_<host>.<port>.rtedump.<timestamp>.oom_memory_release.trc
These runtime dump variations are linked to memory issues like an overall out-of-memory situation or a database request having reached the statement memory limit.
Savepoint 2100009 <service>_<host>.<port>.rtedump.<timestamp>.savepoint_blocked.trc This runtime dump is generated if a blocking savepoint phase exceeds the configured blocked savepoint threshold:

indexserver.ini -> [persistence] -> runtimedump_for_blocked_savepoint_timeout = <seconds>
Index handle waits 1999998 <service>_<host>.<port>.rtedump.<timestamp>.indexhandle_blocked.trc This type of runtime dump is generated if a wait for an index handle takes longer than defined with the following parameter (SAP HANA >= 2.00.010):

indexserver.ini -> [indexmgr] -> dump_wait_seconds = <seconds>
Page corruption 2116157 <service>_<host>.<port>.rtedump.<timestamp>.page.trc This runtime dump variation is generated when a persistence inconsistency is recognized.

5. Is a runtime dump generated on service, host, tenant or system level?

A runtime dump is always created on service level. If you want to generate a runtime dump for multiple services on the same host (e.g. indexserver, nameserver, xsengine) or for services on different hosts, you have to trigger them individually.

The following tools can be used to automatically generate runtime dumps for multiple services:

  • rtedump_script.zip (SAP Note 1813020)
  • Full system info dump (SAP Note 1732157)

6. What is the meaning of trace files containing “rtedump_tmp” in their names?

While a runtime dump is created, the file is named with “rtedump_tmp” rather than “rtedump”. If you see a file with this substring, you can assume that the runtime dump is currently being created or that the creation of a previous runtime dump has intermittently failed and wasn’t properly finished. Once the runtime dump is completed, the file is renamed from “…rtedump_tmp…” to “…rtedump…”.

7. Which parameters influence the generation of a runtime dump?

The following SAP HANA parameters (SAP Note 2186744) have an influence on generation and content of runtime dumps (“indexserver.ini” can be adjusted to other services or “global.ini” depending on the desired context):

Parameter Unit Default SAP Note Details
indexserver.ini -> [persistence] -> runtimedump_for_blocked_savepoint_timeout
s 0 (no timeout) 2100009 When the blocking phase of a savepoint exceeds the configured timeout, a runtime dump of type “savepoint_blocked” is created.
indexserver.ini -> [runtimedump] -> default_sections
* 2400007 Define the sections that should be printed with a runtime dump per default, see “What are the main parts of a runtime dump?” below for the most important sections

Available starting with SAP HANA 1.00.122.07 and 2.00

indexserver.ini -> [sql] -> display_sql_statement_parameters_in_dump
false 2288661 This configuration parameter can be used to activate the display of bind values (“statement parameters”) in SAP HANA dumps.

8. What are the main parts of a runtime dump?

The runtime dump consists of several sections that are described in the table of contents at the beginning of the dump. Particularly important sections are:

Section Description Available details
[BUILD] Build information SAP HANA version
[SYSTEMINFO] System information Instance name, operating system version
[STACK_SHORT] Short call stacks Short call stacks for all threads, important to understand current thread activies (SAP Note 2313619)

You can use the SAP HANADumpViewer (SAP Note 2491748) in order to graphically display the call stack hierarchies of the threads recorded in a runtime dump.

[LIMITS] Limits Operating system related limitations related to memory, files and processes
[STACK_FULL] Full call stacks Full call stacks for all threads, provides further details on top of [STACK_SHORT] that may sometimes be helpful for a better understanding
[STATISTICS] Statistics data Content of various SAP HANA monitoring views like DATABASE_STATISTICS, M_CONDITIONAL_VARIABLES, M_PROCESS_MEMORY, M_VOLUME_IO_DETAILED_STATISTICS, M_VOLUME_IO_TOTAL_STATISTICS, M_EVENTS, M_HEAP_MEMORY, M_JOBEXECUTORS, M_MEMORY_OBJECTS, M_MUTEXES, M_READWRITELOCKS, M_SEMAPHORES, M_LOG_BUFFERS, M_LOG_SEGMENTS, M_SAVEPOINTS, M_CONNECTIONS, M_PREPARED_STATEMENTS, M_TRANSACTIONS, M_SESSION_CONTEXT, M_MVCC_OVERVIEW, M_BLOCKED_TRANSACTIONS, M_SQL_PLAN_CACHE_OVERVIEW or their SAP HANA internal pendants
[PROCESS_INFO] Process information Overview of processes running on operating system level
[INDEXMANAGER_WAITGRAPH] Wait graph for index handles and other internal locks Internal SAP HANA lock dependencies (SAP Note 1999998) can be found in this section (in dot format).
[SAVEPOINT_SHAREDLOCK_OWNERS] Owners of shared ConsistentChangeLock locks In case a savepoint is blocked in the waitForLock phase (SAP Note 2100009), the blocking activities can be found in this section.
[OS_MEMORY] Operating system information about memory Overview of operating system memory information, including the configuration of critical aspects like transparent huge pages or ulimit
[IPMM_MEMORY] Inter process memory management information In this section you can find SAP HANA IPMM information (SAP Note 1999997) including memory allocated by different SAP HANA processes, allocation limit, used and allocated memory and a memory flight recorder.
[THREADS] Running threads Information about running threads including:

  • Thread type
  • Transaction ID
  • Update transaction ID
  • Connection ID
  • User names
  • Application information
  • Executing database requests
  • Thread method
  • Thread detail

9. How long does the creation of a runtime dump typically take and how large will it be?

The actual duration and the size of a runtime dump significantly depends on the size of the system, actual load and resource consumption. Sizes can vary between a few MB and several 100 MB, and the runtime can vary between less than 1 minute and many hours.

10. Is the information collected by a runtime dump always consistent?

A runtime dump isn’t based on a consistent read and so information can be retrieved at different points in time, thus resulting in potentially inconsistent results. For example, the closely connected sections [STACK_SHORT] and [THREADS] may not match in all details and so the call stack of one thread may not match the thread state and thread method. Also within a single section like [STACK_SHORT] the information is not necessarily consistent.

11. Which advanced options exist for runtime dump generation?

the runtimedump option of hdbcons allows – among others – the following advanced options:

Option Details
runtimedump l
List available runtime dump sections
runtimedump dump -c
Write runtime dump to console (stdout)
runtimedump dump -f <file_name>
Write runtime dump to a file with name <file_name>

Be aware that <file_name> must reside within one of the following folders:

  • /tmp
  • $HOME
  • $DIR_INSTANCE/work
  • $SAP_RETRIEVAL_PATH

Otherwise the error “write access for file with name … is prohibited” is issued.

runtimedump dump -i
Generate call stacks also for threads with inactive context
runtimedump dump -s <section1>,...,<sectionN>
Generate runtime dump only with the specified subset of available sections

Example: (runtime dump with sections [STACK_SHORT] and [THREADS])

runtimedump dump -s STACK_SHORT,THREADS

Keywords

SAP HANA runtime dump analysis monitoring call stacks threads stuck hang

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *