What is Data Aging in SAP S/4HANA?

1. What is Data Aging in SAP S/4HANA??

?

Data Aging is a Suite-tailored data management concept for reducing the SAP HANA memory footprint, based on a Data Aging Framework provided by SAP NetWeaver ABAP. Data Aging is available for SAP Business Suite on HANA and SAP S/4HANA applications and offers the option of moving large amounts of data within SAP HANA in order to gain more working memory. Data Aging differentiates between operationally relevant data (Hot/Current), and data that is no longer accessed during normal operation (Cold/Historical). The data temperature can be used to horizontally partition the tables (taking part in Data Aging) for optimizing resource consumption and performance - moving data between the different partitions (i.e. from hot to cold partitions). Hot data resides within SAP HANA main memory, whereas cold data stays primarily stored on disk, but remains accessible via SQL on request.In SAP HANA, Data Aging is different than Archiving in the sense that cold data is still kept within the SAP HANA Database and remains accessible via SQL in the very same table as the hot data (yet in another partition). Whereas archived data - strictly read-only - is written to an archive file and deleted from the database and needs additional access paths (address information or archive indexes) to be read. Aging targets the main memory footprint reduction whereas archiving is the basis for ILM, covering the full life cycle up to the destruction of information. Data Aging offers you the option of moving operationally less relevant data within a database so as to gain more working memory. You use the relevant SAP applications, particularly data aging objects to move data from the current area to the historical area. The move influences the visibility when data is accessed. This means that you can perform queries of large amounts of data in current area in a shorter time. This also means that careful testing is required to ensure that the change of visibility is in line with the business processes requirements.

?What is hot data??

Current data is the data relevant to the operations of application objects, needed in day-to day-business transactions. The application logic determines when current data turns historical by using its knowledge about the object’s life cycle. The application logic validates the conditions at the object level from a business point of view, based on the status, execution of existence checks, and verification of cross-object dependencies.?Examples of current data: Open FI items, items cleared only a few months ago, undelivered purchase orders, sales documents of sales cycle that is still in progress, documents for an ongoing project, and IDocs that need to be processed.?

. What is cold data??

Historical data is data that is not used for day-to day-business transactions. By default, historical data is not visible to ABAP applications. It is no longer updated from a business point of view. The application logic determines when current data turns historical by using its knowledge about the object’s lifecycle. The application logic validates the conditions at object level from a business point of view, based on the status, executing existence checks, and verifying cross object dependencies. Historical data is stored in historical area.?

?

Examples of historical data: Cleared FI items posted two years prior to the current fiscal year, material documents one period older than the current closed period, processed IDocs, and application logs after X number of days.?

?. How does Data Aging in SAP S/4HANA function technically??The data aging mechanism for ABAP applications is based on a data aging framework provided by SAP NetWeaver ABAP. ABAP developers use this framework to specify the data aging objects that are aged as one unit, to identify the involved tables, and to implement the logic for determining the data temperature. The data temperature is set via an additional data temperature column '_DATAAGING' (type DATA_TEMPERATURE with ABAP date format “YYYYMMDD”), which is added to all participating tables. The data temperature can be used to horizontally partition the application data with time selection partitioning (a.k.a. "aging") on the column '_DATAAGING' for optimizing resource consumption and performance. Only one partition contains the hot data (represented by the value “00000000”) and the other partition(s) contain cold data with different data temperature ranges. By default, only hot data is accessed. As the hot data is located in a separate partition, SAP HANA should only load that partition into memory during normal operation. If required, ABAP developers can set the data temperature context to switch between accessing only hot data, all data, and all data above a specified data temperature. The SAP HANA-specific database shared library (DBSL) in the ABAP server adds a corresponding clause to the SQL statements that are sent to SAP HANA. By adding the clause WITH RANGE RESTRICTION ('CURRENT') to a SQL statement, SAP HANA restricts the operation to the hot data partition only. Instead of 'CURRENT' also a concrete value can be specified. This restricts the operation to all partitions with data temperatures above the specified value. The clause WITH RANGE RESTRICTION ('20100101'), for example, tells SAP HANA to search the hot partition and all cold partitions that contain values greater or equal than '20100101'. Range restriction can be applied to SELECT, UPDATE, UPSERT, DELETE statements and to procedure calls. All other clients that want to access these Data Aging tables with proper filtering, the same generic syntax extension may be used. The application knows which business objects are closed and may hence be moved to cold partitions. Therefore the application actively sets values in this column to a date to indicate that the object is closed and the row shall be moved to the Cold partition(s) during an Aging Run. Since the table is partitioned by the temperature column, the rows are automatically moved then to a cold partition. The move influences the visibility of the data and its accessibility. Several configuration steps/prerequisites need to be administered to be able to execute a Data Aging Run (-> "How and when will the data be moved from the hot partition to the cold partition(s)?").?

?. What Data Aging Objects are currently available??

?In SAP Business Suite on HANA only Basis Documents are available for Data Aging like e.g. Application Log, IDocs, and Workflow. In SAP S/4HANA Basis and Application Documents are available for Data Aging like e.g. FI Document, Material Document, and Billing Document. A complete list of all available objects can be found in the SAP Help Portal Data Aging - Available Data Aging Objects and under ABAP transaction DAGOBJ.?

?

15. What are typical tools to administer Data Aging in SAP S4/HANA??

?

Several ABAP transactions are available for Data Aging. A detailed list can be found in the SAP Help Portal Data Aging - Data Aging Transactions.?

?Additionally Data Aging can be administered via the following Fiori applications:?

Manage Data Aging Groups ? Monitor Data Aging Objects ? Database Table Growth?

? How to enable the data access of cold data in ABAP??

You can enable data access from historical area by using the classes CL_ABAP_SESSION_TEMPERATURE and CL_ABAP_STACK_TEMPERATURE.?

Is it possible to modify the data in cold partitions??

Yes, you can modify historical data. But usually, business complete data is aged, so modifications are exceptionally rare.?

?

18. If cold data is accessed, will the partition(s) be loaded to the in-memory part of SAP HANA??Relevant cold data will be loaded in a buffer cache, which is dedicated memory section and separated from the normal column store memory.?. How and when will the data be moved from the hot partition to the cold partition(s)??The application logic determines when current data turns historical by using its knowledge about the object’s life cycle. The application logic validates the conditions at the object level from a business point of view, based on the status, execution of existence checks, and verification of cross-object dependencies. The data will be moved during a Data Aging Run. To set up an Aging Run several tasks need to be fulfilled upfront:? Determining the data: The application-specific runtime class can be used to determine the data for which Data Aging is intended. The SAP application assigns these runtime classes to the relevant Data Aging object so that the runtime class can be called and processed in a Data Aging run. ? Managing Partitions: To be able to move the data from the HOT partition of the database to the COLD partition(s) according to the specified partitioning objects and partitioning groups, all of the participating tables must be partitioned for Data Aging. For each system, you need to define the partitions for corresponding tables of a Data Aging object (DAGPTM), this setting is not transportable. If the conditions are not fulfilled, the Data Aging run is not started. There should be at least one cold?partition covering todays date and for multiple partitions on one table the intervals can have no gaps. ? Activating Data Aging Objects: After the partitions have been defined, choose transaction Data Aging Objects (DAGOBJ) to activate the Data Aging object. The system runs through various checks on each table belonging to the Data Aging Object so that the Data Aging object can be used for the run. ? (specific Settings for Data Aging Objects) ? Managing Data Aging Groups: Define Data Aging Groups via transaction DAGOBJ -> Goto -> Edit Data Aging Groups and select all Data Aging Objects to be processed in one Group.?For scheduling Data Aging Runs go to transaction DAGRUN and select a Data Aging Group, Maximum Runtime and Start Date/Time to schedule the run. The same transaction can be used to monitor Data Aging Runs as the initial screen shows a list of runs with the details, such as, the Data Aging Groups, Start date/time, Duration, and Job name. Further details on the Data Aging Run can be found in the SAP Help Portal Data Aging – Data Aging Procedure (Runtime).?Will the Data Aging Framework decide to move the data from the hot partition to the cold partition(s) automatically??The application logic determines when current data turns historical by using its knowledge about the object’s life cycle. The application logic validates the conditions at the object level from a business point of view, based on the status, execution of existence checks, and verification of cross-object dependencies. The framework executes the move.?How to create partitions for tables participating in Data Aging??The SAP HANA database offers a special time selection partitioning scheme, also called aging. Time selection or aging allows SAP Business Suite application data to be horizontally partitioned into different temperatures like hot and cold. The partitioning of tables that participate in Data Aging is administered on the ABAP layer (Transaction DAGPTM). Partitioning needs to be done in each system individually as partitions are not transportable. Alternately, Partition Ranges can be maintained in customizing under SAP Netweaver > Application Server > Basis Services > Data Aging > Maintain Partition Ranges. Partition Objects must be defined to combine several tables that should be partitioned together according to the same schema (Transaction DAGPTC). Usually SAP applications suggest partitioning objects that can be applied. All participating tables belonging to the data aging objects and enhancements must be assigned to a partitioning object. A table is assigned to exactly one partitioning object and cannot occur in more than one partitioning object. If the assignment of a table to a partitioning object does not fulfill customers’ requirements, they can create customer-specific partitioning groups without making modifications. Partitioning Groups override the assignment of tables to partitioning objects if a partitioning object as well as a partitioning group contains the same table. The partitioning gets active only after the next Data Aging Run.?

What does quick partitioning mean??

Initially all Tables participating in Data Aging have an additional column _DATAAGING, which is the basis for the time selection partitioning. The default value of this column is ‘00000000’. When the time selection (=RANGE) partitioning is performed on this column, all records will remain in the hot partition - no data has to be moved. The cold partitions (=new time RANGEs) will be added empty one by one - again no data has to be moved during partitioning itself. Only during the Data Aging Run column _DATAAGING is filled and the aged data will be moved from the hot partition to the cold partition(s).?

?

. Is it possible to add a different partitioning on top of Data Aging??It is possible to add a partitioning on top of Data Aging/after having introduced Data Aging. However, the preferred way is to start the other way round, in order to not move around massive amounts of aged data again.?

?So it is highly recommended to start with Hash (or Range) on first level, and then add the time selection partitioning using the quick-partitioning possibility on second level. Think upfront about the right partition size, as the second level partitioning will split the data further into smaller chunks.?Nevertheless, see the following example for adding a HASH partitioning after having introduced Data Aging already:?CREATE COLUMN TABLE TAB_DATAAGING (A INT, B INT, _DATAAGING NVARCHAR(8)) WITH PARAMETERS('PARTITION_SPEC' = 'RANGE[TIME SELECTION: PAGED ATTRIBUTES, NO UNIQUE CHECK] _DATAAGING 00000000,20100101-20110101');?To add additional 1-level HASH 2 partitioning with key column A on top of the Data Aging RANGE partitioning use:?ALTER TABLE TAB_DATAAGING WITH PARAMETERS('PARTITION_SPEC' = 'HASH 2 A; RANGE[TIME SELECTION: PAGED ATTRIBUTES, NO UNIQUE CHECK] _DATAAGING 00000000,20100101-20110101' );?It is crucial to always provide all Data Aging sub partitions (RANGES) within the ALTER TABLE statement. Otherwise the re-partitioning can fail with errors. See SAP Note 2044468, Q 35. Which errors can happen in relation to partitioning? for details.?

?What re-partitioning possibilities exist for aging/TIME SELECTION partitioned tables??With SAP HANA 2 SPS03 some new options have been introduced to convert/re-partition TIME SELECTION partitioned tables. See Time Selection Partitioning (Aging) for details.?In former SAP HANA revisions it was already possible to re-partition aging partitioned tables e.g. from HASH/Aging(RANGE) to another HASH/Aging(RANGE).?Example:?Table TAB_DATAAGING is 1-level HASH 2 partitioned on column B and 2-level Aging (RANGE) partitioned with the hot partition (00000000) and one cold partition (20100101-20110101)?CREATE COLUMN TABLE TAB_DATAAGING (A INT, B INT, _DATAAGING NVARCHAR(8)) WITH PARAMETERS('PARTITION_SPEC' = 'HASH 2 B; RANGE[TIME SELECTION: PAGED ATTRIBUTES, NO UNIQUE CHECK] _DATAAGING 00000000,20100101-20110101');?To change the 1-level HASH partition KEY from B to A use:?ALTER TABLE TAB_DATAAGING WITH PARAMETERS('PARTITION_SPEC' = 'HASH 2 A; RANGE[TIME SELECTION: PAGED ATTRIBUTES, NO UNIQUE CHECK] _DATAAGING 00000000,20100101-20110101');?It is crucial to always provide all Data Aging sub partitions (RANGES) within the ALTER TABLE statement and not only use:?ALTER table TAB_DATAAGING partition by hash (A) partitions 2;?Otherwise the re-partitioning will fail with one of the following errors:?SAP DBTech JDBC: [7]: feature not supported: not allowed on table having page loadable column?

?

SAP DBTech JDBC: [2048]: column store error: fail to alter partition: [2593] Error during split/merge operation;If a table is partitioned with Time Selection, it is not allowed to repartition to non Time Selection table except for merging Time Selection range partitions.?

Should Aging partitions be on first level or on second level??

Data Aging Partitioning can only be done on first level without any second level or used as a second-level below a hash- or range-partition. If there is any risk that the remaining amount of records in “current” partition after Data Aging can approach the two billion records limit in the future, choose a two-level approach with aging being the second level. Take future volume growth into consideration as well as residence times for Data Aging.?

? When to create a new partition for the cold data??

We create a new partition range when data to be aged is not covered by an existing partition range in historical area. A new partition range can also be created when the maximum capacity threshold for an existing partition is reached soon.?

?Why is the uniqueness check for cold data switched off in SAP HANA per default??for ABAP applications using the Data Aging Framework, the default is set to “no uniqueness check” by SAP HANA. Unique constraints for primary keys or unique indexes are normally checked across all partitions in case the partitioning is done over the artificial temperature that is not part of the primary key column of a table. This means that the cold partitions have to be accessed whenever data is inserted or updated. To prevent a negative impact on performance, a database application can switch off the uniqueness check for cold data by setting an option in the partition specification. With this relaxation, it is up to the application to either prevent duplicates (e.g. via GUIDs or number ranges) or to cope with duplicates. If uniqueness checks for cold data are switched off, SAP HANA only ensures unique values in the hot partition. Conflicts between records in hot and cold, between two cold partitions, and between records in the same cold partition, are not prevented on database level in that case.? Is there a need to adapt the current backup strategy to include cold partition(s)??No changes need to be done to the current backup strategy as the cold partitions are belonging to the table itself as all other partitions and will be backed up together with the rest of the table.? Is there a need to adapt the current system replication configuration to include cold partition(s)??No changes need to be done to the current system replication configuration as the cold partitions are belonging to the table itself as all other partitions and will be replicated to the replication side together with the rest of the table.?

How to undo a Data Aging Run?

If you want to undo a Data Aging Run and move back cold/aged data to the HOT partition you can use transaction DAGRUN -> Goto -> Undo Mode and schedule a new "undo" Job for a specific Data Aging Object.?To be able to apply Data Aging to your data, you need to fulfill certain requirements regarding the databas