Delta Management

Delta Load Management Framework Overview 

CAF and SAP BW integration supports delta loading for DataSources created by entity and application service extractor methods. When working with applications with large data volumes, it is logical to prevent long loading times and unnecessary locks on the database by only loading new or modified data records into SAP BW.

Features

Generic delta management works as follows:

 

       1.      A data request is combined with particular selection criteria  in an InfoPackage and is to be extracted in delta mode.

       2.      The request is sent to the source system and then received by the SAPI (service application programming interface) request broker.

       3.      The generic delta management is initiated before the data request is transferred to the extractor corresponding to the DataSource. This enhances the selection criteria of the request in accordance with the update mode of the request. If the delta-relevant field is a timestamp, the system then adds a time interval to the selection criteria. 

Delta management can take the lower limit from the last extraction. The upper limit is taken from the current time. For example, date of application server minus a safety margin (SY-DATE, SY-TIME).

       4.      The enhanced request is transferred to the extractor. The update mode is ‘translated’ by the generic delta management into a selection criteria. For this reason, the update mode is set first to full.

       5.      At the end of the extraction, the system informs generic delta management that the pointer can now be set to the upper limit of the previously returned interval.

You can find a description of this transfer process in the figure below.

Structure

Delta Fields

The delta-relevant field of the extract structure meets one of the following criteria:

●      Field type is timestamp.

○       New records that are to be loaded into BW using a delta upload each have a higher entry in this field than records that have already been loaded.

○       The same criteria applies for new records as in the case of a timestamp field.

●      Field type is not timestamp. This case is only supported for SAP Content DataSources. At the start of delta extraction, the maximum value to be read must be returned using a DataSource-specific exit.

You can use special data field to achieve more reliable delta loading from different source systems. They are integrated into the delta management framework. They are:

●      Safety interval upper limit

●      Safety interval low limit

Safety Interval Upper Limit

The upper limit for safety interval contains the difference between the current highest value at the time of the delta or initial delta extraction and the data that has actually been read. If this value is initial, records that are created during extraction cannot be extracted.                      

A timestamp is used to determine the delta value. The timestamp that was read last stands at 12:00:00. The next data extraction begins at 12:30:00. The selection interval is therefore 12:00:00 to 12:30:00. At the end of the extraction, the pointer is set to 12:30:00.                                       

This transaction is saved as a record. It is created at 12:25 but not saved until 12:35. As a result, it is not contained in the extracted data and the timestamp means the record is not included in the subsequent extraction.                                                                            

To avoid this discrepancy, the safety margin between read and transferred data must always be longer than the maximum time the creation of a record for this DataSource can take (for timestamp deltas), or a sufficiently large interval (for deltas using a serial number).

Safety Interval Lower Limit

The lower limit for safety interval contains the value that needs to be taken from the highest value of the previous extraction to obtain the lowest value of the following extraction.

A timestamp is used to determine the delta. The master data is extracted. Only images taken after the extraction are transferred and overwrite the status in BW. Therefore, with such data, a record can be extracted more than once into BW without too much difficulty.

Taking this into account, the current timestamp can always be used as the upper limit in an extraction and  the lower limit of the subsequent extraction does not immediately follow on from the upper limit of the previous one. Instead, it takes a value corresponding to this upper limit minus a safety margin.

This safety interval needs to sufficiently large so that all values that already contain a timestamp at the time of the last extraction, but which have yet to be read (see type 1), are now contained in the extraction. This implies that some records will be transferred twice. However, due to the reasons outlined previously, this is irrelevant.

You should not fill the safety intervals fields with an additive delta update, as duplicate records will invariably lead to incorrect data.

It is not necessary to set safety intervals for DataSources used in CAF and SAP BW integration.