Hello SCN-Community,

as mentioned in my first blog „BRFplus in Big Data scenarios“, I want to share some other topics around processing mass amount of data. Faster as I expected I've solved a problem at a dialog for operational analysis. It’s about topic 3 of my last blog:

“Explorative search techniques to determine anomalies in invoices – how could we support the end user during his daily work? Can we use ALV grid to analyse data models with master-detail relationships?”

Introduction

We take place in the already discussed business case – Processing of a huge amount of invoices sent to a statutory health insurance company.

A set of checks is being processed every time new invoices arrive. Every check focuses on a specific aspect or constellation of an invoice which produce clarification cases that have to be analyzed by the end user. But these checks were developed due to a specific concept or aspect. So they target on the most common known anomalies in these invoices. What about other constellation which have not been considered before?

One solution to this problem may be a dialog in which the user can do some kind of pre analytics to seek for new patterns to identify anomalies, errors or even detect possible fraud cases among those invoices.

Requirements

This task sound easy. You take the invoices to look at and put them into an ALV-Grid. Due to that you can use all filters and sort criteria of the ALV-Grid. Also you are able to save your “query” for reuse as a user-specific layout.

At this point we'll need to raise some important aspects:

• We have to deal with approximately 28.000.000 invoices a year for a single health insurance company. This amount of invoices does not fit into an ALV-Grid.
• Each invoice is supported by some medical documentation. We have to be able to search in invoices for existence of a specific medical treatment, surgery and so on.
• These filter criteria have to be saved like usual ALV variants due to reuse. Unfortunately we can’t use the normal ALV variants since we deal with master/detail views. So we have to develop different techniques which are major topic of this blog entry.
• The user must be able to see all data of a specific invoice even the detail data.
• Our solution cannot be addressed to HANA-specific techniques only. We have to support a kind of HANA-readiness without facing solution with a two different lines of code.

In this case we talk about a generic solution. With this tool the user is able to do an explorative search about billing errors in our scenario without moving into a Business Warehouse.

Introducing our UI prototype

In fact of the matter we cannot build upon HANA-specific features only we chose a solution with a standard ALV-Grid. We split the screen of the dialog into two sections. A master view with one ALV-Grid for displaying the invoices and a detail view containing three ALV-Grids corresponding to three kinds of supporting documentation.

If you double click an invoice all detail data is being loaded into the corresponding ALV-Grids.

Handling of mass data

We decided not to load all invoices into the ALV-Grid to avoid overloading the master view. During an explorative search you are not interested in each single invoice. Rather you have to find another filter to reduce the search result – to give your interest boundaries. For now we talk about an extract of the data (named as “Ausschnitt” in the screenshots) and the whole data of the underlying table (named as “Gesamtdaten”). The size of an extract is defined by the user. With the size of the extract the user decides how many invoices have to be displayed and thus are being transferred to the front-end. It’s a constraint to define the maximum amount of data. To gain this kind of truncation we added a new button to the ALV-Grid.

“Sätze” stands for rows or even invoices which defines a kind of truncation. The given filter and sort criteria of an ALV-Grid operates on the internal data table of the grid – so it only addresses the extract but operates very quickly.

If we want to reduce our search result effectively the filter and sort criteria are to operate on the whole underlying data table. Due to that we added some other functionality to the grid to control the ALV filter and sort actions.

• „Sortierung Ausschnitt“ : Sort the extract at the front-end (ALV-Grid standard)
• „Sortierung Gesamtdaten" : Sort the data in backend (database) and rebuild the internal table (of the ALV)
• „Filterung Ausschnitt“ : Filter the extract at the front-end (ALV-Grid standard)
• „Filterung Gesamtdaten“ : Filter the data in backend and rebuild a new internal table on that result (of the ALV)

With the help of these new functions we are able to search for various facts in a huge amount of data. If the amount of data to be displayed is too big, we’ll see due to the truncation only the tip of the iceberg.

Detail filter – How to deal with them?

The search by detail data was a bigger problem we had to deal with. An ALV filter only operates on its own data context. But the detail data are separated in additional ALV-Grids. So we built a popup with additional filter criteria addressing the keys of the medical documentation we have to deal with.

The constraints of the detail filter are handled by a manager class of our master view. Such a constraint is defined as a single Select-Option. This manager class builds upon these constraints a SQL query. We built all combinations of master to detail view as an EXISTS clause for each detail relationship. We count 8 valid combinations (guess 2^3 ) so the SQL queries can be hard coded in the manager class.

Another problem was to take care of a detail filter while saving an ALV-layout in the master view. If the user defined a detail filter it has to be saved together with the normal layout. We solved this by creating an add-on table holding the detail filter under the same key of the ALV-layout. To do this we added some functionality in to the event handling of the ALV-Layout button. When the user loads a layout the detail filter is loaded too so the original search result is reproduced.

With these extensions to the ALV-Grid we realized a master to detail relationship with standard ABAP coding to allow the user to search explorative in the whole data volume.

Advanced techniques

With this solution we are not at the end. Due to use of Select-Option to define a detail filter we are not able to do all kinds of filtering. Suggest we want to keep track of all invoices containing the medical treatment A und B except C or D. With a Select-Option we are only to select via OR and not via AND.

Additionally we want to transform such a saved layout (with detail filter) to a customer defined check function which is being processed when a new invoice arrives. Under this condition we were faced with a major ABAP limit:

The 8 hard coded EXISTS clauses no longer match the new requirements. So we have to deal with a dynamic WHERE clause which does not support sub queries currently. We hit a dead end.

Without trying to use HANA-specific functionalities the only solutions we mentioned were the use of native-SQL or the generation of ABAP coding. We decided to choose the way of code generation. Our manager class is able to transform a saved ALV-layout with an applied detail filter into a SQL query. This built query gets embedded into a generated ABAP class which could be used in our operational analytics framework.

What should be better? – A wish list / Subject to Change

During the development of this dialog (which is based on the ALV-Grid class CL_GUI_ALV_GRID on NetWeaver 7.40 SP4) we also examined the use of the ALV-Grid with IDA (HANA-Grid class CL_SALV_GUI_TABLE_IDA on both NetWeaver 7.40 SP4 and SP6). We noticed the availability of this grid also on none HANA-Systems. That’s a very important fact. This capability may help us to rely on only one codebase. We do not need to separate HANA-only features in another codebase which has to be switched out on none HANA-Systems. Unfortunately the current version of the HANA-grid doesn’t give us the ability to influence the handling of the grid like a normal ALV-Grid does. We need the ability to take care of the grid’s internal table of data storage in none-HANA scenarios. Another solution could be a refresh function by the grid. By calling the grid’s refresh function a new SQL query is being sent with the defined filter and sort criteria to the database. Also we need the ability to define the truncation.

After solving this we are still not ready to use the HANA-grid due to the fact we are not able to manipulate the WHERE-clause. Well, I think in the meantime will be a solution of a master-detail-relationship based on HANA-features for sure. But this doesn’t help us in our case.

Due to our business scenario our solution has to be used by both, HANA and none-HANA customers.

As I described earlier we use code generation to avoid native-SQL. But the code generation doesn't help us in case of the HANA-Grid. Remember we do not have the ability to query the database by our own.

Summary

With our solution we are able to support the user by an explorative search technique. It’s a kind of generic solution which does not rely on further development once it is transported to the customer. Due to the fact the user never switches the system (for instance to jump into a BW system, do the analytics there and jump back to his operative application) he is able to build a check out of the combined filter criteria to integrate it into the already defined set of checks which are processed when a new invoice arrives.

This prototyping is not at the end. Some technical problems must be solved to build other features on top of this approach. But these problems are tough because they depend on needed extensions to OpenSQL (sub queries in dynamic WHERE clauses) or to the HANA-Grid.

The general availability of the HANA-Platform features not only a huge gain of speed but also on additional functionalities to operate on Big Data. But first we have to lift the customers to that platform. I suggest it is only done by building hybrid solutions which take care of the base business cases which run faster and give some view of benefits if run on HANA. To minimize the effort we have to rely on only one codeline which we have to support.