I've recently run into the challenge of managing a really bulky message (over 25 Mb!) in SAP XI, where both source and target are flat structures. So I wondered: why going through the overhead of memory consuming XML conversion? Can I get rid of it in this case? Sure I can!
Benchmark
At the beginning of May 2006 (after writing this blog) I had chance to benchmark this approach, compared to the classic one (IDoc XML - graphical mapping - JMS conversion from XML to flat). Here are the astonishing results. I won't unveil XI box's sizing here, as I don't consider it very interesting. The only thing I'll say is that both tests were made on the same machine, with the same load conditions.
Test data: 60 messages of 4 Mb each (flat size), "dropped in" XI almost simultaneously.
Processing: pure XI (adapters, routing, mapping)
Results:
- classical method: between 45 and 50 minutes
- this method: 3 minutes
Introduction
The process I am about to describe here could seem a kind of XI nature negation, but I tend to consider it rather another possibility to achieve a goal. Well, you may be asking yourself, what is this goal? It' easier said than done.
I have an IDoc going out of SAP R/3 that must be mapped against a flat target structure (yeah, the good old classic one with a 4 bytes record type at the beginning), which has to be sent to an MQ queue through JMS Adapter. Unfortunately this IDoc is bulky: its size can easily reach 25 Mb, if weighed flat (imagine when converted into XML!). We all know that managing this kind of message in XI with a mapping that takes place in the J2EE stack (be that graphical, Java or XSLT) can be dangerous, and make our XI box sit down and have coffee&cigarette... ;-)
Flow
So I first investigated the IDoc tunneling technique (also described in this Michal's weblog), sadly realizing that it's only possible when both sender and receiver are SAP systems exchanging the same IDoc with no mapping at all.
Then the weird solution came up to my mind:
-
R/3 system dumps the IDoc with a file port (in place of the canonical tRfc port) in flat format
-
XI reads the flat IDoc with a sender file adapter with no content conversion
-
data are mapped with an ABAP mapping, thus avoiding any unnecessary JCo data flow between the two stacks
-
a receiver JMS adapter writes data to the queue with no content conversion
Assumptions
I assume you are familiar enough with WE20 (Partner Profiles) and WE21 (Port definition) in SAP R/3, so I won't go into the details of it. I also assume you're able to configure both communication channels (file sender and JMS receiver), as it should also be easier than ever, considering my technique allows to cut short with them (remember: no content conversion). The only thing that needs to be mentioned is probably how to enable ABAP mapping (which comes disabled in a brand new XI installation): you need to change something in the Exhange Profile and reboot your J2EE stack. For this purpose refer to SAP documentation.
Realization
Before drawning into ABAP code, another general concept. You may be wondering where and how I have designed source and target structures... Well, for what concerns IDoc, I create them dinamically by reading XI definition of it, while for legacy target structures it's up to you, but I decided to define ABAP dictionary structures with SE11 trx (you can also have them in an ABAP include as I did for IDoc, see below).
First step is to create an ABAP mapping class, which must implement the standard interface IF_MAPPING (see this documentation chapter for details). For details about ho to create ABAP package, change request and so on, please refer to SAP documentation or have a look at this weblog o' mine, where this steps are given full details.
Here below is a simplified version of my own ABAP mapping, hopefully commented enough to be self-explanatory.
THE ABAP MAPPING CODE HERE
------------------------------------------------------------------------------------------------------------------------------------
*******************************************************************
*******************************************************************
* Very important prerequisite: for ZGUA_IDOCSTR_GENERATE_INCL to
* work correctly, if IDoc was changed in the backend,
* it must be manually deleted (and optionally regenerated) in IDX2.
* No automatic refresh is performed by XI in this case because no
* IDoc adapter is actually involved.
*******************************************************************
*******************************************************************
METHOD if_mapping~execute.
TYPES: idocline(1065) TYPE c.
DATA: tmp TYPE string,
idocline TYPE idocline,
theidoc TYPE TABLE OF string,
edidc TYPE edi_dc40,
edidd TYPE myedidd,
linenr TYPE i,
lv_str TYPE string,
lv_tab TYPE string,
dummy TYPE TABLE OF string.
DATA: lv_trg TYPE string, " target line
lt_trg TYPE TABLE OF string, " target table
lv_trgstr TYPE string. " target table stringfied
* Target structures (DDIC structures)
DATA: ls_cl TYPE zgua_st_cl,
lt_cl TYPE TABLE OF zgua_st_cl,
ls_rm TYPE zgua_st_rm,
lt_rm TYPE TABLE OF zgua_st_rm,
ls_tc TYPE zgua_st_tc,
lt_tc TYPE TABLE OF zgua_st_tc.
FIELD-SYMBOLS: <idocline> TYPE string.
* Infinite loop for real runtime debugging
* DATA: debug. WHILE debug IS INITIAL. ENDWHILE.
* Process start --------------------------------------------------------------
* Convert source from xstring to string
CALL FUNCTION 'ECATT_CONV_XSTRING_TO_STRING'
EXPORTING
im_xstring = source
IMPORTING
ex_string = tmp.
* Create string table at crlf
SPLIT tmp AT cl_abap_char_utilities=>newline INTO TABLE theidoc.
* Last row is dirty??? (bad, but done by xstr2str function), so get rid of it
* DESCRIBE TABLE theidoc LINES linenr.
* DELETE theidoc INDEX linenr.
* Get the control record
READ TABLE theidoc INTO edidc INDEX 1.
DELETE theidoc INDEX 1.
* Check if include with IDoc segments structure has to be regenerated before including it
SUBMIT ZGUA_idocstr_generate_incl
WITH idoctyp = edidc-idoctyp
WITH cimtyp = edidc-cimtyp
WITH port = edidc-sndpor
AND RETURN.
INCLUDE zi_my_std_or_cust_idoc/.
* Collect data of data segments ----------------------------------------------------
LOOP AT theidoc ASSIGNING <idocline>.
MOVE <idocline> TO edidd.
seg_e2_e1 edidd-segnam.
CASE edidd-segnam.
WHEN 'Z1SEGMENT01'.
MOVE edidd-sdata TO ls_z1segment01.
APPEND ls_z1segment01 TO lt_z1segment01.
WHEN 'Z1SEGMENT02'.
MOVE edidd-sdata TO ls_z1segment02.
APPEND ls_z1segment02 TO lt_z1segment02.
WHEN 'Z1SEGMENT03'.
MOVE edidd-sdata TO ls_z1segment03.
APPEND ls_z1segment03 TO lt_z1segment03.
ENDCASE.
ENDLOOP.
** Mapping ------------------------------------------------------------
*
* Notice that in this simple case I'm doing just a "trivial" move-corresp.
* but the mapping logic could be as complicated as needed
LOOP AT lt_z1segment01 INTO ls_z1segment01.
ls_cl-rectype = 'CLIE'.
MOVE-CORRESPONDING ls_z1segment01 TO ls_cl.
ls_cl-crlf = cl_abap_char_utilities=>cr_lf.
APPEND ls_cl TO lt_cl.
ENDLOOP.
LOOP AT lt_z1segment02 INTO ls_z1segment02.
ls_rm-rectype = 'REME'.
MOVE-CORRESPONDING ls_z1segment02 TO ls_rm.
ls_rm-crlf = cl_abap_char_utilities=>cr_lf.
APPEND ls_rm TO lt_rm.
ENDLOOP.
LOOP AT lt_z1segment03 INTO ls_z1segment03.
ls_tc-rectype = 'TECO'.
MOVE-CORRESPONDING ls_z1segment03 TO ls_tc.
ls_tc-crlf = cl_abap_char_utilities=>cr_lf.
APPEND ls_tc TO lt_tc.
ENDLOOP.
* ...
* ... [ MORE MAPPING LOGIC TO COME HERE ]
* ...
* Final steps ----------------------------------------------------------
* Collect target data and put them as fixed length records
trgstr2flatstring lt_trg ls_cl lt_cl.
trgstr2flatstring lt_trg ls_rm lt_rm.
trgstr2flatstring lt_trg ls_tc lt_tc.
* Convert flat string table to string
LOOP AT lt_trg INTO lv_trg.
CONCATENATE lv_trgstr lv_trg INTO lv_trgstr.
ENDLOOP.
* Convert string to xstring
CALL FUNCTION 'ECATT_CONV_STRING_TO_XSTRING'
EXPORTING
im_string = lv_trgstr
IMPORTING
ex_xstring = result.
ENDMETHOD.
------------------------------------------------------------------------------------------------------------------------------------
MACROS CODE HERE
------------------------------------------------------------------------------------------------------------------------------------
* Get segment type from segment definition
DEFINE seg_e2_e1.
segnam = &1.
seglen = strlen( segnam ).
* describe field segnam length seglen in character mode.
seglen = seglen - 3. " get rid of trailing three chars
&1 = segnam(seglen).
replace '2' with '1' into &1
length 3.
END-OF-DEFINITION.
* Put the given target structure or table to the flat strings table
DEFINE trgstr2flatstring.
* We have a table (0..n element)
if not &3 is initial.
loop at &3 into &2.
oneline = &2.
append oneline to &1.
endloop.
else.
oneline = &2.
append oneline to &1.
endif.
END-OF-DEFINITION.
------------------------------------------------------------------------------------------------------------------------------------
The report invoked is the real magic one: it exploits a standard function module behind trx IDX2 to get the IDoc definition and dinamically create an ABAP include with all segments definition.
THE REPORT CODE HERE
------------------------------------------------------------------------------------------------------------------------------------
*&---------------------------------------------------------------------*
*& Report ZGUA_IDOCSTR_GENERATE_INCL
*&
*&---------------------------------------------------------------------*
*&
*&
*&---------------------------------------------------------------------*
REPORT ZGUA_idocstr_generate_incl.
DATA: lv_incname TYPE programm.
DATA: lv_idocdate TYPE d,
lv_idoctime TYPE t,
lv_idocts(14) TYPE n,
lv_incdate TYPE d,
lv_inctime(6),
lv_incts(14) TYPE n,
ls_segdef TYPE idxedsappl,
lt_segdef TYPE TABLE OF idxedsappl,
lv_strname TYPE string,
lv_tabname TYPE string.
* The dynamic internal table stucture
DATA: BEGIN OF seg,
name(30) TYPE c,
BEGIN OF struct,
fildname(8) TYPE c,
abptype TYPE c,
length TYPE i,
END OF struct,
END OF seg.
* The dynamic program source table
TYPES: BEGIN OF incstr,
line(72),
END OF incstr.
DATA: incstr TYPE incstr,
inctabl TYPE STANDARD TABLE OF incstr,
generr TYPE string.
PARAMETERS: idoctyp TYPE edipidoctp,
cimtyp TYPE edipidoctp,
port TYPE idx_port DEFAULT 'SAPSID'.
START-OF-SELECTION.
* -------------------------------------------------------------------------------- *
* Init include name.
CONCATENATE 'ZI_' idoctyp '/' cimtyp INTO lv_incname.
* Refresh IDoc structure, if needed (the function is smart enough :-)
CALL FUNCTION 'IDX_STRUCTURE_GET'
EXPORTING
port = port
doctyp = idoctyp
cimtyp = cimtyp
release = ''
direction = ''
saprel = '46C'
TABLES
edsappl = lt_segdef
* EXCEPTIONS
* no_doctyp = 1
* wrong_rfc_destination = 2
* communication_error = 3
.
* IF sy-subrc <> 0.
* EXIT.
* ENDIF.
* Caching mechanism: is IDoc metadata younger than generated include?
SELECT SINGLE upddate updtime
FROM idxsload
INTO (lv_idocdate, lv_idoctime)
WHERE port = port
AND idoctyp = idoctyp.
CONCATENATE lv_idocdate lv_idoctime INTO lv_idocts.
SELECT SINGLE sdate stime
FROM trdir
INTO (lv_incdate, lv_inctime)
WHERE name = lv_incname.
CONCATENATE lv_incdate lv_inctime INTO lv_incts.
* Not in synch with IDoc: incldue must be regenerated
IF lv_idocts > lv_incts.
SORT lt_segdef BY segtyp pos.
* Create the dynamic internal table definition in the dyn. program
LOOP AT lt_segdef INTO ls_segdef.
AT NEW segtyp.
* Type for segment
CONCATENATE 'types: begin of' ls_segdef-segtyp ','
INTO incstr-line SEPARATED BY space.
APPEND incstr TO inctabl.
ENDAT.
CONCATENATE ls_segdef-fieldname '(' ls_segdef-expleng '),'
INTO incstr-line.
APPEND incstr TO inctabl.
AT END OF segtyp.
CONCATENATE 'end of' ls_segdef-segtyp '.'
INTO incstr-line SEPARATED BY space.
APPEND incstr TO inctabl.
* Internal table
CONCATENATE 'lt_' ls_segdef-segtyp INTO lv_tabname.
CONCATENATE 'data:' lv_tabname 'type table of' ls_segdef-segtyp '.'
INTO incstr-line SEPARATED BY space.
APPEND incstr TO inctabl.
* Working area
CONCATENATE 'ls_' ls_segdef-segtyp INTO lv_strname.
CONCATENATE 'data:' lv_strname 'type' ls_segdef-segtyp '.'
INTO incstr-line SEPARATED BY space.
APPEND incstr TO inctabl.
ENDAT.
ENDLOOP.
* Create and generate the dynamic include
INSERT REPORT lv_incname FROM inctabl PROGRAM TYPE 'I'.
* GENERATE REPORT lv_incname MESSAGE generr.
COMMIT WORK AND WAIT.
ENDIF.
END-OF-SELECTION.
------------------------------------------------------------------------------------------------------------------------------------
Conclusion
You may find the whole stuff a bit crazy maybe, but I can guarantee that performance is terrific... 1,5 second mapping time for a 20 Mb file in a DEV box!
Last but not least: in my case the source IDoc is custom, so I developed it so that on each segment I always have key field(s) of the parent, just for convenience. If this is not your case, segments numbering and hierarchy is anyway present in the IDoc flat representation and you may need to improve my code in order to handle them in your ABAP mapping.
In the next weblog I will show you how I realized a split of the resulting message into several smaller and packaged messages in order to overcome an MQ limitation of JMS message size.
