Proof-of-concept ABAP Program to extract sales order data to an Elasticsearch server as seen on http://ceronio.net

zslsord_extr_to_es.abap

*--------------------------------------------------------------------*
* Example program to extract data to Elasticsearch
*--------------------------------------------------------------------*
* Be sure to read the accompanying article:
* http://ceronio.net/2015/02/sap-business-intelligence-with-elasticseach-and-kibana/
*&---------------------------------------------------------------------*
*& WARNING!: This program is just a proof of concept. For a proper
*& production-quality extraction program, you would:
*& 1. Use a database cursor for selection
*& 2. Do bulk updates to Elasticsearch for better performance
*& 3. Cater for delta/full selection (and additional criteria)
*& 3. Use language-dependent texts everywhere
*& 4. Investigate the quickest way to get values into the (stab) table
*& 5. Possibly normalize the data in Elasticsearch?
*& 6. etc.
*& IN ADDITION, YOU WOULD HAVE TO SET MOST STRING FIELDS TO "NOT ANALYZED"
*& IF YOU WANT TO REPORT ON THEM!!!
*&---------------------------------------------------------------------*

report zslsord_extr_to_es.

* Set this to point to your Elasticsearch installation
data: gv_es_url type string value 'http://10.0.0.2:9200'.

* For JSON transformation:
data gr_writer type ref to cl_sxml_string_writer.
data gv_json type xstring.

* Data needed for HTTP communication
data: gv_client type ref to if_http_client.
data: gv_request type ref to if_http_request.
data: gv_response type ref to if_http_response.
data: gv_code type i.
data: gv_reason type string.

* For RTTI of structure with fields and dynamic transfer of data
data: gr_sdesc type ref to cl_abap_structdescr.
data: gt_comp type abap_component_tab.
data: gs_comp type abap_componentdescr.
data: gv_fname type string.
field-symbols: <field> type any.

* Because we don't want an unnecessary root element in our JSON and we want all
* the keys lower case, we use (stab)
data: gt_stab type abap_trans_srcbind_tab.
data: gs_stab type abap_trans_srcbind.

* Definition of the data we want to retrieve from the database
data: begin of gs_order,
        vbeln type vbak-vbeln,
        erdat type vbak-erdat,
        erzet type vbak-erzet,
        audat type vbak-audat,
        vbtyp type vbak-vbtyp,
        auart type vbak-auart,
        vkorg type vbak-vkorg,
        vtweg type vbak-vtweg,
        spart type vbak-spart,
        bstnk type vbak-bstnk,
        kunnr type vbak-kunnr,
        name1 type kna1-name1,
        posnr type vbap-posnr,
        matnr type vbap-matnr,
        matkl type vbap-matkl,
        pstyv type vbap-pstyv,
        netwr type vbap-netwr,
        waerk type vbap-waerk,
        kwmeng type vbap-kwmeng,
        vrkme type vbap-vrkme,
      end of gs_order.
data: gt_order like table of gs_order.

* Get a description of the above data structure for its fields
gr_sdesc ?= cl_abap_structdescr=>describe_by_data( gs_order ).
gt_comp = gr_sdesc->get_components( ).

* Data selection which must be maintained to correspond to the above data definition
select k~vbeln k~erdat k~erzet k~audat k~vbtyp k~auart k~vkorg k~vtweg k~spart k~bstnk k~kunnr k1~name1
       p~posnr p~matnr p~matkl p~pstyv p~netwr p~waerk p~kwmeng p~vrkme
  into table gt_order
  from vbak as k
  join vbap as p
    on k~vbeln = p~vbeln
  join kna1 as k1
    on k~kunnr = k1~kunnr.
*  up to 2000 rows.

* Create HTTP client
call method cl_http_client=>create_by_url
  exporting
    url                = gv_es_url
  importing
    client             = gv_client
  exceptions
    argument_not_found = 1
    plugin_not_active  = 2
    internal_error     = 3
    others             = 4.
if sy-subrc <> 0.
* Here you can report the problem somewhere, like a list and exit the program
  write: / 'Failure instantiating HTTP client'.
  exit.
endif.

* Get handles on the request and response obejcts
gv_request = gv_client->request.
gv_response = gv_client->response.

loop at gt_order into gs_order.

* Fill the stab table with values of the current record
  clear gt_stab.
  loop at gt_comp into gs_comp.
    assign component gs_comp-name of structure gs_order to <field>.
    if sy-subrc = 0.
      gs_stab-name = to_lower( gs_comp-name ).
      get reference of <field> into gs_stab-value.
      append gs_stab to gt_stab.
    endif.
  endloop.

* Transform data into JSON string
  gr_writer = cl_sxml_string_writer=>create( type = if_sxml=>co_xt_json ).
  call transformation id
    source (gt_stab)
    result xml gr_writer.
  gv_json = gr_writer->get_output( ).

* Set up HTTP request to post the data to ElasticSearch
  gv_request->set_method( 'POST' ).

  call method cl_http_utility=>set_request_uri
    exporting
      request = gv_request
      uri     = |/orders/order_item/{ gs_order-vbeln }{ gs_order-posnr }|.

  gv_request->set_data( gv_json ).

  call method gv_client->send
    exceptions
      http_communication_failure = 1
      http_invalid_state         = 2
      http_processing_failed     = 3
      http_invalid_timeout       = 4
      others                     = 5.
  if sy-subrc <> 0.
* Here you can report the problem somewhere, like a list
    write: / 'Failed exporting (HTTP send)', gs_order-vbeln, gs_order-posnr.
    continue.
  endif.

  call method gv_client->receive
    exceptions
      http_communication_failure = 1
      http_invalid_state         = 2
      http_processing_failed     = 3
      others                     = 5.
  if sy-subrc <> 0.
* Here you can report the problem somewhere, like a list
    write: / 'Failed exporting (HTTP receive)', gs_order-vbeln, gs_order-posnr.
    continue.
  endif.

  call method gv_response->get_status
    importing
      code   = gv_code
      reason = gv_reason.
  if gv_code >= 400.
    write: / 'Failed exporting (HTTP response)', gv_reason, gs_order-vbeln, gs_order-posnr.
* Here you can report the problem somewhere, like a list
  endif.

endloop.