ESQL for creating the bitmap for an 'unpacked' ISO8583 message

readme.md

If you have to create ISO8583 credit/debit card messages in IBM Integration Bus, but the messages use 'unpacked' bitmaps instead of the more usual 'packed' bitmaps, then read on ...

ISO8583 messages consist of a number of optional fixed-length or prefixed-length fields without initiators. The presence of a given data field in the message is indicated by the setting of a corresponding bit in a bitmap. The bitmaps occur near the start of the message, and can either be in 'packed' format where each bitmap is encoded as 8 bytes of binary data (8 bits per byte), or in 'unpacked' format where each bitmap is encoded as 16 text characters (4 bits per character).

The IBM Integration Bus v9 sample for ISO8583 contains message flows that parse and create ISO8583 messages with 'packed' bitmaps. The 'create' flow contains ESQL code that automatically creates the bitmap elements and sets them correctly according to the existence of the data elements in the IIB message tree. The code to do that is straightforward as each bit is its own element and just needs setting to 0 or 1 depending on the existence of its corresponding data element.

The IIB v9 sample for ISO8583 can be used as the basis for message flows that parse and create ISO8583 messages with 'unpacked' bitmaps. The DFDL schemas for 'unpacked' ISO8583 are available on the DFDLSchemas GitHub site. The 'parse' message flow just needs to be updated to use the 'unpacked' schemas instead of the 'packed' schemas. The 'create' flow needs the same update, but it also needs new ESQL code to create the bitmap elements - and that's the tricky bit.

In an 'unpacked' bitmap, each group of 4 bits is treated as a hex integer which is then converted to a character. Each bit in a group of 4 is assigned a numeric value, 8, 4, 2, 1 from left to right. For example, bit 001 has value 8, bit 002 has value 4, bit 003 has value 2 and bit 004 has value 1. If data elements 002 and 004 exist in the tree but data elements 001 and 003 do not, then the numeric value of the first group of 4 bits is 5 (0 + 4 + 0 + 1 = 5). The number is then converted to hex and then to a character, giving '5'. Similarly, if data elements 005 and 006 exist in the tree but data elements 007 and 008 do not, then the numeric value of the second group of 4 bits is 12 (8 + 4 + 0 + 0 = 12). The number is then converted to hex and then to a character, giving 'C'.

The ESQL in the Gist creates the 'unpacked' bitmap, and may be used to replace the ESQL in the IIB sample's 'create' flow. In a typical ISO8583 message only a small number of the 128 fields are used, so for efficiency the ESQL code walks the data elements in the tree, updating the bitmap as it goes, rather then walking the bitmap elements then looking for each data element. The file extension must be renamed to .esql after download.

Please observe the accompanying license file.

gistfile1.txt

Copyright (c) 2014 International Business Machines Corporation

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

XML_to_ISO8583_unpacked.sql

CREATE COMPUTE MODULE XML_to_ISO8583_unpacked
    CREATE FUNCTION Main() RETURNS BOOLEAN
    BEGIN
        SET OutputRoot.Properties = InputRoot.Properties;
        SET OutputRoot.Properties.MessageType = '{}:ISO8583_1987_Unpacked';
        CREATE LASTCHILD OF OutputRoot DOMAIN('DFDL');
        DECLARE InRef REFERENCE TO InputRoot;
        CALL RemoveUnrequiredXMLAttributes(InRef);
        SET OutputRoot.DFDL = InRef.XMLNSC;
        CREATE NEXTSIBLING OF OutputRoot.DFDL.ISO8583_1987_Unpacked.MTI_MessageOrigin NAME 'PrimaryBitmap';
        DECLARE DFDLPointer REFERENCE TO OutputRoot.DFDL.ISO8583_1987_Unpacked.PrimaryBitmap;

        CALL PopulatePrimaryBitmap(DFDLPointer);

        DECLARE SecondaryBitmap BOOLEAN FALSE;
        IF CAST(SUBSTRING(FIELDNAME(OutputRoot.DFDL.ISO8583_1987_Unpacked.*[<]) AFTER '_') AS INT) > 64 THEN
            SET SecondaryBitmap = TRUE;
        END IF;

        IF SecondaryBitmap THEN
            -- This means a SecondaryBitmap will be required
            CREATE NEXTSIBLING OF OutputRoot.DFDL.ISO8583_1987_Unpacked.PrimaryBitmap NAME 'SecondaryBitmap';            
            MOVE DFDLPointer TO OutputRoot.DFDL.ISO8583_1987_Unpacked.SecondaryBitmap;
            CALL PopulateSecondaryBitmap(DFDLPointer);

            -- SecondaryBitmap is present, so set logical Bit001
            SET OutputRoot.DFDL.ISO8583_1987_Unpacked.PrimaryBitmap.Bits001to004 = '8';
        ELSE
            MOVE DFDLPointer TO OutputRoot.DFDL.ISO8583_1987_Unpacked.PrimaryBitmap;
            -- SecondaryBitmap is not present. Logical Bit001 will already be initialised to 0.
        END IF;

        -- Having created the Bitmap structures with all bits set to 0, now set the Bit values ...
        CALL UpdateBitmapsBasedOnAvailableFields(DFDLPointer);

        RETURN TRUE;
    END;

    CREATE PROCEDURE GenerateThreeDigitIndex(IN IndexInt INTEGER, OUT IndexChar CHAR)
    BEGIN
        SET IndexChar =
        CASE
            WHEN IndexInt < 10 THEN ('00' || CAST(IndexInt AS CHAR))
            WHEN IndexInt < 100 THEN ('0' || CAST(IndexInt AS CHAR))
            ELSE CAST(IndexInt AS CHAR)
        END;
    END;

    CREATE PROCEDURE RemoveUnrequiredXMLAttributes(INOUT InRef REFERENCE)
    BEGIN
        SET InRef.XMLNSC.*:XmlDeclaration = NULL;
        DECLARE NumberRootAttributes INTEGER CARDINALITY(InRef.XMLNSC.ISO8583_1987_Unpacked.(XMLNSC.Attribute)*[]);
        DECLARE I1 INTEGER 1;
        WHILE I1 <= NumberRootAttributes DO
            SET InRef.XMLNSC.ISO8583_1987_Unpacked.(XMLNSC.Attribute)*[1] = NULL;
            SET I1 = I1 + 1;
        END WHILE;
    END;

    CREATE PROCEDURE PopulatePrimaryBitmap(INOUT DFDLPointer REFERENCE)
    BEGIN
        DECLARE I2 INTEGER 1;
        DECLARE IndexChar1 CHAR;
        DECLARE IndexChar2 CHAR;
        -- Create unpacked PrimaryBitmap structure, with all bits set to 0
        WHILE I2 < 65 DO
            CALL GenerateThreeDigitIndex(I2,IndexChar1);
            CALL GenerateThreeDigitIndex(I2+3,IndexChar2);
            CREATE LASTCHILD OF DFDLPointer NAME ('Bits' || IndexChar1 || 'to' || IndexChar2) VALUE '0';
            SET I2 = I2 + 4;
        END WHILE;
    END;

    CREATE PROCEDURE PopulateSecondaryBitmap(INOUT DFDLPointer REFERENCE)
    BEGIN
        DECLARE J INTEGER 65;
        DECLARE IndexChar1 CHAR;
        DECLARE IndexChar2 CHAR;
        WHILE J < 129 DO
            -- Create unpacked SecondaryBitmap structure, with all bits set to 0
            CALL GenerateThreeDigitIndex(J,IndexChar1);
            CALL GenerateThreeDigitIndex(J+3,IndexChar2);
            CREATE LASTCHILD OF DFDLPointer NAME ('Bits' || IndexChar1 || 'to' || IndexChar2) VALUE '0';
            SET J = J + 4;
        END WHILE;
    END;

    CREATE PROCEDURE UpdateBitmapsBasedOnAvailableFields(INOUT DFDLPointer REFERENCE)
    BEGIN
        -- Loop over the data fields which are present to update the Bitmaps ...
        DECLARE BitName CHAR;
        DECLARE BitMapName CHAR 'PrimaryBitmap';
        DECLARE CurrentFieldName CHAR;
        DECLARE CurrentFieldNumber INT;
        DECLARE CurrentFieldBase INT;
        DECLARE IndexChar1 CHAR;
        DECLARE IndexChar2 CHAR;
        DECLARE Value INT;
        DECLARE CharValue CHAR;
        
        MOVE DFDLPointer NEXTSIBLING;
        WHILE LASTMOVE(DFDLPointer) DO
            -- Get field name (eg PrimaryAccountNumber_002)
            SET CurrentFieldName = FIELDNAME(DFDLPointer);
            -- Get the numeric part (eg 002)
            SET CurrentFieldNumber = CAST(SUBSTRING(CurrentFieldName AFTER '_') AS INT);
            -- Work out which BitsXXXtoYYY field contains the bit for the field
            -- (eg the bit for _002 is in Bits001to004)
            SET CurrentFieldBase = FLOOR((CurrentFieldNumber - 1) / 4) * 4 + 1;
            -- Calculate the value to represent the bit as set - 8, 4, 2 or 1
            -- (eg, _002 is 2nd bit in Bits001to004 so has value 4)
            SET Value = CAST(POWER(2, 3 - MOD(CurrentFieldNumber - 1, 4)) AS INT);
            -- Compute the correct BitsXXXtoYYY name
            CALL GenerateThreeDigitIndex(CurrentFieldBase, IndexChar1);
            CALL GenerateThreeDigitIndex(CurrentFieldBase + 3, IndexChar2);
            SET BitName = 'Bits' || IndexChar1 || 'to' || IndexChar2;
            IF CurrentFieldNumber > 64 THEN
                SET BitMapName = 'SecondaryBitmap';
            END IF;
            -- Add the bit value to the existing value
            SET CharValue = OutputRoot.DFDL.ISO8583_1987_Unpacked.{BitMapName}.{BitName};
            SET CharValue = '000000000000000' || CharValue; -- Make 16 long for CAST to INT
            SET Value = CAST(CAST(CharValue AS BLOB) AS INT) + Value;
            SET CharValue = CAST(CAST(Value AS BLOB) AS CHAR);
            SET CharValue = UPPER(SUBSTRING(CharValue FROM 18 FOR 1)); -- Extract n from X'000000000000000n'
            SET OutputRoot.DFDL.ISO8583_1987_Unpacked.{BitMapName}.{BitName} = CharValue;

            MOVE DFDLPointer NEXTSIBLING;
        END WHILE;        
    END;

END MODULE;