EBCDIC to ASCII in RPG IV

3 Ways to Translate Data in your RPG IV Code

There are now at least 3 ways to translate PC ASCII to IBM i EBCDIC characters. Why do you need to do that? If you're working with 3rd party data or conversing with a webserver, you probably need to do this in RPG IV.

Translating Option 1: The Bad: QDCXLATE/QTBXLATE

About 40 years ago, IBM created an API to translate EBCDIC to ASCII for data stored on Diskettes on System/38 using translation tables. The QTBXLATE and its doppelganger QDCXLATE use translation tables to do the conversions. The two APIs are identical; I do not know why they created a new name for the API. The only reason I can think of is that it may have been moved to another group within IBM and therefore the prefix of the API changes.

This API is still advocated today for ASCII->EBCDIC->ASCII conversions; normally by Developers who either don't care about accuracy or don't know they should use something else. To quick read of the documentation for either of these APIs would give you a clue:

"This API is available for compatibility purposes or user-defined mappings only. Do not use this API in new development; instead, use the iconv API instead."

The RPG IV %XLATE built-in function can do the same thing as QDCXLATE with user-defined "mappings" (so %XLATE is a 4th option if you need it).

Translation Option 2: The Good: iconv

The API named iconv is a CCSID translation API that converts between most CCSIDs. This API requires that you setup a translation environment, do your translations and then close down the translation environment. If you forget to, or simply do not close down the translation environment(s) you created, they go away when your IBM i job ends.

The iconv API is a collection of 3 APIs, and one alternate API. The alternate is a simpler choice and is what I use.

1. iconv_open - Create the translation environment
2. QtqIconvOpen - Alternate to create the translation environment
3. iconv - Perform translation
4. iconv_close - Delete (free) the translation environment.

Using the iconv_open, you establish the translation environment by providing the FROM and TO CCSID values. These CCSIDs are stored in a data structure that needs some very specific initialization. For that reason, IBM created a simplified version named QtqIconvOpen that while still requiring a data structure, is much simpler to initialization. The iconv_open and QtqIconvOpen return a translation "handle" that you need to save and use in subsequent calls to iconv and iconv_close.

The iconv API is passed this translation handle, the target (output) buffer and length, the source (input) buffer and length. It does the translation for you and returns the number of unique translations it performs. But it also updates the output buffer, the output length and the input lengths in a somewhat vague manner. For example, if the output buffer length is 255 and 10 characters are to be translated (the input buffer length is 10) then the output buffer length parameter is updated upon return and is set to 245. Effectively giving you a "bytes remaining" value. If you subtract that returned value from the original output length parameter value, you get the byte length of the output.

You can call iconv over and over during your program and it'll do the conversion. You only need to do the iconv_open or QtqIconvOpen call once per program and then iconv as often as necessary.

When you're finished, you should call the iconv_close API, passing it the translation handle to shutdown, free up the translation environment.

The disadvantage of using iconv is that is is very easy to call from the C or C++ languages, but in RPG IV, it is a bit vexing. I've created a C service program wrapper for iconv that you can easily call from RPG IV. In my opinion, these wrappers have a simplified RPG-friendly interface.

Translation Option 3: The Best: RPG IV Built-in Translation

Believe it or not, EBCDIC to ASCII translation is built-in to RPG IV and has been for years. So don't worry too much about learning iconv, and please dump your QDCXLATE calls in favor of this option instead.

RPG IV fields support the CCSID keyword. This causes RPG IV to treat the data in the field as if it is in the CCSID designated. How do you convert EBCDIC to ASCII in RPG IV?

dcl-s  ebcdicData char(255) inz('Hello World');
dcl-s  asciiData  char(255) CCSID(1208); 

   // Convert EBCDIC to ASCII
  asciiData = ebcdicData;          

I'm sorry, is that too easy? ??

It works the other way around as well. If you get some data from a Web Service, or from the HTTP Server itself, you can easily convert between your that ASCII data and your Job's CCSID. Remember, there are many "ASCII CCSID" codes. But here's the most often used:

• 1208 - UTF-8
• 1200 - UTF-16
• 819 - PC ASCII (legacy, single-byte version of ASCII)

While full UTF-16 support isn't there yet, you can convert from UTF-16 to UTF-8 and then to your job's EBCDIC CCSID. This long-way-around seems to work.

Caveat: A reader pointed out that IBM added a UTF-8 CCSID figurative constant. Instead of specifying CCSID(1208) can may specify CCSID(*UTF8) as well. (Thanks Ringer!)

Conclusions

I recommend using the built-in RPG IV conversions when possible. Since I mostly write in C/C++ I've created helper function around the iconv_open and iconv APIs that greatly simplify their use in C, C++ and RPG IV. But again, RPG IV's built-in CCSID conversion is so easy, you'd be hard pressed to justify using something else.

I've included the code for my RPG and C routines below. Feel free to cut/paste into your own IBM i environment and pretend you wrote it. ??Apparently that's a thing nowadays.

Examples

               ctl-opt MAIN(MAIN) ;

      /INCLUDE iOpen/qcpysrc,cozxlate

           dcl-proc main;
                dcl-s  ebcdicData char(255) inz('Hello World');
                dcl-s  asciiData  char(255) CCSID(1208);

                dcl-s  RPGTarget  char(255);
                dcl-s  webTarget char(255);

                dcl-s  dataLen int(10);
                dcl-s  xLen int(10);
                dcl-ds cd likeDS(iconv_t) Inz(*LIKEDS);

                     // Convert from Job CCSID to PC ASCII (UTF-8)
                asciiData = ebcdicData;
                     // Convert PC ASCII (UTF-8) to Job CCSID
                RPGTarget = asciiData;

                xlateOpen(cd : 1208 : 0);  // Xlate from Job CCSID to UTF-8

                dataLen = %len(%trimR(ebcdicData));

                     // Convert from Job CCSID to PC ASCII (UTF-8) using iconv()
                xLen = xlate( cd : webTarget : %size(webTarget) :
                                   ebcdicData: dataLen);

                snd-msg 'Translated ' + %char(xLen) + ' bytes to ASCII.';
                return;

                on-exit;
                  xlateClose(cd);
           end-proc;         

In the above example, I illustrate converting between UTF-8 and the Job CCSID (which is always 0) using iconv and native RPG IV techniques. Below is the code for the C module that I wrote to simplify calling iconv from C and RPG IV. It is compiled into a *MODULE and then I create a service program from it. The RPG IV prototype is listed below the C "H" file.

// COZXLATE.C

#include <stdlib.h>
#include <stdio.h>
#include <except.h>
#include <signal.h>

#include <qp0ztrc.h>   /* Qp0zLprintf() */

#include <iconv.h>
#include <qtqiconv.h>
#include <cpybytes.mih>

#include <errno.h>
#include <unistd.h>

#include <iopen/h/cozxlate>

#pragma datamodel(P128)

   // Helper macro to set struct's to hex zeros
#define clearStruct(s) memset((char*)&s,0x00,sizeof(s))

int coz_xLateOpen( iconv_t* hConv, int toCCSID, int fromCCSID )
{
     QtqCode_T QtoCode;
     QtqCode_T QfromCode;

     iconv_t    cd;

     clearStruct(QtoCode);
     clearStruct(QfromCode);
     clearStruct(cd);

     QtoCode.CCSID   = toCCSID;
     QfromCode.CCSID = fromCCSID;

     cd = QtqIconvOpen( &QtoCode, &QfromCode );

    if (cd.return_value == -1)
    {
       Qp0zLprintf("coz_xlateOpen failed to open %d -> %d translation: %s\n",
                   fromCCSID, toCCSID, strerror(errno));
    }
    else
    {      // copy valid iConv handle to return parameter 1
       _CPYBYTES((char*)hConv, &cd, sizeof(iconv_t));
    }
    return cd.return_value;
}

void coz_xLateClose( iconv_t* hConv)
{
   if (hConv->return_value >= 0)
   {
     iconv_close( *hConv );
   }
   memset(hConv, 0x00, sizeof(iconv_t));
}

int coz_xLate( iconv_t*    hConv,
               char*       pOutBuffer, int oLen,
               const char* pInBuffer,  int iLen)
{
     int            nLen = 0;
     char *__ptr128 inp = NULL;
     char *__ptr128 out = NULL;

     size_t  nRtnBytes, inLen, outLen;
     iconv_t cd;

     _CPYBYTES((char*)&cd, hConv, sizeof(iconv_t));

     if (pInBuffer != NULL)
     {
       inp = (char *__ptr128) pInBuffer;
       out = (char *__ptr128) pOutBuffer;
       inLen  = iLen;
       outLen = oLen;
       nRtnBytes = iconv( cd, &inp, &inLen, &out, &outLen );

       if ((int)nRtnBytes == -1)
       {
            Qp0zLprintf("coz_xlate->iconv returned %d - %s\n",
                         errno, strerror(errno));
            return 0 ;
       }
     }

     return oLen - outLen;
}

#pragma datamodel(pop)
         

Here is its header file:

 // cozxlate.h    
 // (c) 2016 - R. Cozzi, Jr.


#ifndef COZ_ICONV_
#define COZ_ICONV_

#include <iconv.h>
#include <qtqiconv.h>

#pragma datamodel(P128)

#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */


     int  coz_xLateOpen( iconv_t* hConv, int fromCCSID, int toCCSID );

     int  coz_xLate( iconv_t* hConv,          // iConv handle
                     char*       pOutBuffer,  // Output buffer (target)
                     int         oLen,        // Output buffer length
                     const char* pInBuffer,   // Input buffer (original data)
                     int         iLen         // Input buffer length
                   );

     void coz_xLateClose( iconv_t* hConv);


#ifdef __cplusplus
}
#endif

#pragma datamodel(pop)

#endif
         

To provide these functions to RPG IV, you'll need the below /COPY member:

      /IF NOT DEFINED(COZ_XLATE)
      /DEFINE COZ_XLATE
           // TODO: Add the "BNDSRVPGM(COZXLATE)" parameter to CRTPGM
          CTL-OPT  BNDDIR('IOPEN/IOPEN');
      *********************************************************
      **  This control structure is used by cvtCase/cvtCaseEx
      **  to call the QlgConvertCase API.
      *********************************************************
           dcl-ds iconv_t  ALIGN QUALIFIED INZ TEMPLATE;
               return_value  int(10);
               cd            int(10) DIM(12);
           end-ds;

           dcl-PR xLateOpen extproc('coz_xLateOpen');
             hConv      LIKEDS(iconv_t);
             toCCSID    int(10) VALUE;
             fromCCSID  int(10) VALUE;
           end-pr;

           dcl-PR xLate INT(10) extproc('coz_xLate'); // Returns length xlated
             hConv         LIKEDS(iconv_t);
             toBuffer      char(1) OPTIONS(*VARSIZE);
             toBufferLen   int(10) VALUE;
             fromBuffer    CHAR(1)  OPTIONS(*VARSIZE);
             fromBufferLen int(10) VALUE;
           end-pr;

           dcl-PR xLateClose extproc('coz_xLateClose');
             hConv      LIKEDS(iconv_t);
           end-pr;
      /ENDIF