Establish connection with FTP server and read files from Server using WSO2 EI File connector

In this article I’m going to show you how we can establish connection with FTP server and read files from source directory using WSO2 EI File connector.

Scenario:

• Check specific type of file exist or not in source directory.
• If exist, list all the files from directory with specific extension.
• Read all the files one by one from source directory.
• Move the file to the Destination directory.

We are going to use below file connector operations to achieve our scenario:

1. isFileExist:

Using the isFileExist operation, we can check whether the source directory has any files with any extension or a specific extension.

<fileconnector.isFileExist> <source>ftp://username:password@hostname:port/source_directory_path</source>
<filePattern>.*\.pdf</filePattern>
</fileconnector.isFileExist>        

Here I have put the file pattern to only check .pdf files in the directory. We can put * If we do not need any specific file pattern.

Following property will give the result:

<property name="isFileExist" expression="json-eval($.fileExist)" scope="default" type="STRING"/>        

Note: If password contain @ like p@ssword then use password as p%40ssword.

2. Search:

Search operation returns the absolute file path of the list of files in the mentioned source directory.

<fileconnector.search>   <source>ftp://username:password@hostname:port/source_directory_path</source>
<filePattern>.*\.pdf</filePattern>
<recursiveSearch>false</recursiveSearch>
</fileconnector.search>        

Search operations give the absolute file path of each file but we need to be careful here if there is a single file this response the absolute file path and there are multiple files it response array of file paths.

<property name="file-search-response" expression="json-eval($.result.file)" scope="default" type="STRING"/>
<property name="file-search-response-first-element" expression="json-eval($.result.file[0])" scope="default" type="STRING"/>        

Use a iterate mediator to get multiple files as one by one:

<iterate xmlns:ns="https://org.wso2.esbconnectors.FileConnector" xmlns:soapenv="https://schemas.xmlsoap.org/soap/envelope/" id="filesIterate" expression="https://ns:file" sequential="true">
  <target>
   <sequence>
     <property name="file-search-response-current-element" expression="https://ns:file" scope="default" type="STRING" />
     <property name="source" expression="fn:concat('ftp://username:password@hostname:port',$ctx:file-search-response-current-element)" scope="default" type="STRING" />
   </sequence>
  </target>
</iterate>        

Now we have absolute file paths we can use Read operation to read files.

Note:

We need to do a configuration change before using Read operation as below.

<EI_HOME>/conf/axis2/axis2.xml file:

Add <messageFormatter contentType=”application/file” class=”org.wso2.carbon.relay.ExpandingMessageFormatter”/>

Add <messageBuilder contentType=”application/file” class=”org.apache.axis2.format.BinaryBuilder”/>

3. Read:

We can use Read operation with the source as absolute file path.

<fileconnector.read>
  <source>{$ctx:source}</source>
  <contentType>application/file</contentType>
  <filePattern>.*\.pdf</filePattern>
  <streaming>true</streaming>
</fileconnector.read>        

4. Move:

We can use the Move operation to either move each file from source to Destination directory one by one or as all.

<fileconnector.move>
<source>{$ctx:source}</source> <destination>ftp://username:password@hostname:port/target_directory_path</destination>
<includeParentDirectory>false</includeParentDirectory>
<filePattern>.*\.pdf</filePattern>
</fileconnector.move>						          

You can find the below API for whole flow code.

<?xml version="1.0" encoding="UTF-8"?>
<api xmlns="https://ws.apache.org/ns/synapse" name="FileProcessing" context="/fileprocessing">
   <resource methods="POST" uri-template="/fetch">
      <inSequence>
         <fileconnector.isFileExist>
            <source>ftp://username:password@hostname:port/source_directory_path</source>
            <filePattern>.*\.pdf</filePattern>
         </fileconnector.isFileExist>
         <property name="isFileExist" expression="json-eval($.fileExist)" scope="default" type="STRING" />
         <log level="custom">
            <property name="responselog" expression="get-property('isFileExist')" />
         </log>
         <filter source="get-property('isFileExist')" regex="true" description="Check files exists in the directory">
            <then>
               <fileconnector.search>
                  <source>ftp://username:password@hostname:port/source_directory_path</source>
                  <filePattern>.*\.pdf</filePattern>
                  <recursiveSearch>false</recursiveSearch>
               </fileconnector.search>
               <property name="file-search-response" expression="json-eval($.result.file)" scope="default" type="STRING" />
               <property name="file-search-response-first-element" expression="json-eval($.result.file[0])" scope="default" type="STRING" />
               <log>
                  <property name="file-search-result" expression="json-eval($.result)" />
                  <property name="file-search-response" expression="json-eval($.result.file)" />
                  <property name="file-search-response-first-element" expression="json-eval($.result.file[0])" />
               </log>
               <filter source="get-property('file-search-response-first-element')" regex="[^\n]+" description="Check file search multiple or single">
                  <then>
                     <iterate xmlns:ns="https://org.wso2.esbconnectors.FileConnector" xmlns:soapenv="https://schemas.xmlsoap.org/soap/envelope/" id="filesIterate" expression="https://ns:file" sequential="true">
                        <target>
                           <sequence>
                              <property name="file-search-response-current-element" expression="https://ns:file" scope="default" type="STRING" />
                              <log>
                                 <property name="file-search-response-current-element-response" expression="$ctx:file-search-response-current-element" />
                              </log>
                              <property name="source" expression="fn:concat('ftp://username:password@hostname:port',$ctx:file-search-response-current-element)" scope="default" type="STRING" />
                              <log>
                                 <property name="ReadRequest" expression="$ctx:source" />
                              </log>
                              <fileconnector.read>
                                 <source>{$ctx:source}</source>
                                 <contentType>application/file</contentType>
                                 <filePattern>.*\.pdf</filePattern>
                                 <streaming>true</streaming>
                              </fileconnector.read>
                              <property name="API-respons" expression="json-eval($.binary)" scope="default" type="STRING" />
                              <log>
                                 <property name="ReadResponse" expression="$ctx:API-respons" />
                              </log>
                              <log>
                                 <property name="MoveRequest" expression="$ctx:source" />
                              </log>
                              <fileconnector.move>
                                 <source>{$ctx:source}</source>
                                 <destination>ftp://username:password@hostname:port/target_directory_path</destination>
                                 <includeParentDirectory>false</includeParentDirectory>
                                 <filePattern>.*\.pdf</filePattern>
                              </fileconnector.move>
                              <log>
                                 <property name="MoveResponse" value="File successfully moved to Out Folder" />
                              </log>
                           </sequence>
                        </target>
                     </iterate>
                  </then>
                  <else>
                     <property name="source" expression="fn:concat('ftp://username:password@hostname:port',$ctx:file-search-response)" scope="default" type="STRING" />
                     <log>
                        <property name="ReadRequest" expression="$ctx:source" />
                     </log>
                     <fileconnector.read>
                        <source>{$ctx:source}</source>
                        <contentType>application/file</contentType>
                        <filePattern>.*\.pdf</filePattern>
                        <streaming>true</streaming>
                     </fileconnector.read>
                     <property name="API-respons" expression="json-eval($.binary)" scope="default" type="STRING" />
                     <log>
                        <property name="ReadResponse" expression="$ctx:API-respons" />
                     </log>
                     <log>
                        <property name="MoveRequest" expression="$ctx:source" />
                     </log>
                     <fileconnector.move>
                        <source>{$ctx:source}</source>
                        <destination>ftp://username:password@hostname:port/target_directory_path</destination>
                        <includeParentDirectory>false</includeParentDirectory>
                        <filePattern>.*\.pdf</filePattern>
                     </fileconnector.move>
                     <log>
                        <property name="MoveResponse" value="File successfully moved to Out Folder" />
                     </log>
                  </else>
               </filter>
            </then>
            <else />
         </filter>
      </inSequence>
      <outSequence />
      <faultSequence />
   </resource>
</api>