Skip to content

Output Files to an S3 Bucket

Overview

Digital Process Factory can output files to an S3 bucket, similar to the S3 output for PLOSSYS Output Engine. This is mainly relevant for SaaS installations, but not limited to them.

The s3.copyFromRli node working unit copies the files listed in an RLI to a configured S3 bucket.

Prerequisites

  • An existing S3 bucket that matches your infrastructure. Digital Process Factory does not create it.
  • Access credentials for the bucket: an access key ID and a secret access key.
  • The s3.copyFromRli node working unit. It ships as part of seal-dpf-wustandard and is included by default. No separate installation needed.
  • A workflow that implements the s3.copyFromRli node working unit.

Configure the S3 Connection

  1. Specify the S3 connect with the following environment variables.

    • S3_BUCKET: S3 bucket for uploading files. Mandatory.
    • S3_ACCESS_KEY_ID: Access key ID for uploading files to S3. Mandatory.
    • S3_SECRET_ACCESS_KEY: Secret access key for uploading files to S3. Mandatory.
    • S3_REGION: S3 region for uploading files. Optional. Default: eu-central-1.
    • S3_ENDPOINT: URL of the S3 endpoint. Optional. Only needed for an S3-compatible service that is not accessible via AWS's default per-region endpoints, for example MinIO.
    • S3_FILE_NAME_TEMPLATE: Template for target file names on S3, with placeholders for RLI item parameters. Optional. Default: original file names.
    • S3_BASE_DIR_TEMPLATE: Template for the job directory name on S3, with placeholders for RLI parameters. Optional. Default: original job directory.
    • S3_METADATA_KEYS: RLI item metadata keys, separated with commas, for filtering S3 object metadata when it exceeds S3's 2 KB metadata size limit. Optional.

    Hint - subdirectories

    The job directory name (S3_BASE_DIR_TEMPLATE or baseDirTemplate) and the target file names (S3_FILE_NAME_TEMPLATE or the TargetFileName RLI item parameter) may contain /, which is interpreted as subdirectory on S3.

    Hint - corresponding WU parameters

    The input parameters region, bucket, accessKeyId, secretAccessKey, endpoint, fileNameTemplate, baseDirTemplate, metadataKeys overwrite the corresponding environment variable listed above.

    1. If you don't already have a workflow that calls s3.copyFromRli, create one for testing, for example test.s3CopyFromRli:

      <?xml version="1.0" encoding="ISO-8859-1"?>

<DPF>
   <config>
      <workflows>
         <test.s3CopyFromRli COMMENT="" NAME="Test S3 CopyFromRli" DPFINT-Interactive="0" PROJECT="test" VERSION="$Id$" INTERNALNAME="test.s3CopyFromRli" DPFINT-Customer="0" DPFINT-Type="WorkflowDescription">
            <MAIN BEGIN="CopyToS3" NAME=""/>
            <CopyToS3 ERROR="END" NEXT="END" NAME="s3.copyFromRli" TYPE="WU" POSY="400" POSX="300" COMMENT="Upload the files listed in the RLI to S3.">
               <DEFAULTS/>
               <MAPPING/>
               <CONNECTIONS ERROR="DIRECT" NEXT="DIRECT"/>
            </CopyToS3>
            <END POSY="100" POSX="200" COMMENT="" NAME="END" TYPE="STOP"/>
         </test.s3CopyFromRli>
      </workflows>
   </config>
</DPF>

      Hint - workflow deployment

      Where you save this file depends on your installation type. On an on-premise installation, save customer-specific workflows below %SEAL_CUSTOMDIR%, for example in a subdirectory named after the project prefix of the workflow's internal name, test. Refer to Customer-Specific Files on Linux or Windows. Then restart the seal-dpf-process-manager service. It reads workflow files at startup only, refer to Administrate the DPF Services. In a SaaS installation, workflows are deployed to MongoDB with the DPF CLI instead, refer to Administrate DPF in SaaS. Confirm the correct method for your environment.

    2. Call the workflow with DPF Batch Client to validate your configuration.

      Example - calling a workflow using DPF Batch Client in a PowerShell

      dpfclient -wf test.s3CopyFromRli -protocol http -host localhost:4303 -f "test.rli"

      To overwrite set environment variables for the job, use the -s option:

      dpfclient -wf test.s3CopyFromRli -protocol http -host localhost:4303 -s bucket="blubb" -f "test.rli"

Output Parameters

  • CurrRliFileName: new RLI file.
  • IGNORED_ERRORS: number of ignored errors if IGNORE_MISSING_FILES is set to Y.
Back to top