Output Files to an S3 Bucket¶

Overview¶

Digital Process Factory can output files to an S3 bucket, similar to the S3 output already implemented for PLOSSYS Output Engine. This is primarily relevant for SAAS installations, which have no direct filesystem access for output, but the working unit itself is not restricted to SAAS and works in any installation.

The s3.copyFromRli node working unit copies the files listed in an RLI to a configured S3 bucket.

Install the Working Unit¶

The s3.copyFromRli node working unit ships as part of seal-dpf-wustandard. For how to make it available in DPF Director, refer to Add a Node Working Unit to DPF Director.

Configure the S3 Connection¶

Configure the S3 connection either with environment variables or with input parameters of the working unit. An input parameter overwrites the corresponding environment variable.

Environment variables, set for the seal-dpf-wustandard service:
- S3_BUCKET: Mandatory.
- S3_ACCESS_KEY_ID: Mandatory.
- S3_SECRET_ACCESS_KEY: Mandatory.
- S3_REGION: Optional. Default: eu-central-1.
- S3_ENDPOINT: Optional. Only needed for an S3-compatible service that is not accessible via AWS's default per-region endpoints, for example MinIO.
- S3_FILE_NAME_TEMPLATE: Optional. Default: original file names.
- S3_BASE_DIR_TEMPLATE: Optional. Default: original job directory.
- S3_METADATA_KEYS: Optional.
Input parameters:
- CurrRliFileName: name of the RLI file, with or without CurrFileName column. Mandatory
- IGNORE_MISSING_FILES: ignore missing files. Default: No
- RlicSourceFile: parameter name in the RLI item containing the source file name. Default: CurrFileName
- RlicTargetFile: parameter name in the RLI item containing the target file name. Default: TargetFileName
- region, bucket, accessKeyId, secretAccessKey, endpoint, fileNameTemplate, baseDirTemplate, metadataKeys: overwrite the corresponding environment variable listed above.

Hint - subdirectories

The job directory name (S3_BASE_DIR_TEMPLATE or baseDirTemplate) and the target file names (S3_FILE_NAME_TEMPLATE or the TargetFileName RLI item parameter) may contain /, which is interpreted as subdirectory on S3.

Output Parameters¶

CurrRliFileName: new RLI file.
IGNORED_ERRORS: number of ignored errors if IGNORE_MISSING_FILES is set to Y.

Use the Working Unit in a Workflow¶

The s3.copyFromRli working unit does nothing on its own. Add it as a step to a process, like any other working unit. Without a process referencing it, the working unit is loaded but never called.

Hint - SAAS installation

In a SAAS installation, workflow and working unit configuration is administrated via DPF CLI's dpf saas command.