Skip to content

Filter RLI Lines in Working Units

Overview

Many classic working units process the lines of an RLI one by one. With an RLI filter, a working unit processes only the lines that match defined criteria. For example, Rli.copyFiles copies only the files whose RLI column IS_DIST_FILE contains Y.

You define an RLI filter with working unit parameters. Each filter compares the value of one RLI column with one or more values. Working units that support RLI filters list the parameters RlicFilter, FilterOperator, and FilterValue in their working unit definition, for example Rli.copyFiles and Rli.fillRliColumn.

Hint - operators of the if-then-else module

The operators of RLI filters differ from the operators of the if-then-else process module. For the if-then-else operators, refer to Operators.

Prerequisites

  • A workflow with a working unit that supports RLI filters, for example dpf4distribute.DistributeFileSystem and the working unit Rli.copyFiles.
  • An RLI whose lines contain the column you want to filter on.

Define a Filter

  1. Set the following parameters for the working unit, for example in the MAPPING element of its workflow position:

    • RlicFilter: name of the RLI column to check. Without RlicFilter, the working unit processes all lines.
    • FilterOperator: comparison operator, refer to Operators. Optional. Default: eq.
    • FilterValue: one or more values to compare the column value with, refer to Compare with Several Values. Optional. Default: empty. File test operators do not evaluate FilterValue.

    Example - process only lines marked for distribution

    <MAPPING RlicFilter="IS_DIST_FILE" FilterOperator="eq" FilterValue="Y"/>

    Add the filter parameters to the existing MAPPING element of the workflow position. A workflow position has only one MAPPING element.

    You can also pass the parameters as job parameters, for example with the -s option of DPF Batch Client.

    Example - passing filter parameters with DPF Batch Client

    dpfclient -wf test.copyDistFiles -protocol http -host localhost:4303 -s RlicFilter=IS_DIST_FILE -s FilterOperator=eq -s FilterValue=Y -f "test.rli"

  2. If you changed the workflow file, restart the seal-dpf-process-manager service. It reads workflow files at startup only, refer to Administrate the DPF Services.

    dpf service stop seal-dpf-process-manager

    dpf service start seal-dpf-process-manager

  3. Run a test job and check the result, refer to Validate the Filter.

Operators

DPF inserts FilterOperator into a Perl expression of the form <column value> <operator> <filter value>. Therefore, Perl comparison operators work. Text comparisons are case-sensitive.

Operator Type Matches if the column value …
eq text equals the filter value.
ne text differs from the filter value.
lt, gt, le, ge text is alphabetically less than, greater than, less than or equal to, or greater than or equal to the filter value.
==, != number equals or differs from the filter value. For example, 01 == 1 matches.
<, >, <=, >= number is less than, greater than, less than or equal to, or greater than or equal to the filter value.
=~ regular expression contains a match of the expression.
!~ regular expression contains no match of the expression.
-e file test is the path of an existing file or directory.
-f file test is the path of a file.
-d file test is the path of a directory.
-s file test is the path of a file that exists and is not empty.
-z file test is the path of a file that exists and is empty.

Use numeric operators only for columns that contain numbers. File tests do not evaluate FilterValue. Use them on columns with a file path, for example CurrFileName. To negate a file test, prepend !, for example !-f.

Regular Expressions

The following rules apply to =~ and !~:

  • The expression matches anywhere in the column value. For example, NOMEN matches NOMENCLATURE. To match the whole value, use ^ and $, for example ^NOMEN$.
  • You can enclose the expression in slashes. To ignore case, append i, for example /\.pdf$/i. Other modifiers do not work.
  • Escape a slash inside the expression as \/.
  • A comma separates several expressions, refer to Compare with Several Values. Expressions containing a comma, for example A{1,3}, therefore do not work as expected.

Examples

Operator Column value Filter value Result Reason
eq PDF PDF matches The values are equal.
eq pdf PDF no match Text comparisons are case-sensitive.
ne TXT PDF matches The values differ.
lt A100 B200 matches A comes before B.
lt 10 9 matches Text comparison goes character by character. 1 comes before 9.
gt a B matches Lowercase letters come after uppercase letters.
le ABC ABC matches The values are equal.
== 01 1 matches Both values are the number 1.
< 10 9 no match 10 is greater than 9 as a number.
>= 10 10 matches The numbers are equal.
=~ NOMENCLATURE NOMEN matches The expression matches a part of the value.
=~ NOMENCLATURE ^NOMEN$ no match ^ and $ require the whole value to match.
=~ invoice.PDF /\.pdf$/i matches i ignores case.
!~ notes.txt \.pdf$ matches The value does not end with .pdf.
-f D:\output\invoice.pdf matches The file exists.
-d D:\output matches The path is a directory.
-z D:\output\empty.txt matches The file exists and has a size of 0 bytes.
!-f D:\output\missing.pdf matches The file does not exist.

Invalid Operators

With an invalid operator, no line matches. For the operator ~, the job log additionally shows the error '~' is not a valid operator. DPF then sets DpfJobIcon to warn and writes the error to DpfJobMessage.

Compare with Several Values

Separate several values in FilterValue with commas. Spaces after a comma are ignored, for example PDF, TXT.

A filter matches if the comparison is true for at least one of the values:

  • eq with PDF, TXT matches lines whose column value is PDF or TXT.
  • =~ with ^A, ^B matches lines whose column value starts with A or B.

Caution - ne and !~ with several values

With several values, ne and !~ match a line as soon as one comparison is true. For example, ne with PDF, TXT matches every line, because every column value differs from at least one of the two values. To exclude several values, use !~ with a single expression, for example ^(PDF|TXT)$.

To compare with an empty value, leave FilterValue empty, or set it to "" or ''. For example, eq with an empty FilterValue matches lines whose column is empty.

Combine Several Filters

Add further filters with numbered parameters: RlicFilter2, FilterOperator2, FilterValue2, then RlicFilter3 and so on. The first filter has no number. The numbering must be continuous. The working unit stops evaluating at the first missing RlicFilter<n>.

By default, a line must match all filters.

  • AtLeastOneFilterMustMatch: specifies whether a line must match only one of the filters. Optional.

    Available values:

    • Y, J, X, or 1

      A line must match at least one filter.

    • N

      A line must match all filters.

    Default: N

Example - PDF files marked for distribution

<MAPPING RlicFilter="IS_DIST_FILE" FilterValue="Y" RlicFilter2="CurrFileName" FilterOperator2="=~" FilterValue2="/\.pdf$/i"/>

Set Filters per Workflow Position

With a prefix, several working units in the same job can use different filters, even though they all read the same job parameters.

  • FilterPrefix: prefix for the names of all filter parameters of this working unit. Optional.

With FilterPrefix="Copy", the working unit reads CopyRlicFilter, CopyFilterOperator, CopyFilterValue, CopyRlicFilter2, and so on. It ignores filter parameters without the prefix. AtLeastOneFilterMustMatch takes no prefix.

A common setting is FilterPrefix="%WFPOSITION%". The prefix is then the name of the workflow position, refer to WfPosition. For a position named DistributeToOutputDirectory, the working unit reads DistributeToOutputDirectoryRlicFilter, and so on.

Example - passing prefixed filter parameters with DPF Batch Client

The working unit in the workflow test.copyFilesFilter has FilterPrefix="Copy":

dpfclient -wf test.copyFilesFilter -protocol http -host localhost:4303 -s CopyRlicFilter=IS_DIST_FILE -s CopyFilterValue=Y -f "test.rli"

Validate the Filter

  1. Run a test job.

  2. In DPF Tracker, open the job details and select the Log File tab, refer to Administrate Jobs in DPF Tracker.

  3. Check that the job log contains one line per filter, for example:

    Using RLI column IS_DIST_FILE, equation operator 'eq' and filter 'Y'

    If the line is missing, the working unit did not receive the filter parameters. Check the parameter names and FilterPrefix.

  4. If the RLI does not contain the filter column, the job log shows a warning, and the working unit processes no line:

    RLI column IS_DIST_FILE does not exist. NO RLI lines will be processed!

  5. To check the result per line, set LogLevel to LOG_DEBUG. The job log then shows the evaluated expression for each line, followed by Filter expression matches the filter or Filter expression does not match the filter.

Back to top