Skip to main content

LQL tutorial

A link connects workflow nodes: it reads inputs and outputs of some steps and writes to others. The Link Query Language (LQL) is how a link names those inputs and outputs.

This page covers the queries most links need. Fan-out over repeated steps, reference selectors, tags and the advanced examples are in LQL advanced. Every selector, flag and rule is listed in the LQL reference.

Two concepts: ids and positions​

Workflows range from a fixed list of steps to dynamic ones where users add, remove and reorder steps. LQL therefore combines two concepts:

  • Ids are the names of steps and nested workflows. When every step has a unique id, a path of ids is all a link needs.
  • Position selectors relate a step to the steps before or after it. In a dynamic workflow several steps may share an id, and a selector picks the first, the last, all of them, or the one right after a given step.

A path of bare ids is the common case. Selectors appear only where the workflow shape makes an id ambiguous.

Query structure​

Every query has the form:

NAME(FLAGS)?:PATH

NAME is the alias the handler uses to read or write the matched value. FLAGS is an optional comma-separated list inside parentheses; the flags are optional, call and template. PATH is a /-separated list of segments resolved relative to the workflow that declares the link. A leading / is not allowed.

A query that targets a script input:

value:myworkflow/subworkflow/script/input1

myworkflow, subworkflow and script are ids. A bare id means "the first node with this id", so the query above is the same as:

value:first(myworkflow)/first(subworkflow)/first(script)/input1

A query with no path targets the workflow that declares the link. The forms NAME, NAME:, NAME:. and NAME:./ are equivalent, and NAME:./path is accepted. A . in the middle of a path is not.

Simple cases​

Every case below lives in this one workflow. load produces a table, clean filters it and reports statistics, fit fits a model, and summary reports. Between them, reviews is a nested dynamic workflow with a single step type: users add as many review steps as they want, or remove them all.

{
  id: 'demo',
  type: 'static',
  steps: [
    {id: 'load', nqName: 'Pkg:LoadTable'},     // out: table
    {id: 'clean', nqName: 'Pkg:CleanTable'},   // in: table, threshold; out: table, stats
    {id: 'fit', nqName: 'Pkg:FitModel'},       // in: table, stats, iterations; out: model
    {id: 'reviews', type: 'dynamic',
      stepTypes: [{id: 'review', nqName: 'Pkg:Review'}],  // in: model, notes; out: score
      initialSteps: ['review']},
    {id: 'summary', nqName: 'Pkg:Summary'},    // in: model, scores, notes
  ],
  links: [/* the cases below */],
}

Connect two steps​

{id: 'toClean', from: 'table:load/table', to: 'input:clean/table'}

The alias before the colon (table, input) is what the handler works with. The path after it names the step and its io. In a static workflow every id is unique, so a path of ids is all the link needs. Without a handler the default handler copies each from value to the to entry at the same position.

Reach into a nested workflow​

{id: 'toReview', from: 'model:fit/model', to: 'target:reviews/review/model'}

Paths are relative to the workflow that declares the link, and cross into nested workflows segment by segment. Inside reviews the id review is not unique, and a bare id means the first match, so this link feeds the first review only. The same link declared inside reviews would write target:review/model.

Read several values in one handler​

{
  id: 'iterations',
  type: 'validator',
  from: ['threshold:clean/threshold', 'n:fit/iterations'],
  to: 'target:fit/iterations',
  handler: ({controller}) => {
    if (controller.getFirst('threshold') > 0.5 && controller.getFirst('n') < 100)
      controller.setValidation('target', {warnings: ['Few iterations for a strict threshold']});
    else
      controller.setValidation('target', undefined);
  },
}

Aliases are distinct within a link, across from and to alike. A validator reads its inputs and writes a result to the io named in to.

Several inputs and outputs of one step​

When a link moves several ios between the same two steps, the template flag expands a |-list in the last segment into one query per io. The alias becomes the prefix plus the io name.

{
  id: 'toFit',
  from: 'in_(template):clean/table|stats',
  to: 'out_(template):fit/table|stats',
}

This is the same as writing in_table, in_stats, out_table and out_stats by hand. Use _ as the prefix to get aliases equal to the io names. The list may only appear in the last segment, never in the path to the step.

The same paths in checks and rules​

Checks and rules use the same paths. A check names one io and has no alias:

{id: 'iterationsRange', type: 'check', io: 'fit/iterations', check: {min: 1, max: 10000}}

A rule reads and writes through aliases like any link. Rule queries take no flags but optional and template:

{
  id: 'strict',
  type: 'rule',
  from: 'threshold:clean/threshold',
  to: 'n:fit/iterations',
  when: 'gt(threshold, 0.5)',
  effects: ['warning(n, "Consider more iterations for a strict threshold")'],
}

A step that may be absent​

A link whose query matches nothing is not created. The optional flag creates it anyway. Reading an alias that matched nothing throws, so the handler checks getMatchedInputs first:

{
  id: 'carryNotes',
  from: 'notes(optional):reviews/review/notes',
  to: 'target:summary/notes',
  handler: ({controller}) => controller.setAll('target',
    controller.getMatchedInputs().has('notes') ? controller.getFirst('notes') : ''),
}

Here review means the first review. When the user removes every review the link still runs and writes an empty string.

Every step with one id​

all matches every direct child with the id, so one query yields many values. The handler reads them with getAll:

{
  id: 'collectScores',
  from: 'scores:reviews/all(review)/score',
  to: 'target:summary/scores',
  handler: ({controller}) => controller.setAll('target', controller.getAll('scores')),
}

This is one link that sees all reviews at once. When each review should get a link of its own, the advanced page's expand is the tool.

The host workflow​

A pipeline validator targets a workflow node. A query with no path is the workflow that declares the link:

{
  id: 'needsReviews',
  type: 'pipelineValidator',
  from: 'scores:reviews/all(review)/score',
  to: 'self',
  handler: ({controller}) => {
    const scores = controller.getAll('scores');
    controller.setValidation('self', scores.length ? undefined : {errors: ['Add at least one review']});
  },
}

What the advanced page adds​

LQL advanced covers a base query with expand, which turns one link definition into one link per matching step, and the reference selectors same, before and after that anchor from and to to each of those steps. It also covers tags, which match across nesting levels, and the wildcard io selectors, with one example per topic on a single workflow. Form logic built from rules, such as a computed default and a lookup table, is in Rules and checks.