Model Inputs

Every model receives its input data as files, served to the Docker container via environment variables. Each input you define in your .pd4castrrc.json becomes an INPUT_<KEY>_URL environment variable that your model must fetch with a standard HTTP GET request. Inputs can be uploaded manually (static) at publish time or fetched automatically from external systems (fetched). An input with a single file or fetcher has one source. An input can also declare several sources in a sources map, each a file or a fetcher. On an AUTOMATIC model, each source triggers its own runs. See Input sources for how to declare them.

How Inputs Work

When a model run starts, the platform makes each input file available at a unique URL. Your model must read these URLs from environment variables and download the data. The environment variable name is derived from the input’s key field, converted to uppercase with hyphens replaced by underscores. For example, an input with "key": "demand" becomes INPUT_DEMAND_URL, and an input with "key": "spot-price" becomes INPUT_SPOT_PRICE_URL.

Your model code fetches the file from that URL, parses it, and uses it as input data for your forecasting logic.

JSON Input Format

JSON input files must contain an array of objects that share a consistent schema. Each object represents one row of data, matching how CSV and Parquet inputs are structured. For example:

[
  { "timestamp": "2026-01-01T00:00:00Z", "price": 42.5 },
  { "timestamp": "2026-01-01T00:05:00Z", "price": 43.1 }
]

This requirement applies any time the platform processes your file:

• Format conversion, when uploadFileFormat and targetFileFormat differ.
• Sensitivities, which load every JSON input into a SQL database, apply a SQL query, and write the data back out.

Input Configuration Fields

Each entry in the inputs array in .pd4castrrc.json supports the following fields.

A storage location is a storage bucket and folder that the platform watches for input files. Every input uses the platform’s shared storage location unless you declare the ID of your own, which Pipelabs sets up on request.

Static Inputs

Static inputs are files you can upload automatically at publish time. They’re ideal for reference data, lookup tables, or configuration files that don’t change with every model run.

Each static input declares the local file the CLI uploads via the file field, relative to the project root. When you run pd4castr publish, the CLI uploads that file to its storage location.

A static parquet input can be partitioned by mapping a column name to a model axis value prior to being used by a model run. When partitioned this way, your queries only receive the input’s rows specific to that variant’s model run.

A minimal static input configuration looks like this:

{
  "key": "reference_prices",
  "file": "test_input/reference_prices.csv",
  "trigger": "USE_MOST_RECENT_FILE",
  "uploadFileFormat": "csv",
  "targetFileFormat": "csv"
}

Dynamic Inputs (Data Fetchers)

Fetched inputs pull data automatically from external systems on a schedule. This is useful for models that need fresh data without manual intervention.

AEMO MMS is the only data fetcher pd4castr ships. It supplies Australian energy market data. Data from another provider can go in as a static input.

pd4castr supports custom data integrations. We can adapt to your preferred technology or process. To learn more, contact us to connect your data provider.

To configure a dynamic input, add a fetcher block to the input entry:

{
  "key": "predispatch_price",
  "trigger": "WAIT_FOR_LATEST_FILE",
  "uploadFileFormat": "json",
  "targetFileFormat": "json",
  "fetcher": {
    "type": "AEMO_MMS",
    "checkInterval": 300,
    "config": {
      "checkQuery": "queries/data-fetchers/predispatch-price-check.sql",
      "fetchQuery": "queries/data-fetchers/predispatch-price-fetch.sql"
    }
  }
}

The fetcher fields are:

The platform runs the check query at the configured interval. When the results differ from the previous check, it executes the fetch query and writes the output to storage. This new data can then trigger an automatic model run. See SQL queries for the tables these queries read and the limits they run under.

Input Sources

Declare more than one source for an input when more than one publisher produces it and each publication needs its own runs. Two distinct AEMO queries feeding the same demand input are a common case: each query’s data lands on its own schedule, and each landing should queue its own run rather than wait for the other.

To give an input several sources, declare a sources map keyed by source key in place of a single file or fetcher, and make the input participate in a source-binding axis with "axes": { "<axisKey>": { "kind": "source-binding" } }. The input declares no file, fetcher, or storageLocation of its own, because each entry of sources declares its own:

{
  "axes": [
    {
      "key": "demand_source",
      "label": "Demand source",
      "type": "string",
      "valuesFrom": { "kind": "source-binding" },
      "default": "aemo"
    }
  ],
  "inputs": [
    {
      "key": "demand_forecast",
      "trigger": "WAIT_FOR_LATEST_FILE",
      "uploadFileFormat": "json",
      "targetFileFormat": "json",
      "axes": { "demand_source": { "kind": "source-binding" } },
      "sources": {
        "aemo": {
          "fetcher": {
            "type": "AEMO_MMS",
            "checkInterval": 300,
            "config": {
              "checkQuery": "queries/aemo-check.sql",
              "fetchQuery": "queries/aemo-fetch.sql"
            }
          }
        },
        "nz": { "file": "inputs/nz.json" }
      }
    }
  ]
}

Each entry of sources supports the following fields.

A source key follows the same grammar as an input key (^[a-z0-9][a-z0-9_-]*$). A source-binding axis takes its values from the source keys of the inputs that participate in it, in the order sources declares them. Source keys that are whole numbers without a leading zero, such as 2024, come first in ascending numeric order. See Source-binding axes.

A file source uploads to its own path inside its storage location, so two sources of one input never share an arrival. pd4castr publish uploads every file entry of an input’s sources through its own signed upload URL, and fails before uploading anything if one of the files is missing.

Make every source of an input produce files with the same columns. The container reads each source’s file from the same INPUT_<KEY>_URL, and no environment variable tells it which source the file came from.

The axis’s default selects the input’s default source, aemo in the example above. pd4castr test and pd4castr fetch both run against the default source of an input with sources. See Fetching test data for what each command does with it.

Updating a model’s current revision in place can add or remove a non-default source. Changing the default source, or moving the axis between static and source-binding, needs a new revision instead; see Changing axes on a published model. When a republish changes a source’s check or fetch query, that source’s fetch state resets, and pd4castr publish warns, naming the source, so you can time the publish into a quiet part of its arrival cycle.

Declaring sources needs @pd4castr/cli 5.1.0 or later. An earlier CLI either rejects the sources key as unrecognised or fails the publish.

If you want to upload data outside the publish flow to trigger a model run, contact us about having access to your own storage location. On an input with sources, the source’s key selects which source such an upload lands in.

Trigger Types and Run Modes

Each input has a trigger that controls how it interacts with automatic model runs:

• WAIT_FOR_LATEST_FILE — The platform waits until this input has received new data before triggering a run. All inputs with this trigger type must have fresh data before the run starts.
• USE_MOST_RECENT_FILE — The platform uses whatever data was last available for this input. It doesn’t block a run from starting.

For models with runMode set to AUTOMATIC, the platform monitors all inputs. A run is triggered only when every WAIT_FOR_LATEST_FILE input has received new data. USE_MOST_RECENT_FILE inputs are included in the run using their most recently available file.

On a model with a source-binding axis, each source queues its own runs on its own cadence. The file of an input with a single file or fetcher reaches every source. A source still waits on every input’s trigger before it queues a run, as this table shows for a model with one input that declares sources and one other input with a single file or fetcher:

An input with a single file or fetcher on WAIT_FOR_LATEST_FILE holds every source to its own cadence. If that input refreshes once a day, an hourly source runs once a day too, waiting on that input’s next file.

When every input of a model declares sources, a source can queue a run only after every participating input has at least one file from that source. Each participating input on WAIT_FOR_LATEST_FILE must then supply a new file since the source’s last run. If no participating input uses WAIT_FOR_LATEST_FILE, any new file from that source queues a run. If only one input participates and uses WAIT_FOR_LATEST_FILE, its source file queues the run when it arrives. Confirm every input’s trigger before your first publish.

Example Configuration

Here’s a complete inputs array with both a static and a fetched input:

{
  "inputs": [
    {
      "key": "dispatch_price",
      "trigger": "WAIT_FOR_LATEST_FILE",
      "uploadFileFormat": "json",
      "targetFileFormat": "json",
      "fetcher": {
        "type": "AEMO_MMS",
        "checkInterval": 300,
        "config": {
          "checkQuery": "queries/data-fetchers/dispatch-price-check.sql",
          "fetchQuery": "queries/data-fetchers/dispatch-price-fetch.sql"
        }
      }
    },
    {
      "key": "regional_historical_prices",
      "file": "test_input/regional_historical_prices.csv",
      "trigger": "USE_MOST_RECENT_FILE",
      "uploadFileFormat": "csv",
      "targetFileFormat": "csv"
    }
  ]
}

In this example, dispatch_price is fetched automatically from AEMO every 5 minutes. regional_historical_prices is a static CSV file uploaded during publish. The model won’t run automatically until fresh dispatch price data arrives, but it uses the latest available regional historical prices data.

Next Steps

• Use pd4castr fetch to pull live data from your configured fetchers into your local test_input/ directory.
• See Source-binding axes for how an input’s sources multiply triggers instead of runs.
• See the Configuration File reference for the full schema.