kedro.config.TemplatedConfigLoader¶

class kedro.config.TemplatedConfigLoader(conf_source, env=None, runtime_params=None, *, base_env='base', default_run_env='local', globals_pattern=None, globals_dict=None)[source]¶

Extension of the ConfigLoader class that allows for template values, wrapped in brackets like: ${…}, to be automatically formatted based on the configs.

The easiest way to use this class is by setting the CONFIG_LOADER_CLASS constant in settings.py.

Example:

# in settings.py
from kedro.config import TemplatedConfigLoader

CONFIG_LOADER_CLASS = TemplatedConfigLoader
CONFIG_LOADER_ARGS = {
    "globals_pattern": "*globals.yml",
}

The contents of the dictionary resulting from the globals_pattern get merged with the globals_dict. In case of conflicts, the keys in globals_dict take precedence. If the formatting key is missing from the dictionary, the default template value is used (the format is “${key|default value}”). If no default is set, a ValueError will be raised.

Global parameters can be namespaced as well. An example could work as follows:

globals.yml

bucket: "my_s3_bucket"

environment: "dev"

datasets:
    csv: "pandas.CSVDataSet"
    spark: "spark.SparkDataSet"

folders:
    raw: "01_raw"
    int: "02_intermediate"
    pri: "03_primary"
    fea: "04_feature"

catalog.yml

raw_boat_data:
    type: "${datasets.spark}"
    filepath: "s3a://${bucket}/${environment}/${folders.raw}/boats.csv"
    file_format: parquet

raw_car_data:
    type: "${datasets.csv}"
    filepath: "s3://${bucket}/data/${environment}/${folders.raw}/cars.csv"

This uses jmespath in the background. For more information see: https://github.com/jmespath/jmespath.py and https://jmespath.org/.

Attributes

conf_paths

Property method to return deduplicated configuration paths.

Methods

get(*patterns)

Tries to resolve the template variables in the config dictionary provided by the ConfigLoader (super class) get method using the dictionary of replacement values obtained in the __init__ method.

__init__(conf_source, env=None, runtime_params=None, *, base_env='base', default_run_env='local', globals_pattern=None, globals_dict=None)[source]¶

Instantiates a TemplatedConfigLoader.

Parameters

conf_source (str) – Path to use as root directory for loading configuration.
env (Optional[str]) – Environment that will take precedence over base.
runtime_params (Optional[Dict[str, Any]]) – Extra parameters passed to a Kedro run.
base_env (str) –
default_run_env (str) –
globals_pattern (Optional[str]) – Optional keyword-only argument specifying a glob pattern. Files that match the pattern will be loaded as a formatting dictionary.
globals_dict (Optional[Dict[str, Any]]) – Optional keyword-only argument specifying a formatting dictionary. This dictionary will get merged with the globals dictionary obtained from the globals_pattern. In case of duplicate keys, the globals_dict keys take precedence.

property conf_paths¶: Property method to return deduplicated configuration paths.

get(*patterns)[source]¶

Tries to resolve the template variables in the config dictionary provided by the ConfigLoader (super class) get method using the dictionary of replacement values obtained in the __init__ method.

Parameters: *patterns – Glob patterns to match. Files, which names match any of the specified patterns, will be processed.
Return type: Dict[str, Any]
Returns: A Python dictionary with the combined configuration from all configuration files. Note: any keys that start with _ will be ignored. String values wrapped in ${…} will be replaced with the result of the corresponding JMESpath expression evaluated against globals (see __init for more configuration files. Note: any keys that start with _ details).
Raises: ValueError – malformed config found.