kedro.config.TemplatedConfigLoader¶

class kedro.config.TemplatedConfigLoader(conf_paths, *, globals_pattern=None, globals_dict=None)[source]¶

Extension of the ConfigLoader class that allows for template values, wrapped in brackets like: ${…}, to be automatically formatted based on the configs.

The easiest way to use this class is by registering it into the KedroContext using hooks. This can be done by updating the hook implementation register_config_loader in hooks.py, making it return a TemplatedConfigLoader object instead of a ConfigLoader object.

Example:

from kedro.config import TemplatedConfigLoader


class ProjectHooks:
    @hook_impl
    def register_config_loader(self, conf_paths: Iterable[str]) -> ConfigLoader:
        return TemplatedConfigLoader(
            conf_paths,
            globals_pattern="*globals.yml",
            globals_dict={"param1": "pandas.CSVDataSet"}
        )

The contents of the dictionary resulting from the globals_pattern get merged with the globals_dict. In case of conflicts, the keys in globals_dict take precedence. If the formatting key is missing from the dictionary, the default template value is used (the format is “${key|default value}”). If no default is set, a ValueError will be raised.

Global parameters can be namespaced as well. An example could work as follows:

globals.yml

bucket: "my_s3_bucket"

environment: "dev"

datasets:
    csv: "pandas.CSVDataSet"
    spark: "spark.SparkDataSet"

folders:
    raw: "01_raw"
    int: "02_intermediate"
    pri: "03_primary"
    fea: "04_feature"

catalog.yml

raw_boat_data:
    type: "${datasets.spark}"
    filepath: "s3a://${bucket}/${environment}/${folders.raw}/boats.csv"
    file_format: parquet

raw_car_data:
    type: "${datasets.csv}"
    filepath: "s3://${bucket}/data/${environment}/${folders.raw}/cars.csv"

This uses jmespath in the background. For more information see: https://github.com/jmespath/jmespath.py and https://jmespath.org/.

Methods

get(*patterns)

Tries to resolve the template variables in the config dictionary provided by the ConfigLoader (super class) get method using the dictionary of replacement values obtained in the __init__ method.

__init__(conf_paths, *, globals_pattern=None, globals_dict=None)[source]¶

Instantiate a TemplatedConfigLoader.

Parameters

conf_paths (Union[str, Iterable[str]]) – Non-empty path or list of paths to configuration directories.
globals_pattern (Optional[str]) – Optional keyword-only argument specifying a glob pattern. Files that match the pattern will be loaded as a formatting dictionary.
globals_dict (Optional[Dict[str, Any]]) – Optional keyword-only argument specifying a formatting dictionary. This dictionary will get merged with the globals dictionary obtained from the globals_pattern. In case of duplicate keys, the globals_dict keys take precedence.

get(*patterns)[source]¶

Tries to resolve the template variables in the config dictionary provided by the ConfigLoader (super class) get method using the dictionary of replacement values obtained in the __init__ method.

Parameters: *patterns – Glob patterns to match. Files, which names match any of the specified patterns, will be processed.
Return type: Dict[str, Any]
Returns: A Python dictionary with the combined configuration from all configuration files. Note: any keys that start with _ will be ignored. String values wrapped in ${…} will be replaced with the result of the corresponding JMESpath expression evaluated against globals (see __init for more configuration files. Note: any keys that start with _ details).
Raises: ValueError – malformed config found.