docs for muutils v0.6.17
View Source on GitHub

# muutils

`muutils`, stylized as "$\mu$utils" or "μutils", is a collection of miscellaneous python utilities, meant to be small and with no dependencies outside of standard python.

# installation

PyPi: muutils

    pip install muutils

Note that for using `mlutils`, `tensor_utils`, `nbutils.configure_notebook`, or the array serialization features of `json_serialize`, you will need to install with optional `array` dependencies:

    pip install muutils[array]

# documentation

hosted html docs: https://miv.name/muutils

- single-page html docs (absolute source link)
- single-page markdown docs (absolute source link)
- Test coverage: webpage (absolute source link) (plain text)

# modules

## `statcounter`

an extension of `collections.Counter` that provides "smart" computation of stats (mean, variance, median, other percentiles) from the counter object without using `Counter.elements()`

## `dictmagic`

has utilities for working with dictionaries, like:

- converting dotlist-dictionaries to nested dictionaries and back:

      >>> dotlist_to_nested_dict({'a.b.c': 1, 'a.b.d': 2, 'a.e': 3})
      {'a': {'b': {'c': 1, 'd': 2}, 'e': 3}}
      >>> nested_dict_to_dotlist({'a': {'b': {'c': 1, 'd': 2}, 'e': 3}})
      {'a.b.c': 1, 'a.b.d': 2, 'a.e': 3}

- `DefaulterDict` which works like a `defaultdict` but can generate the default value based on the key

- `condense_tensor_dict` takes a dict of dotlist-tensors and gives a more human-readable summary:

      >>> model = MyGPT()
      >>> print(condense_tensor_dict(model.named_parameters(), 'yaml'))

      embed:
          W_E: (50257, 768)
      pos_embed:
          W_pos: (1024, 768)
      blocks:
        '[0-11]':
          attn:
            '[W_Q, W_K, W_V]': (12, 768, 64)
          W_O: (12, 64, 768)
            '[b_Q, b_K, b_V]': (12, 64)
          b_O: (768,)
      <...>

## `kappa`

Anonymous gettitem, so you can do things like

    >>> k = Kappa(lambda x: x**2)
    >>> k[2]
    4

## `sysinfo`

utility for getting a bunch of system information. useful for logging.

## `misc`:

contains a few utilities: - `stable_hash()` uses `hashlib.sha256` to compute a hash of an object that is stable across runs of python - `list_join` and `list_split` which behave like `str.join` and `str.split` but for lists - `sanitize_fname` and `dict_to_filename` for simplifying the creation of unique filename - `shorten_numerical_to_str()` and `str_to_numeric` turns numbers like `123456789` into `"123M"` and back - `freeze`, which prevents an object from being modified. Also see gelidum

## `nbutils`

contains utilities for working with jupyter notebooks, such as:

- quickly converting notebooks to python scripts (and running those scripts) for testing in CI
- configuring notebooks, to make it easier to switch between figure output formats, locations, and more
- shorthand for displaying mermaid diagrams and TeX

## `json_serialize`

a tool for serializing and loading arbitrary python objects into json. plays nicely with `ZANJ`

## [`tensor_utils`]

contains minor utilities for working with pytorch tensors and numpy arrays, mostly for making type conversions easier

## `group_equiv`

groups elements from a sequence according to a given equivalence relation, without assuming that the equivalence relation obeys the transitive property

## `jsonlines`

an extremely simple utility for reading/writing `jsonl` files

## `ZANJ`

is a human-readable and simple format for ML models, datasets, and arbitrary objects. It's build around having a zip file with `json` and `npy` files, and has been spun off into its own project.

There are a couple work-in-progress utilities in `_wip` that aren't ready for anything, but nothing in this repo is suitable for production. Use at your own risk!

View Source
View on GitHub

     1"""
     2.. include:: ../README.md
     3"""
     4
     5from __future__ import annotations
     6
     7__all__ = [
     8    # submodules (with sub-submodules)
     9    "json_serialize",
    10    "logger",
    11    "misc",
    12    "nbutils",
    13    # submodules
    14    "dictmagic",
    15    "errormode",
    16    "group_equiv",
    17    "interval",
    18    "jsonlines",
    19    "kappa",
    20    "mlutils",
    21    "spinner",
    22    "statcounter",
    23    "sysinfo",
    24    "tensor_utils",
    25    "timeit_fancy",
    26    "validate_type",
    27]