Metadata-Version: 2.1
Name: machine-common-sense
Version: 0.4.1.1
Summary: Machine Common Sense Python API to Unity 3D Simulation Environment
Home-page: https://github.com/NextCenturyCorporation/MCS/
Maintainer: Next Century, a wholly owned subsidiary of CACI
Maintainer-email: mcs-ta2@machinecommonsense.com
License: Apache-2
Description: # MCS Python Package
        
        - [Installation](#installation)
        - [Download](#download)
        - [Training Datasets](#training-datasets)
        - [Usage](#usage)
        - [Logging](#logging)
        - [Run with Human Input](#run-with-human-input)
        - [Run with Scene Timer](#run-with-scene-timer)
        - [Config File](#config-file)
        - [Running Remotely](#running-remotely)
        - [Containerization](#containerization)
        - [Documentation](#documentation)
        - [Other MCS GitHub Repositories](#other-mcs-github-repositories)
        - [Troubleshooting/Email](#troubleshooting)
        - [License](#apache-2-open-source-license)
        
        ## Installation
        
        The latest release of the MCS Python library is `0.4.1.1`.
        
        ### Virtual Environments
        
        Python virtual environments are recommended when using the MCS package. All steps below presume the activation of the virtual environment as shown. These instructions are for Ubuntu linux.
        
        The Machine Common Sense package has a requirement of Python 3.6 or greater.
        
        #### Traditional Python Environment
        
        ```bash
        $ python3.6 -m venv venv
        $ source venv/bin/activate
        (venv) $ python -m pip install --upgrade pip setuptools wheel
        ```
        
        #### Anaconda Environment
        
        From the base Anaconda environment, create your project virtual environment.
        
        ```bash
        (base) $ conda create -n myenv python=3.8
        (base) $ conda env list
        # conda environments:
        #
        base                  *  /home/user/anaconda3
        myenv                    /home/user/anaconda3/envs/myenv
        (base) $ conda activate myenv
        (myenv) $
        ```
        
        ### Install MCS
        
        With the activated virtual environment, install the MCS package from the git url. MCS has a dependency on an ai2thor fork and will take a while to install. Please be patient.
        
        ```bash
        (venv) $ python -m pip install git+https://github.com/NextCenturyCorporation/MCS@master#egg=machine_common_sense
        ```
        
        ## MCS Package Developer Installation
        
        For MCS package developer, follow these [alternate instructions](./machine_common_sense/DEV.md)
        
        ## Download
        
        Here are the instructions for downloading and installing our latest Unity release. For our previous releases, please see [this page](https://github.com/NextCenturyCorporation/MCS/releases).
        
        ### Unity Application
        
        The latest release of the MCS Unity app is `0.4.1.1`.
        
        Please note that our Unity App is built on Linux or Mac.
        
        Linux Version:
        
        1. [Download the latest MCS Unity App](https://github.com/NextCenturyCorporation/MCS/releases/download/0.4.1-1/MCS-AI2-THOR-Unity-App-v0.4.1.1.x86_64)
        
        2. [Download the latest MCS Unity Data Directory TAR](https://github.com/NextCenturyCorporation/MCS/releases/download/0.4.1-1/MCS-AI2-THOR-Unity-App-v0.4.1.1_Data.tar.gz)
        
        3. [Download the latest UnityPlayer.so file](https://github.com/NextCenturyCorporation/MCS/releases/download/0.4.1-1/UnityPlayer.so)
        
        4. Ensure that both the Unity App and the TAR are in the same directory.
        
        5. Untar the Data Directory:
        
        ```
        tar -xzvf MCS-AI2-THOR-Unity-App-v0.4.1.1_Data.tar.gz
        ```
        
        6. Mark the Unity App as executable:
        
        ```
        chmod a+x MCS-AI2-THOR-Unity-App-v0.4.1.1.x86_64
        ```
        
        Mac Version:
        
        [Download the Mac ZIP](https://github.com/NextCenturyCorporation/MCS/releases/download/0.4.1-1/MCS-AI2-THOR-Unity-App-v0.4.1.1-mac.zip)
        
        
        ## Training Datasets
        
        ### Summer 2020
        
        #### Passive Agent
        
        Subtasks:
        
        - Single object scenes (~10K), just like Eval 3
        - Object preference scenes (~10K), just like Eval 3
        - Multiple agents scenes (4K), new to Eval 4
        - Instrumental action scenes (4K), new to Eval 4
        
        JSON scene configuration files:
        
        - https://eval-4-data.s3.amazonaws.com/eval_4_agent_training_dataset.zip
        
        Rendered videos:
        
        - https://nyu-datasets.s3.amazonaws.com/agent_instrumental_action_training_videos.zip
        - https://nyu-datasets.s3.amazonaws.com/agent_multiple_agents_training_videos.zip
        - https://nyu-datasets.s3.amazonaws.com/agent_object_preference_training_videos.zip
        - https://nyu-datasets.s3.amazonaws.com/agent_single_object_training_videos.zip
        
        #### Passive Intuitive Physics
        
        Please generate your own training datasets using our Scene Generator software here:
        
        - https://github.com/NextCenturyCorporation/mcs-scene-generator
        
        #### Interactive
        
        For the container, obstacle, and occluder tasks, please generate your own training datasets using our Scene Generator software here:
        
        - https://github.com/NextCenturyCorporation/mcs-scene-generator
        
        For the new interactive object permanence and reorientation tasks, please generate your own training datasets using our example scene templates here:
        
        https://github.com/NextCenturyCorporation/MCS/tree/master/machine_common_sense/scenes#interactive-object-permanence-and-reorientation-tasks
        
        ### Winter 2020
        
        *Please use the most recent 0.3.X release version*
        
        Passive Agent (only expected scenes):
        - https://evaluation-training-scenes.s3.amazonaws.com/eval3/training-single-object.zip
        - https://evaluation-training-scenes.s3.amazonaws.com/eval3/training-object-preference.zip
        
        Passive Intuitive Physics (only plausible scenes):
        - https://evaluation-training-scenes.s3.amazonaws.com/eval3/training-passive-physics.zip
        
        Example Scenes:
        - https://github.com/NextCenturyCorporation/MCS/tree/master/machine_common_sense/scenes
        
        ### Summer 2020
        
        *Use a release version between 0.0.9 and 0.1.0*
        
        Interactive:
        https://evaluation2-training-scenes.s3.amazonaws.com/interaction-scenes.zip
        
        Passive IntPhys:
        https://evaluation2-training-scenes.s3.amazonaws.com/intphys-scenes.zip
        
        Passive IntPhys Validation Data:
        https://evaluation2-training-scenes.s3.amazonaws.com/validation-intphys-scenes.zip
        
        ## Usage
        
        Example usage of the MCS library:
        
        ```python
        import machine_common_sense as mcs
        
        # We will give you the Unity app file.
        controller = mcs.create_controller(unity_app_file_path, config_file_path='./some-path/config.ini')
        
        # Either load the scene data dict from an MCS scene config JSON file or create your own.
        # We will give you the training scene config JSON files and the format to make your own.
        scene_data, status = mcs.load_scene_json_file(scene_json_file_path)
        
        output = controller.start_scene(scene_data)
        
        # Use your machine learning algorithm to select your next action based on the scene
        # output (goal, actions, images, metadata, etc.) from your previous action.
        action, params = select_action(output)
        
        # Continue to select actions until your algorithm decides to stop.
        while action != '':
            controller.step(action, params)
            action, params = select_action(output)
        
        # For interaction-based goals, your series of selected actions will be scored.
        # For observation-based goals, you will pass a classification and a confidence
        # to the end_scene function here.
        controller.end_scene()
        ```
        
        Example running multiple scenes sequentially:
        
        ```python
        import machine_common_sense as mcs
        
        # Only create the MCS controller ONCE!
        controller = mcs.create_controller(unity_app_file_path)
        
        for scene_json_file_path in scene_json_file_list:
            scene_data, status = mcs.load_scene_json_file(scene_json_file_path)
            output = controller.start_scene(scene_data)
            action, params = select_action(output)
            while action != '':
                controller.step(action, params)
                    action, params = select_action(output)
            controller.end_scene()
        ```
        
        ### Logging
        #### Initialize development logging config settings
        In startup script make sure you include:
        ```python
        import machine_common_sense as mcs
        
        #This code initializes logging using config files
        mcs.init_logging()
        ```
        #### Logging configuration files
        This logging configuration system uses a file at [scripts/log.config.py](./scripts/log.config.py) as a default configuration file.  See that file for links to documentation and some descriptions.
        
        Users can copy that file to scripts/log.config.user.py to create their own configuration.  This file is gitignored so it will not be shared.
        
        #### Why use logging and configuration files
        
        We use logging configuration files so users have a reasonable default configuration but also have the ability to override when circumstances demand.  This eliminates the need to alter code when changing what logging is necessary.  In the future, additional features to the initialization function to support difficult overrides if necessary (i.e. different logging override files for dev, test, eval, etc).
        
        #### Example with logging config file:
        
        Sample of how to log in all files:
        ```python
        import logging
        
        # logging by name is useful to turn certain logs on/off and
        # also to help determine where an error log might have originated from
        logger = logging.getLogger(__name__)
        
        def any_function(self):
            #basic log functions in order of level.  If logger or handler is set to a level, that level and above will be logged.
            logger.critical("This message is see as long as the logger is enabled")
            logger.error("This message reports an error")
            logger.info("This message is general info")
            logger.debug("This message is mainly for debugging and shouldn't cause usage problems if missing")
            # Unlike many common loggers in other languages, python logging doesn't seem to have TRACE
        ```
        
        ## Run with Human Input
        
        To start the Unity application and enter your actions and parameters from the terminal, you can run the `run_in_human_input_mode` script that was installed in the package with the MCS Python Library (the `mcs_unity_build_file` is the Unity executable downloaded previously):
        
        ```
        run_in_human_input_mode <mcs_unity_build_file> <mcs_scene_json_file>
        ```
        
        Run options:
        - `--config_file_path <file_path>`
        
        ## Run with Scene Timer
        
        To run the Unity application and measure your runtime speed, you can run the `run_scene_timer` script that was installed in the package with the MCS Python Library:
        
        ```
        run_scene_timer <mcs_unity_build_file> <mcs_scene_file_folder>
        ```
        
        This will run all of the MCS scene configuration JSON files in the given folder, use the PASS action for 20 steps (or for a number of steps equal to the last_step of the scene file's goal, if any) in each scene, and print out the total, average, minimum, and maximum run time for all the scenes and the steps.
        
        ## Config File
        
        To use an MCS configuration file, you can either pass in a file path via the `config_file_path` property in the create_controller() method, or set the `MCS_CONFIG_FILE_PATH` environment variable to the path of your MCS configuration file (note that the configuration must be an INI file -- see [sample_config.ini](./sample_config.ini) for an example).
        
        ### Config File Properties
        
        #### history_enabled
        
        (boolean, optional)
        
        Whether to save the scene history output data in your local directory. Default: True
        
        #### metadata
        
        (string, optional)
        
        The `metadata` property describes what metadata will be returned by the MCS Python library. The `metadata` property is available so that users can run baseline or ablation studies during training. It can be set to one of the following strings:
        
        - `oracle`: Returns the metadata for all the objects in the scene, including visible, held, and hidden objects. Object masks will have consistent colors throughout all steps for a scene.
        - `level2`: Only returns the images (with depth maps AND object masks), camera info, and properties corresponding to the player themself (like head tilt or pose). No information about specific objects will be included. Note that here, object masks will have randomized colors per step.
        - `level1`: Only returns the images (with depth maps but NOT object masks), camera info, and properties corresponding to the player themself (like head tilt or pose). No information about specific objects will be included.
        - `none`: Only returns the images (but no depth maps or object masks), camera info, and properties corresponding to the player themself (like head tilt or pose). No information about specific objects will be included.
        
        If no metadata level is set:
        - `default`: Fallback if no metadata level is specified. Only meant for use during development (evaluations will never be run this way). Includes metadata for visible and held objects in the scene, as well as camera info and properties corresponding to the player. Does not include depth maps or object masks.
        
        #### noise_enabled
        
        (boolean, optional)
        
        Whether to add random noise to the numerical amounts in movement and object interaction action parameters. Will default to `False`.
        
        #### save_debug_images
        
        (boolean, optional)
        
        Save RGB frames, depth masks, and object instance segmentation masks (if returned in the output by the chosen metadata tier) to image files on each step. Default: False
        
        #### save_debug_json
        
        (boolean, optional)
        
        Save AI2-THOR/Unity input, AI2-THOR/Unity output, and MCS StepMetadata output to JSON file on each step. Default: False
        
        #### seed
        
        (int, optional)
        
        A seed for the Python random number generator (defaults to None).
        
        #### size
        
        (int, optional)
        
        Desired screen width. If value given, it must be more than `450`. If none given, screen width will default to `600`.
        
        #### video_enabled
        
        (boolean, optional)
        
        Create and save videos of the RGB frames, depth masks, object instance segmentation masks (if returned in the output by the chosen metadata tier), 2D top-down scene views, and the heatmap images given to us in `make_step_prediction` by the AI performer. Default: False
        
        ### Using the Config File to Generate Scene Graphs or Maps
        
        1. Save your .ini MCS configuration file with:
        ```
        [MCS]
        metadata: oracle`
        ```
        
        2. Create a simple Python script to loop over one or more JSON scene configuration files, load each scene in the MCS controller, and save the output data in your own scene graph or scene map format.
        
        ```python
        import os
        import machine_common_sense as mcs
        
        os.environ['MCS_CONFIG_FILE_PATH'] = # Path to your MCS configuration file
        
        scene_files = # List of scene configuration file paths
        
        unity_app = # Path to your MCS Unity application
        
        controller = mcs.create_controller(unity_app)
        
        for scene_file in scene_files:
            scene_data, status = mcs.load_scene_json_file(scene_file)
        
            if status is not None:
                print(status)
            else:
                output = controller.start_scene(scene_data)
                # Use the output to save your scene graph or map
        ```
        
        ### Example Scene Configuration Files
        
        [machine_common_sense/scenes/README.md](./machine_common_sense/scenes/README.md)
        
        ## Running Remotely
        
        To run MCS on a remote GPU server, use the following steps to launch an X11 server.
        
        ```bash
        # query the gpu bus ID
        $ nvidia-xconfig --query-gpu-info
        
        GPU #0:
          Name      : Tesla K80
          UUID      : GPU-d03a3d49-0641-40c9-30f2-5c3e4bdad498
          PCI BusID : PCI:0:23:0
        # create the xserver configuration
        $ sudo nvidia-xconfig --user-display-device=None --virtual=600x400 --output-xconfig=/etx/X11/xorg.conf --busid=PCI:0:23:0
        # launch Xserver
        $ sudo /usr/bin/Xorg :0 &
        # test using glxinfo
        $ DISPLAY=:0 glxinfo
        name of display: :0
        display: :0  screen: 0
        direct rendering: Yes
        server glx vendor string: NVIDIA Corporation
        server glx version string: 1.4
        
        ```
        
        ### Remote Run Test Script
        
        Run the following script to test MCS with the X11 server created above.
        
        ```python
        # test.py
        import machine_common_sense as mcs
        # use your path to the MCS Unity executable
        controller = mcs.create_controller('MCS.x86_64')
        # find a test scene
        scene_file_path = 'playroom.json'
        scene_data, status = mcs.load_scene_json_file(scene_file_path)
        scene_file_name = scene_file_path[scene_file_path.rfind('/')+1]
        if 'name' not in scene_data.keys():
            scene_data['name'] = scene_file_name[0:scene_file_name.find('.')]
        output = controller.start_scene(scene_data)
        for i in range(1, 12):
            output = controller.step('RotateLook')
            for j in range(len(output.image_list)):
                output.image_list[i].save(f'{i}-{j}.jpg')
        ```
        
        From your python environment, run test.py and check the output images for proper rendering.
        
        ```
        DISPLAY=:0 python test.py
        ```
        
        ## Containerization
        
        ### GPU Image
        Please note that the GPU image requires an Nvidia GPU and Nvidia Docker to be installed.
        
        You can run the GPU image with:
        ```shell
        docker run -it -e PYTHONIOENCODING=utf8 -e XAUTHORITY=/tmp/.docker.xauth -e DISPLAY=:1 \
                   -v ${PWD}/machine_common_sense/scenes:/input -v ${PWD}/scripts:/scripts \
                   -v /tmp/.X11-unix:/tmp/.X11-unix \
                   -v /tmp/.docker.xauth:/tmp/.docker.xauth \
                   --net host --gpus all --rm mcs-playroom:0.0.6 bash
        ```
        
        You can then run a scene like this:
        ```shell
        python3 /scripts/run_human_input.py /mcs/MCS-AI2-THOR-Unity-App-v0.4.0.x86_64 --config_file_path /scripts/config_oracle.ini /input/hinged_container_example.json
        ```
        
        #### Missing X Authorization Error
        If you encounter an error like the following:
        ```
        config_file_path $MCS_CONFIG_FILE_PATH /input/agents_preference_expected.json 
        No protocol specified
        xdpyinfo:  unable to open display ":1".
        Exception in create_controller() Invalid DISPLAY :1 - cannot find X server with xdpyinf
        ```
        please try resetting your X authorization:
        ```shell
        sudo rmdir /tmp/.docker.xauth
        touch /tmp/.docker.xauth
        xauth nlist $DISPLAY | sed -e 's/^..../ffff/' | xauth -f /tmp/.docker.xauth nmerge -
        ```
        
        ### CPU Image
        
        There is an image to run CPU-only without requiring an Nvidia GPU. Please only use it if your system does not meet the
        prerequisites for the GPU image, since GPU acceleration will yield significantly better performance.
        
        #### Build Image
        ```shell
        docker build -f CPU_Container.dockerfile -t mcs-playroom-cpu .
        ```
        
        #### Run Image (bash)
        ```shell
        docker run -it -p 5900:5900 -v ${PWD}/machine_common_sense/scenes:/input -v ${PWD}/scripts:/scripts mcs-playroom-cpu bash
        ```
        
        #### Run with VNC
        Unless stated otherwise, the following commands are intended to be run inside the container.
        Run tmux with `tmux` and open two panes via `C-b %`.
        ```shell
        Xvnc :33 &
        export DISPLAY=:33
        python3 /scripts/run_human_input.py ${MCS_EXECUTABLE_PATH} --config_file_path ${MCS_CONFIG_FILE_PATH} /input/hinged_container_example.json
        ```
        
        Switch panes with `C-b <arrow>`
        ```shell
        window_id=$(xwininfo -root -tree | grep MCS-AI2-THOR | tail -n1 | sed "s/^[ \t]*//" | cut -d ' ' -f1) && echo ${window_id}
        x11vnc -id ${window_id} &
        ```
        
        Afterwards, you should be able to connect to the VNC server from the host by running `vncviewer` and connecting to
        `localhost:5900`.
        
        #### Run Fully Headless
        
        As an alternative for batch runs you can also run MCS against X virtual framebuffers. In this case you do not get visual
        output, but can run the images on headless servers without X server. To do so, please execute the following command
        from inside the container:
        
        ```shell
        xvfb-run -s "-screen 0 1440x900x24" python3 /scripts/run_human_input.py ${MCS_EXECUTABLE_PATH} --config_file_path ${MCS_CONFIG_FILE_PATH} /input/agents_preference_expected.json
        ```
        
        ## Documentation
        
        - [Python API](./machine_common_sense/API.md)
        - [Example Scene Configuration Files](./machine_common_sense/scenes/README.md)
        - [Scene Configuration JSON Schema](./machine_common_sense/scenes/SCHEMA.md)
        - [Developer Docs](./machine_common_sense/DEV.md)
        
        ## Other MCS GitHub Repositories
        
        - [Unity code](https://github.com/NextCenturyCorporation/ai2thor)
        - [Scene Generator](https://github.com/NextCenturyCorporation/mcs-scene-generator)
        - [Data Ingest](https://github.com/NextCenturyCorporation/mcs-ingest)
        - [Evaluation Dashboard (UI)](https://github.com/NextCenturyCorporation/mcs-ui)
        
        ## Troubleshooting
        
        [mcs-ta2@machinecommonsense.com](mailto:mcs-ta2@machinecommonsense.com)
        
        ## Apache 2 Open Source License
        
        Code in this repository is made available by [CACI][4] (formerly [Next Century
        Corporation][1]) under the [Apache 2 Open Source License][2].  You may
        freely download, use, and modify, in whole or in part, the source code
        or release packages. Any restrictions or attribution requirements are
        spelled out in the license file.  For more information about the
        Apache license, please visit the [The Apache Software Foundation’s
        License FAQ][3].
        
        [1]: http://www.nextcentury.com
        [2]: http://www.apache.org/licenses/LICENSE-2.0.txt
        [3]: http://www.apache.org/foundation/license-faq.html
        [4]: http://www.caci.com
        
        Copyright 2021 CACI (formerly Next Century Corporation)
        
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Requires-Python: >3.6
Description-Content-Type: text/markdown
