.. _install: Installing Zoe ============== If you are looking for the five-minutes install procedure, just for testing, check the :ref:`test install ` section, below. When installing Zoe for production you should, first of all, look at the following requirements and take a decision about each of them: * Container back-end * Shared filesystem: we know of NFS or CephFS, but anything similar should work * Network: how your users will connect to the containers * Authentication back-end: how your users will authenticate to Zoe (LDAP or text file) * How to manage Zoe Applications (ZApps) * ZApp output logs: see :ref:`logging` After, you can start the installation, as outlined in the :ref:`manual install ` section. Choosing the container back-end ------------------------------- At this time Zoe supports three back-ends: * Docker engine: useful for running Zoe on a single PC, it is the simplest to setup. * Legacy Docker Swarm: simple to install, additional features like SSL, high-availability and dynamic host discovery can be added as needed. Please note that Zoe does not support the new Swarm Mode of Docker Engine as the API is too limited. * Kubernetes: the most complex to setup, we suggest using it only if you already have (or need) a Kubernetes setup for running other software. Shared filesystem ----------------- Users need to put data and binaries in a place accessible by Zoe and need to be able to access the results and the logs generated by running ZApp. Zoe uses the concept of workspaces: each user has a private directory that is attached to all the containers of each ZApp belonging to her in a well-known location. This filesystem can be accessed by a special gateway container spawned by the administrator (see `gateway containers `_) or by other methods (direct mount on user machines, webdav, web file managers). Zoe implements a "directory" back-end for workspaces. Container back-ends may implement more volume technologies: Zoe is not involved, it needs only the information on how to attach the user volume to the container, so the effort required to support new volume types should be minimal. At Eurecom we use CephFS, but we know of successful Zoe deployments based on NFS. Networking ---------- Most of the ZApps expose a number of interfaces (web, REST and others) to the user. Zoe configures the active back-end to expose these ports, but does not perform any additional action to configure routing or DNS to make the ports accessible. Keeping in mind that the back-end network configuration is outside Zoe's competence area, here there a non-exhaustive list of the possible configurations: * expose the hosts running the containers by using public IP addresses * use a proxy, like the one developed for Zoe: :ref:`proxy` * use back-end network plugins to build custom topologies Authentication back-end ----------------------- Zoe has a simple user model: users are authenticated against an external source of truth, that assigns also one of three roles: * guest: cannot access the API (and the command-line tools) and can run one execution at a time * user: can use the API and has no limits on executions * admin: can operate on executions belonging to other users, can delete records of past executions Zoe supports two authentication back-ends: * LDAP and LDAP+SASL * Text file As most of Zoe, the authentication back-end is pluggable and others can be easily implemented. LDAP ^^^^ Plain LDAP or LDAP+SASL GSSAPI are available. The implementation will use the ``uid`` as the user_id and ``gidNumber`` to decide the role. In Zoe configuration you need to specify the mapping between group IDs and roles. Text file ^^^^^^^^^ For testing and for simple deployments with a few users, a CSV text file can be used. Its format is:: ,, The file location can be specified in the ``zoe.conf`` file and it needs to be readable only be the Zoe processes. Managing Zoe applications ------------------------- At the very base, ZApps are composed of a container image and a JSON description. The container image can be stored in a local, private, registry, or in a public one, accessible via the Internet. Zoe does not provide a way to automatically build images, push them to a local registry, or pull them to the hosts when needed. At Eurecom we provide an automated environment based on GItLab's CI features: users are able to customize their applications (JSON and Dockerfiles) by working on git repositories. Images are rebuilt and pushed on commit and JSON files are generated and copied to the ZApp shop directory. You can check out how we do it here: https://gitlab.eurecom.fr/zoe-apps The ZApp Shop ^^^^^^^^^^^^^ The Zoe web interface provides a ZApp shop to showcase available ZApps and have a friendly and easy way for users to list and access ZApps. The shop is managed locally. It looks for ZApps in a configured directory (option ``zapp-shop-path``). Each ZApp must live in its own directory, that must contain: * manifest.json : a JSON file that describes the contents of the ZApp * a logo that is displayed on the web interface * one or more text files in markdown format with ZApp information and documentation * one or more JSON Zoe application descriptions The ``manifest.json`` file drives the ZApp Shop. Its format is as follows:: { "version": 1, "zapps": [ { "category": "TensorFlow", "name": "Google TensorFlow notebook", "description": "tf-google.json", "readable_descr": "README-goog.md", "parameters": [] }, { "category": "TensorFlow", "name": "Google TensorFlow batch", "description": "tf-google.json", "readable_descr": "README-batch.md", "parameters": [ { "kind": "command", "name": "tf-jupyter", "readable_name": "Command", "description": "The Python script to run, relative to the workspace directory", "type": "string", "default": "./my-tf-app/main.py" } ] }, { "category": "TensorFlow", "name": "DRAGNN SyntaxNet model", "description": "stnet-google.json", "readable_descr": "README-syntaxnet.md", "parameters": [] }, { "category": "TensorFlow", "name": "Magenta model", "description": "mag-google.json", "readable_descr": "README-magenta.md", "parameters": [] } ] } * version : a internal version, used by Zoe to recognize the manifest format. For now only 1 is supported. * zapps : a list of related ZApps that have to be shown in the shop For each ZApp: * category : the category this ZApp belongs to, it is used to group ZApps in the web interfaces * name : the human-readable name * description : the name of the json file with the Zoe description * readable_descr : the name of the markdown file containing user documentation for the ZApp * parameters : a list of parameters the user can set to tune the ZApp before starting it Parameters: Parameters are values of the JSON description that are modified at run time. * kind : the kind of parameter, it can be ``command`` or ``environment`` * name : the machine-friendly name of the parameter * readable_name : the human-friendly name of the parameter * description : an helpful description * type : string or integer, used for basic for validation * default : the default value Parameters can be of two kinds: * environment : the parameter is passed as an environment variable. The name of the environment variable is stored in the ``name`` field. The JSON description is modified by setting the user-defined value in the environment variable with the corresponding name. All services that have the variable defined are modified. * command : the service named ``name`` has its start-up command changed to the user-defined value To get started, in the ``contrib/zapp-shop-sample/`` directory there is a sample of the structure needed for a working zapp-shop, including some data science related ZApps. Copy it as-is in your ZApp shop directory to have some Zapps to play with. Example of distributed environment ---------------------------------- For running heavier workloads and distributed applications, you need a real container cluster. In this example we will use Docker Swarm, as it is simpler to setup than Kubernetes. Software: * Docker Swarm * Optional: ZooKeeper for Docker Engine discovery by Swarm (etcd and consul are also supported by Swarm) * Zoe * NFS (or another distributed filesystem like CephFS) Swarm supports several discovery mechanisms and you should select the most appropriate for your environment. An explicit host list is the most simple to setup, while ZooKeeper provides a resilient and dynamic way of attaching Docker Engines to Swarm. Topology: * On node running Zoe, a Docker Engine and the following containers: * Swarm manager * Docker registry (Optional, but strongly suggested) * Optional `gateway containers `_ for SSH and/or SOCKS proxies * Three nodes for ZooKeeper: if used only for Swarm the load will be very low and ZooKeeper can be co-located with Zoe and the Docker Engines * At least one more node with a Docker Engine * A file server running NFS: depending on the workload it can be co-located with Zoe and the Swarm manager * Another node running a Swarm manager, if you want to enable HA for Swarm To configure Swarm and the Registry, please refer to the documentation available on the Docker website. To configure container networking, we suggest the standard Swarm overlay network. In this configuration Zoe expects the network filesystem to be mounted in the same location on all hosts registered in Swarm. This location is specified in the ``workspace-base-path`` Zoe configuration item. Zoe will create a directory under it named as ``deployment-name`` by default or ``workspace-deployment-path`` if specified. Under it a new directory will be created for each user accessing Zoe. .. _test-install-label: Stand-alone environment for development and testing --------------------------------------------------- A simple deployment for development and testing is possible with just: * A Docker Engine * Zoe Please note: since Zoe will use the Swarm API to talk to the Docker Engine, the addresses of the exposed ports for running ZApp may be ``0.0.0.0``, causing the links generated by the web interface and the command line tool to be wrong. ZApps can be accessed by using ``127.0.0.1`` instead. Docker compose ^^^^^^^^^^^^^^ In the root of the repository you can find a ``docker-compose.yml`` file that should help get you started. Look also at the deployment scripts below, as they provide an option for a simple, zoe-on-a-laptop, install. Deployment scripts ^^^^^^^^^^^^^^^^^^ Refer to `zoe-deploy `_ repository for automated deployment scripts for configurations with Swarm or Kubernetes back-ends. .. _manual-install-label: Manual install (recommended for production) ------------------------------------------- This section shows how to install the components outlined in the distributed environment outlined above. A lot of other options and possibilities exist for deploying Zoe. Requirements ^^^^^^^^^^^^ * Python 3. Development happens on Python 3.4 and 3.5. * Docker Swarm * A shared filesystem, mounted on all hosts part of the Swarm. Optional: * A Docker registry containing Zoe images for faster container startup times * A logging pipeline able to receive GELF-formatted logs, or a Kafka broker Swarm/Docker ^^^^^^^^^^^^ Install Docker and the Swarm container: * https://docs.docker.com/installation/ubuntulinux/ * https://docs.docker.com/swarm/install-manual/ Network configuration ^^^^^^^^^^^^^^^^^^^^^ Docker 1.9/Swarm 1.0 multi-host networking can be used in Zoe: * https://docs.docker.com/engine/userguide/networking/get-started-overlay/ This means that you will also need a key-value store supported by Docker. We use Zookeeper, it is available in Debian and Ubuntu without the need for external package repositories and is very easy to set up. Images: Docker Hub Vs local Docker registry ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A few sample ZApps have their images available on the Docker Hub. We strongly suggest setting up a private registry, containing your customized Zoe Service images. Zoe ^^^ Zoe is written in Python and uses the ``requirements.txt`` file to list the package dependencies needed for all components of Zoe. Not all of them are needed in all cases, for example you need the ``kazoo`` library only if you use Zookeeper to manage Swarm high availability. Currently this is the recommended procedure, once the initial Swarm setup has been done: 1. Clone the zoe repository 2. Install Python package dependencies: ``pip3 install -r requirements.txt`` 3. Create new configuration files for the master and the api processes (:ref:`config_file`), you will need also access to a postgres database 4. Setup supervisor to manage Zoe processes: in the ``contrib/supervisor/`` directory you can find the configuration file for supervisor. You need to modify the paths to point to where you cloned Zoe and the user (Zoe does not need special privileges). 5. Start running ZApps! In case of troubles, check the logs for errors. Zoe basic functionality can be tested via the ``zoe.py stats`` command. It will query the ``zoe-api`` process, that in turn will query the ``zoe-master`` process. .. _api-manager-label: API Managers ------------ To provide TLS termination, authentication, load balancing, metrics, and other services to the Zoe API, you can use an API manager in front of the Zoe API. For example: * Tyk: https://tyk.io/tyk-documentation/get-started/with-tyk-on-premise/ * Kong: https://getkong.org/docs/0.10.x/proxy/