Dockers: part 4 - Managing multiple services

Introduction:

1. Your application might have a lot of services. Its best to run each individual service inside their own docker container.
2. However, dealing manually with multiple docker containers is hard. Building them, bringing them up and down, linking them withe each other and killing them are not so easy tasks to do one by one manually.
3. docker-compose is a utility who makes dealing with orchestration processes (start, shutdown, link, volumes) of docker containers very easy task.
4. Using compose is a three step process
   • Define your app's environment with DockerFile.
   • Define the services that make up your app in docker-compose.yml so they can be run together in an isolated environment.
   • Use docker-compose commands to manage whole lifecycle of your application:
         Start/Stop/rebuild services
         View status of running services
         Stream the log output of running services
         Run a one-off command on service
5. Read "docker-compose File" section to see how the .yml file looks like.

Installation (Ubuntu):

1. Install Docker. See instructions here.
2. Run following
   

   curl -L https://github.com/docker/compose/releases/download/1.6.2/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose

3. Make the binary executable
   chmod +x /usr/local/bin/docker-compose
4. Test installation
   docker-compose --version

docker-compose File:

build:

Configuration options that are applied at build time.

context:     Define build context.

dockerfile: File from which image needs to be build.

image:

Image to start the container from.

- If the image does not exist, Compose attempts to pull it.
- In case, you have also specified "build", image will be build from dockerfile and tags it with the "image" tag.
- Default name of the image in absence of "image" would be <project_name>_<service_name>.
container_name: Specify a custom container name, rather than a generated default name.
entrypoint:           Override the default entrypoint.
command:            Override the default command.
ports:

  - Expose ports in form of (HOST:CONTAINER).
  - NOTE: HOST part is optional. In absence of HOST, a random host port will be chosen.
volumes:
  - Mount paths or named volumes in form of HOST:CONTAINER.

Example docker-compose file:

version: '2'
services:
  server:
    build:
       context: ./riemann_server
       dockerfile: Dockerfile

    entrypoint: ["/bin/nc"]
   command: ["-luv", "15555"]
   image: server_img
   container_name: server
   ports:
     - "15555:15555/udp"
   volumes:
     - /opt/pg/log:/opt/pg/log
 client:
   build:
     context: ./log_aggregator
     dockerfile: Dockerfile
   image: clinet_img
   container_name: client
   links:
     - server
   entrypoint: ["/bin/nc"]
   command: ["-vu", "server", "15555"]
   depends_on:
     - server

Dockerfiles for both client and server are holding only one line i.e "From ubuntu:14.04.3".

NOTE: For further details, see this.

docker-compose CLI:

You can see help using

docker-compose --help

build:

• Services are build and tagged as "project_service".
• In case of change in docker file or directory, run command again to rebuild.

config:   Validate and view the compose file.

create:  Create containers for service.

start:     Start existing containers for service.

up:         Builds, (re)creates, starts, and attaches to containers for a service.

ps:         List containers.

Stop:     Stops running containers without removing them

rm:        Removes stopped service containers.

kill:        Forces running containers to stop by sending a SIGKILL signal

down:    Stop containers and remove containers, networks.

Docker Cheatsheet

BUILD
sudo docker build --tag <image-name> --build-arg argument=value .

REMOVE:
image:                    docker rmi <image-name>
container:               docker rm <container-name>
all-image:               docker rmi $(sudo docker images -qf "dangling=true")
all-container:          docker rm $(sudo docker ps -aq)

LIST:
images:                    docker images
containers:               docker ps -a

RUN:
docker run --name <container-name-once-it-runs> -ti <image-name> /bin/bash

EXEC:
sudo docker exec -ti <container-name> /bin/bash
e.g. sudo docker exec <container-name> /bin/sh -c "echo 172.17.0.2 dockerelk_logstash_1 >> /etc/hosts"

Get IP address:
sudo docker inspect --format '{{ .NetworkSettings.IPAddress }}' <container-name>

Dockers: Part 3 - DockerFile instructions

A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image.It actually allows you to automate the steps you would normally manually take to create an image.

Below is the format of DockerFile instructions 

INSTRUCTION arguments

DockerFile Instructions:

Docker runs the instructions in Dockerfile in order.

FROM:

First instruction must be FROM. It will specify the base image on top of which you will build your images.
FROM <image>
FROM <image>:<tag>

FROM ubuntu:14.04.3

ENV:

1. Sets the environment variable. 
   ENV var val
   ENV var1="val1" var2="val2"
2. It also makes interpolation in the dockerfile available from the next statement via ${var}.
3. These variables will be persisted into any containers created from the image where these variables are defined.

RUN:

1. Run a command and commit the image.
   RUN <command>                                          # Shell form. 
   RUN ["executable" "param1" "param2"]        # exec form. 
2. Shell form: Args are automatically prepended with "/bin/sh -c".
   Exec form: makes it possible to avoid shell string prepending, and to RUN commands using a base image that does not contain /bin/sh.RUN echo hello                        # /bin/sh -c echo hello
   RUN [ "echo", "hello" ]            # echo hi
3. To use a different shell other that /bin/sh and keep using the shell form, you can do following on top of DockerFile.
   RUN rm /bin/sh && ln -s /bin/bash /bin/sh
4. The RUN instruction will execute any commands in a new layer on top of the current image and commit the results. The resulting committed image will be used for the next step in the Dockerfile.
   Containers can be created from any point in an image’s history.
5. Use backslash "\" to continue the RUN command on multiple lines.

CMD:

1. CMD specifies the command to run when the container is launched.
   In another form, it can also specify the arguments to ENTRYPOINT.
   CMD ["executable" "param1" "param2"]
   CMD ["param1" "param2"]                            # as default parameters to ENTRYPOINT. ENTRYPOINT is must in this case.
2. CMD defaults will be override by run arguments.
   sudo docker run -it <container-name> <arguments which will override>.

NOTE: There can be only one CMD command in Dockerfile.

ENTRYPOINT:

1. Allow you to specify the default executable.
2. CMD contents will be passed as argument to ENTRYPOINT.
3. Any "docker run" arguments will be passed as parameter to ENTRYPOINT.
4. To override DockerFile EntryPoint, you need to supply --entrypoint to docker run
   sudo docker run -it --entrypoint="/bin/bash" ent_cmdd
5. Example:
   FROM ubuntu:14.04.3
   ENTRYPOINT ["/bin/ping"]
   CMD ["localhost" "-c 2"]
   NO argument (sudo docker -it ent_cmd)                    : ping localhost
   argument (sudo docker run -it ent_cmd google.com) : ping google.com
   
   NOTE: You can use the exec form of ENTRYPOINT to set fairly stable default commands and arguments and then use CMD to set additional defaults that are more likely to be changed.

LABEL:

1. LABEL adds metadata to the image in key-value pair format.
2. LABEL multi.label1="value1" \
   multi.label2="value2" \
   other="value3"
   LABEL version="0.5.1" \
   Description="This image holds Riemann Server."
3. New label values will override previous labels. So, your base image key value will be override by new value.
4. Docker inspect <image-name>will show you image labels..

EXPOSE:

1. The EXPOSE instruction informs Docker that the container listens on the specified network ports at runtime. 
2. EXPOSE does NOT make the ports of the container accessible to the host. Exposed ports are just available to processes inside the container.
3. Given the limitation of the EXPOSE instruction, a Dockerfile author will often include an EXPOSE rule only as a hint to which ports will provide services.
4. One can see ExposedPorts of image under Config section using
sudo docker inspect <image-name>
5. To make the ports of the container accessible to the host, you must use either the -p flag to publish a range of ports or the -P flag to publish all of the exposed ports.
   EXAMPLE:
   FROM ubuntu:14.04.3
   ENTRYPOINT [ "/bin/nc" ]
   CMD [ "-luv","6871" ]         # (nc -luv 6871) will set a udp server listening on port 6871.
   EXPOSE 6871/udp
   sudo docker build -t expose .
   sudo docker run -ti expose
   From host try to send udp packet to udp port 6871.
   nc -vu 127.0.0.1 6871
   Nothing will receive inside container because port is exposed (mere a hint) but not published (only publish will make it accessible to host).
6. sudo docker run -ti -p 127.0.0.1:6871:6871/udp expose
   nc -vu 127.0.0.1 6871
   Now typing anything on prompt will be received inside docker.
7. Moreover, you can see the port mapping as well like given below
   sudo docker ps -l
8. CONTAINERID    IMAGE   COMMAND   PORTS   NAMES
   facbb246bfe6         expose     "/bin/nc -luv    6871"      127.0.0.1:6871->6871/udp evil_darwin
   NOTE: Format for publishing a container᾿s port or a range of ports to the host: ip:hostPort:containerPort | ip::containerPort | hostPort:containerPort | containerPort
   containerPort is necessary. In case you donot provide hostPort, Docker will assign some available random host port.
   The port number inside the container (where the service listens) does not need to match the port number exposed on the outside of the container (where clients connect).

WORKDIR:

1. WORKDIR instruction sets the working directory for ADD, COPY, RUN, CMD, ENTRYPOINT.
2. EXAMPLE: 
   FROM ubuntu:14.04.3
   WORKDIR /var/log
   ENTRYPOINT ["/bin/ls"]
   Upon running above container, you will see contents of /var/log.
3. It can resolve environment variables set by ENV.
   ENV DIR /varWORKDIR $DIR/logENTRYPOINT ["/bin/ls"]
4. It can not expand linux variables like $USER or $HOME etc.

ADD:

1. ADD instructions adds <src> from build-context to <dst>.ADD <src> <dst>
2. <src> can be a file, directory, URL or tar file. In case of "tar" source, Docker will untar and then copy.
3. If <dst> does not exist, ADD will create the full path for us.
4. New files will be created with 755 permission. In case of URL, permissions will be 600.
5. <src> may contain the regular expression.
   ADD hom* /mydir/
6. You can not add from outside of build or context.

COPY:

Copy is same as 'ADD' but without the tar and remote url handling.
Docker's best practice document suggests to use COPY because it is more transparent than ADD.

USER:

1. USER instruction will set the user for any following RUN, CMD, ENTRYPOINT command and to use when running the image.
2. It can be override by using -u flag with "run" command.

ARG:

1. The ARG instruction defines a variable that users can pass at build-time to the builder with the docker build command using the --build-arg <varname>=<value> flag.
   ARG logstash_version 
2. User can define the default value for argument as well.
   ARG logstash_version=1.2.3
3. User will build the image using
   docker build --build-arg logstash_version=2.2.2 .

ONBUILD:

1. The ONBUILD instruction adds to the image a trigger instruction.
2. Instructions will be triggered later when the image is used as the base for another build. 
3. The trigger will be executed in the context of the downstream build, as if it had been inserted immediately after the FROM instruction in the downstram Dockerfile.

Docker: Part 2 - Ubuntu Installation

You can install docker by following below commands

1. wget -qO- https://get.docker.com/ | sh 
   The above command downloads and executes a small installation script written by the Docker team.
2. Docker daemon will be run automatically after installation. To check whether it is running, use following command
   sudo service docker status
3. The docker daemon binds to a Unix socket (/var/run/docker.sock) instead of a TCP port. By default that Unix socket is owned by the user root and other users can access it with sudo.
   For this reason, docker daemon always runs as the root user.
4. "docker" binary acting as a client of daemon also needs to be run as a root.
5. To avoid having to use sudo when you use the docker command, create a Unix group called docker and add users to it. When the docker daemon starts, it makes the ownership of the Unix socket read/writable by the docker group.
   sudo usermod -aG docker <user>
6. We can use -H/--host flag (specify daemon socket to connect to) to specify different interface and port configuration. For example, binding to the network
   sudo /usr/bin/docker -d -H tcp://0.0.0.0:2375
   above command would bind to all interfaces of host.
7. Client will not become informed about interface change automatically. You have to pass -H flag while running commands
   docker -H :2375
8. If you do NOT want to give the -H flag in every command, you can export environment variable export
   DOCKER_HOST="tcp://0.0.0.0:2375"

Finally, verify the installation using
docker run hello-world

Dockers: Part 1 - Basic Components

Docker is an open-source engine that automates the deployment of applications into containers.

Docker architecture:

1. Docker uses a client/server architecture.
2. "docker daemon" is server and responsible for build/run and distribute your docker containers.The user can not directly interact with the daemon, but instead through the Docker client.
3. "docker" is a command line client binary which talks to docker daemon. It accepts commands from the user and communicates back and forth with a Docker daemon
4. The Docker client and daemon communicate via sockets or through a RESTful API.
5. Docker daemon and client can be run on same machine or two different machines.

NOTE: Single binary, docker, acts both as a server daemon and client.

Docker images:

1. It is "build" component of docker. It can be considered as read-only templates for docker containers.
2. Images are built step-by-step using a series of instructions  like
   a. ADD file
   b. RUN command
   c. Open port
3. You can build new images, updates existing one or download images which people already has created.
4. You launch your containers from images.

Docker Registries:

1. It is "distribution" component of Docker platform.
2. You can have public/private registries.
3. Public docker registry is provided with the Docker Hub. It serves a huge collection of existing images for your use.
4. You can upload/download Docker images to/from these registries.

Docker Containers:

1. It is "run" or "execution" component of docker. 
2. Containers are created from Docker images. 
3. You can create multiple containers from single image.
4. Containers can be run, started, stopped, moved, and deleted.
5. Each container is an isolated and secure application platform.

Summary:

1. You can build a Docker images that hold your applications.
2. You can create a docker containers from those Docker images to actually run your applications.
3. You can share those Docker images via Docker Hub or your own registry.

How Docker images work?

1. Each docker image starts from a base image (Ubuntu, Fedora etc).
2. Further construction is done using simple, descriptive set of steps called instructions.
3. Instructions are stored in a file called DockerFile. Docker reads this file and execute instructions to construct and returns the final image.
4. Instruction include actions like ADD, COPY, RUN etc.
5. Each instruction creates a new layer in our image. 
6. Docker makes use of Union File System to create the final image.
7. That final image tells Docker what the container holds, what process to run when the container is launched, and a variety of other configuration data

How does container work?

1. The Docker image is read-only. When Docker runs a container from an image, it adds a read-write layer on top of the image (using a union file system as we saw earlier) in which your application can then run.