steveej/msc-thesis
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 4 Packages Wiki Activity Actions

thesis/context: more on VMs and containers

Browse source
This commit is contained in:
steveej 2016-11-02 19:29:04 +01:00
parent 414e01dc47
commit b76229840d
2 changed files with 107 additions and 63 deletions
Download patch file Download diff file Expand all files Collapse all files

18
src/docs/glossary.tex
View file

@ -2,13 +2,21 @@
\newglossaryentry{OS}{ \newglossaryentry{OS}{
name = Operating System, name = Operating System,
description = {
The software that manages the system's hardware ressources.
Other \glspl{app} can access the ressources only through the interface provided by the \gls{OS}.
},
}
\newglossaryentry{fs}{
name = filesystem,
description = { description = {
TODO TODO
}, },
} }
\newglossaryentry{virtualization}{ \newglossaryentry{virt}{
name = Virtualization, name = virtualization,
description = { description = {
TODO TODO
}, },
@ -51,8 +59,8 @@
plural=Linuces plural=Linuces
} }
\newglossaryentry{OBSD}{ \newglossaryentry{BSD}{
name = OpenBSD, name = BSD,
description = { description = {
TODO TODO
} }
@ -68,7 +76,7 @@
} }
\newglossaryentry{app}{ \newglossaryentry{app}{
name=Software Application, name=software-application,
description={ description={
TODO TODO
} }

148
src/docs/parts/context/context.tex
View file

@ -1,92 +1,128 @@
% // vim: set ft=tex: % // vim: set ft=tex:
\chapter{Overview} \chapter{Introduction}
This thesis is a scientific approach to analyze and solve the practical problems of packaging and deploying \glspl{app} in the context of \gls{sac} technology. This thesis is a scientific approach to analyze and solve the practical problems of packaging and deploying \glspl{app} in the context of \gls{sac} technology.
For a lack of an official definition and common understanding what this technology is, the term \gls{sac} is defined in this chapter as a reference for the rest of the thesis. For a lack of an official definition and common understanding what this technology is, the term \gls{sac} is defined in this chapter as a reference for the rest of the thesis.
The technology combines \gls{virtualization} techniques with new approaches to \gls{app} development and deployment. The technology combines \gls{virt} techniques with new approaches to \gls{app} development and deployment.
The two main drivers for this technology have been long standing problems in information technology; optimal utilization of hardware, and correct deployment of software to said hardware, both without sacrificing security. The two main drivers for this technology have been long standing problems in information technology; optimal utilization of hardware and correct deployment of software to said hardware, both without sacrificing security.
The optimal utilization of hardware is done by collocating and running multiple \glspl{app} running simultaneously on the same hardware. The optimal utilization of hardware is done by collocating and running multiple \glspl{app} simultaneously on the same hardware.
In order to increase security, these applications are separated by applying \gls{virtualization} techniques. In order to increase security, these applications are separated by applying \gls{virt} techniques. By developing \gls{virt} techniques into the \gls{OS} and thereby providing low-level security mechanisms, the foundation of \glspl{sac} was formed. In the next chapter an introduction to \gls{virt} is given, as it is important to understand this aspect of the \gls{sac} technology.
An introduction to \gls{virtualization} is given in this chapter, as it is important to understand this aspect of the \gls{sac} technology.
The correctness of software deployment is not measurable as a simple number, but involves many factors. The correctness of software deployment is not easily measurable by quantity like hardware utilization, but is of qualitative nature.
The \gls{sac} approach is to create self-contained bundles for \glspl{app} in form of \glspl{saci}. The \gls{sac} approach is to deploy self-contained bundles for \glspl{app} in form of \glspl{saci}. The creation of these \glspl{saci} is the main concern of this thesis and will be the subject to research and development in part \ref{part:research}. The underlying problems related to conventional \gls{app} deployment are covered in chapter \ref{chap:sdd}. State of the art attempts to solve these problems using \glspl{saci} are introduced and critically discussed in chapter \ref{chap:saci}, detailing the problem statement for this thesis.
As the main concern of this thesis, related problems and state of the art solutions are explained in this chapter.
In the following chapters it is explained which technologies of \gls{virtualization} and \gls{OS} security mechanisms that form the foundation of \glspl{sac}. \chapter{\Gls{virt}}
Also explained are the difficulties of correct software development and deployment in form of \glspl{saci}. \label{chap:virt}
These difficulties are the main subject of this thesis. Since the first \gls{VM} \gls{OS} \cite[p.~217-218]{Sarton1975} was created, \gls{virt} has been an important field in computer sciences, both in academic research and in the industry, and has been subject to continuous development, improvement, and adoption.
The very basic understanding of virtualizing a system or an application is to isolate and to control it.
\chapter{Virtualization} In greater detail it is the principle of controlling and monitoring the availability and the access to soft- and hardware resources for users, their applications or whole virtual systems running on top of existing systems.
Since the first \gls{VM} \gls{OS} \cite[p.~217-218]{Sarton1975} was created, \gls{virtualization} has been an important field in computer sciences, both in research and in the industry, and has been subject to continuous development, improvement, and adoption.
The basics of virtualization boil down to one principle.
It is the principle of controlling and monitoring the availability and the access to soft- and hardware resources for users, their applications and whole virtual systems running on top of existing systems.
Virtualization techniques can be grouped by two categories: \glspl{hypervisor} and \gls{osvirt}. Virtualization techniques can be grouped by two categories: \glspl{hypervisor} and \gls{osvirt}.
\section{\glspl{hypervisor}} \section{\glspl{hypervisor}}
The term \gls{hypervisor} is synonymous to the more self-explanatory terms control program \cite[p. 217]{Sarton1975} and \gls{VM} monitor. The term \gls{hypervisor} is synonymous to the more self-explanatory terms control program \cite[p. 217]{Sarton1975} and \gls{VM} Monitor.
The \gls{hypervisor} operates on a host machine and can control multiple \glspl{VM}. The \gls{hypervisor} operates on a host machine and can control multiple \glspl{VM}.
The principle is easy to understand, because one can simply picture one or many virtual computers running on a real computer. The principle is easy to understand, because one can simply picture one or many virtual computers running on a real computer.
\glspl{VM} are presented with a set of virtual hardware resources that don't necessarily exist in the presented form on the underlying hardware machine they are being executed on. \glspl{VM} are presented with a set of virtual hardware resources. These don't necessarily exist in the presented form on the underlying hardware machine.
\subsection{Running \glspl{OS}} \subsection{Guest \glspl{OS}}
In order to be able to boot the virtual hardware and run services, \glspl{VM} need an \gls{OS} to begin with. In order to be able to boot the virtual hardware and run \glspl{app}, \glspl{VM} need an \gls{OS} to run applications.
Specific to the environment and features of the \gls{hypervisor}, there are different storage formats for the file(s) that contain the \gls{OS} for the \gls{VM}. Specific to the environment and features of the \gls{hypervisor}, there are different storage formats for the file(s) that contain the \gls{OS} for the \gls{VM}.
More insight on this is given in section \ref{sect:vm-fs-isolation}.
In general, these files contain the \gls{OS} itself, as well as the installed applications in order to run the desired services within the \gls{VM}. In general, these files contain the \gls{OS} itself, as well as the installed applications in order to run the desired services within the \gls{VM}.
Compared to running such a service on the host machine directly, one obvious overhead is that this requires a said \gls{OS} to be installed and configured once upfront, and virtually booted for every execution of the service. \subsection{Flexible Guest \gls{OS} Choice}
Because each \gls{VM} on is necessary to store and run a separate \gls{OS}, they can run whatever the \gls{hypervisor} supports, and are not tied to run the same \gls{OS}, or even the same platform, as the host machine.
This allows to create heterogeneous scenarios like running an ARM \gls{VM} using \gls{BSD} on a x86 \gls{Linux} host, or vice versa.
\subsubsection{Support for multiple \glspl{OS}} \subsection{\Gls{fs} Storage Isolation}
Because each \gls{VM} on a host has a separate \gls{OS}, they can run whatever the \gls{hypervisor} supports, and are not tied to run the same \gls{OS}, or even the same platform, as the host machine. \label{sect:vm-fs-isolation}
This allows to create heterogeneous scenarios like running an ARM \gls{VM} using \gls{OBSD} on a x86 \gls{Linux} host, or vice versa. The \gls{OS} running inside a \gls{VM} is typically presented with a virtual disk drive. The guest \gls{OS} has to implement the driver for this virtual disk drive. The drive backed by a file on the host system, which can either be a file on a \gls{fs} or point to a block device.
By exclusively assigning one virtual disk drive per \gls{VM}, \glspl{VM} not access other \glspl{VM} data.
% TODO: think about if I want to show this or not. \subsection{\gls{Linux}'s Kernel Virtual Machine \gls{hypervisor}} As a result, the \gls{hypervisor} features full isolation between guest's \gls{fs} and also prevents them from accessing the host's files.
% it's not part of the topic.
% it's not necessary to understand containers
\subsection{Overhead In Application Virtualization} \subsection{Overhead In \Gls{app} \Gls{virt}}
In case that multiple \glspl{VM} are supposed to run the same application, e.g. with different configuration files, each of them will have a separate copy of the \gls{OS} and the application itself. \label{sect:virt-overhead-app-virt}
It's assumable that solely the applications running on top of the virtualized \gls{OS} is the required subject to virtualization. Compared to running services directly on the host machine, one obvious overhead is that an additional \gls{OS} is required to be installed and configured once upfront, and virtually booted to enable execution of applications within.
Therefore, it might not be necessary to run a separate virtualized \glspl{OS} just to for the sake of virtualizing the applications, especially if they are compatible with the same type of \gls{OS}. In case that multiple \glspl{VM} are supposed to run the same application, e.g. with different configuration, each of them will have a separate copy of the \gls{OS} and the application itself, differing only by configuration and runtime data.
This the main scenarios for the use of \gls{osvirt}, which is better suited because it doesn't require another full-fledged \gls{OS} to virtualize an application. For this thesis, the use-case is in which solely the applications running on top of the virtualized \gls{OS} is the required subject to gls{virt} is of highest interest. Further the focus lies on applications that can run on \gls{Linux}.
Instead, the applications run in the same \gls{OS} instance as separates processes, explained in the following section \ref{sect:sac-osvirt}.
Running a separate virtualized \glspl{OS} in the use-case of virtualizing \gls{Linux} applications involves considerable and unnecessary overhead.
In addition there are several performance aspects that are slowed down when running software inside \Glspl{VM}\cite{Felter2014}.
\section{\gls{osvirt}} \section{\gls{osvirt}}
\label{sect:sac-osvirt} \label{sect:sac-osvirt}
% TODO: explain generically what APC is and compare to Hypervisors Virtualizing \glspl{app}, as described in section \ref{sect:virt-overhead-app-virt}, is the primary use-case for \gls{osvirt}.
Compared to \gls{hypervisor}, \gls{osvirt} doesn't require another virtualized, full-fledged \gls{OS}, to virtualize an application.
Instead, the application's processes run in the same \gls{OS} in isolated and controlled virtual environments, called containers\cite{Reshetova2014}.
The technology of \gls{osvirt} has been under active development for about decade now\cite{Reshetova2014}, with the purpose of supporting virtualized applications rather than virtualizing whole machines. The technology of \gls{osvirt} has been under active development for about decade now\cite{Reshetova2014}, with the purpose of supporting virtualized applications rather than virtualizing whole machines.
The popularity of the technology has spiked with the release of one specific user-facing implementation named \gls{Docker}, which was originally exclusively available for the \gls{Linux}-platform. The popularity of the technology burgeoned with the release of one specific user-facing implementation named \gls{Docker}, which was originally exclusively available for the \gls{Linux}-platform.
Section \ref{virt-advent-sac} contains an overview of \gls{Docker}'s features that presumably helped to make it popular in a relatively short amount of time, compared to how long the underlying technology and similar tools have existed. Section \ref{sect:virt-advent-sac} contains an overview of \gls{Docker}'s features that presumably helped to make it very popular in a relatively short amount of time, compared to how long the underlying technology and similar tools have existed.
Beforehand, it is useful to learn the low-level mechanisms implemented in the \gls{Linux} kernel. To form a complete view it is useful to learn the leveraged low-level mechanisms implemented in the \gls{Linux} kernel.
\subsection{Process Isolation} \subsection{\gls{Linux} Process Separation and (Resource) Access Control}
\gls{osvirt} allows to virtualize applications on the \gls{OS} level instead of the machine level. \label{sect:linux-process-isolation}
Its features are implemented in the kernel of the \gls{OS}, providing a low-overhead isolation and resource-control for user-space processes, in short: virtualization. \gls{osvirt} on \gls{Linux} allows to virtualize applications on the process level instead of the machine level.
It's based on the \gls{OS} primitive of processes, providing a low-overhead isolation and resource-control for user-space processes, creating a form of \gls{virt}.
Every \gls{app} instance, virtualized or not, runs in the context of a separate process known to and managed by the very same host \gls{OS}.
With \gls{osvirt}, the control as well as the responsibility for the virtualized application belongs to the host \gls{OS}.
In contrast to running an application inside a \gls{VM}, where the host \gls{OS} doesn't know anything beyond the border of the process that runs the \gls{VM} and has no direct control over the application processes inside the \gls{VM}.
% TODO: refer to Linux Namespaces and Cgroups On \gls{Linux}, there are many different mechanisms to regulate a process's access or its view on resources of all kinds.
% TODO show chroot It is important to understand not all of these mechanisms contribute to security, and that \gls{osvirt} is more difficult to secure compared to \Glspl{VM}.
% TODO compare security features
% TODO compare performance \subsubsection{The chroot System Functionality}
The oldest one contributing one of the core functionalities to realize containers is the \textit{change root} functionality\cite{Reshetova2014}.
It allows a privileged process to change its effective root \gls{fs}, which can be any directory on the host.
Effectively every file accessed by path changing the root \gls{fs} will be prefixed with the new root path automatically by the \gls{OS}.
As an example, when a process changes its root \gls{fs} to \textit{/newroot} and then requests to open the file \textit{/file}, the application will transparently access \textit{/newroot/file}.
\subsection{\gls{app} Deployment} Note that \textit{chroot} has not been designed as a security feature, and therefore privileged \textit{chroot}'ed applications can easily access the host's filesystem if no countermeasures are taken.
% TODO compare productivity and deployment
\section{Its Advent With \glspl{sac}} \subsubsection{Namespaces}
\label{virt-advent-sac} % TODO explain namespaces in general
% TODO uts
% TODO pid
% TODO mount
% TODO net
% TODO ipc
% TODO cgroup
% TODO user
\subsubsection{Control Groups}
% TODO
\subsubsection{Security Modules}
% TODO explain what LSM is a framework
% TODO AppArmor
% TODO SELinux
\subsection{\Gls{fs} Storage Isolation And \Gls{app} Deployment}
With the help of the different \ref{sect:linux-process-isolation} techniques ... TODO
\subsection{\Gls{app} \Gls{virt}}
% TODO compare app productivity and deployment by example of chroot vs VM
\subsection{The Advent of \glspl{sac} with \gls{Docker}}
\label{sect:virt-advent-sac}
Even though the underlying technology \gls{osvirt} had been available for a relatively long time, \gls{Docker}\cite{Fink2014}, since its release in 2014\footnote{http://blog.docker.com/2014/06/its-here-docker-1-0}, has brought \glspl{sac} to the attention and hands of the masses in the \gls{OSS} community. Even though the underlying technology \gls{osvirt} had been available for a relatively long time, \gls{Docker}\cite{Fink2014}, since its release in 2014\footnote{http://blog.docker.com/2014/06/its-here-docker-1-0}, has brought \glspl{sac} to the attention and hands of the masses in the \gls{OSS} community.
From a psychological standpoint this is not surprising, as it has abstracted most complexities of the technology, adding ease of deployment, a platform for hosting the \gls{saci} in a Docker specific format, as well as a very convenient way for building the like using Dockerfiles(TODO reference). From a psychological standpoint this is not surprising, as it has abstracted most complexities of the technology, adding ease of deployment, a platform for hosting the \gls{saci} in a Docker specific format, as well as a very convenient way for building the like using Dockerfiles(TODO reference).
Its popularity has come to a point where the term \textit{Docker} is being used interchangeably with the \gls{sac} technology itself. Its popularity has come to a point that the term \textit{Docker} is being used interchangeably with the \gls{sac} technology itself.
% TODO: references for this claim % TODO: references for this claim
% TODO introduce \gls{LXC}, \gls{systemd-nspawn}, \gls{Docker}, and \gls{rkt}. % TODO introduce \gls{LXC}, \gls{systemd-nspawn}, \gls{Docker}, and \gls{rkt}.
The first part of this section analyzes the \gls{sacr} aspects of the implementations, while the second part demonstrates currently popular approaches to assemble \glspl{saci}. The first part of this section analyzes the \gls{sacr} aspects of the implementations, while the second part demonstrates currently popular approaches to assemble \glspl{saci}.
\section{Virtualization Summary}
% TODO: graphic to show the differences
\chapter{Software Development \& Deployment} \chapter{Software Development \& Deployment}
\label{chap:sdd}
\section{Bringing Together Developers \& Operators} \section{Bringing Together Developers \& Operators}
The technology that is currently available and combined to form \glspl{sac} reuses different patterns and techniques to solve a combination of problems that either software developers or system operators have faced separately. The technology that is currently available and combined to form \glspl{sac} reuses different patterns and techniques to solve a combination of problems that either software developers or system operators have faced separately.
@ -99,17 +135,17 @@ These problems are all related to software deployment and system operation and c
\item How do we guarantee that the application works on every target machine the same as on the developer machine? \item How do we guarantee that the application works on every target machine the same as on the developer machine?
\item How do we build multiple variants and/or versions of an application? \item How do we build multiple variants and/or versions of an application?
\item How do we install, configure and run them simultaneously on the same target machine without exhibiting conflicts or sacrifice a single application's security? \item How do we install, configure and run multiple instances simultaneously on the same target machine without exhibiting version conflicts and without sacrificing application security?
\item How do we verify that an application runs on the target system has not been altered maliciously at one point in the deployment chain? \item How do we verify that an application runs on the target system has not been altered maliciously at one point in the deployment chain?
\end{enumerate} \end{enumerate}
Only a subset of these problems and attempted solutions will be subject to research for this thesis. Only a subset of these problems and attempted solutions will be subject to research for this thesis.
Questions 2 - 4 are in the research scope of this thesis, while the concern of question 5 is declared optional and is not strictly required for the completion of this project. Questions 2 - 4 are in the scope of this thesis, while the concern of question 5 is declared as an optional question.
These questions are very important to the ideology of \glspl{sac}, and they have their origin in the conventional methods of software deployment. All of these questions are very important to the ideology of \glspl{sac}, and they have their origin in the conventional methods of software deployment.
More thorough examination of the questions and their presented problems is found in section \ref{sect:sd-challenges} of this chapter. More thorough examination of the questions and their presented problems is found in section \ref{sect:sd-challenges} of this chapter.
\section{Challenges of Software Development \& Deployment} \section{Challenges of Software Development \& Deployment}
\label{sect:sd-challenges} \label{sect:sd-challenges}
Software is typically developed on a workstation or laptop, brought to a format that is understood by the target machine to which it must be is transfered, and where it is configured and finally executed to serve its purpose. Software is typically developed on a workstation or laptop, translated to a format that is understood by the target machine to which it must be transfered, and where it is configured and finally executed to serve its purpose.
In order to be executable on the target machine, the software needs to be translated into the target-platform specific format before it can be executed on to the target system. In order to be executable on the target machine, the software needs to be translated into the target-platform specific format before it can be executed on to the target system.
If the software is changed and updated, the cycle has to be repeated. If the software is changed and updated, the cycle has to be repeated.
@ -121,9 +157,10 @@ The binary files are then made available as software packages that can be downlo
Typically and ideally, this is done with the help of a software package manager, which itself is a software that is included in most modern operating systems. Typically and ideally, this is done with the help of a software package manager, which itself is a software that is included in most modern operating systems.
The location where the files of the package are installed on the target machine depends on how it is configured at packaging time, but in most popular software package approaches this location is agnostic of software application version changes. The location where the files of the package are installed on the target machine depends on how it is configured at packaging time, but in most popular software package approaches this location is agnostic of software application version changes.
Another challenge is to be able to verify that software hasn't been altered to differ from its intentional behavior at any point of the deployment process. Another challenge is to be able to verify that the software hasn't been altered, either accidentally or maliciously, and differ from its intentional behavior at any point of the deployment process.
\chapter{\glspl{saci}} \chapter{\glspl{saci}}
\label{chap:saci}
\section{No truly declarative method to create \gls{saci}} \section{No truly declarative method to create \gls{saci}}
@ -132,7 +169,6 @@ Another challenge is to be able to verify that software hasn't been altered to d
\section{Customization} \section{Customization}
\chapter{Scope} \chapter{Scope}
%TODO: explain scope of OSS systems running a \gls{Linux} based \gls{OS}.
\section{Goals} \section{Goals}
The primary concern is to find a viable method for declaring and assembling \glspl{saci} deterministically. The primary concern is to find a viable method for declaring and assembling \glspl{saci} deterministically.