Karl Fischer

Measuring Traffic with iptables

Sat, 30 Mar 2019 10:30:00 +0000

I recently read about a neat method of measuring traffic with iptables on linux hosts, which is nice for pentesting or infrastructure debugging. This is a rather short post describing that approach.

Benefits of using iptables for traffic measurements

One nice thing about iptables is that it is very likely to be present on any linux server/client you run, so you dont need to install any extra packages.
For infrastructure debugging/planning purposes you might need to know quickly how much traffic flows between 2 specific hosts/ports/… . Maybe you do not have monitoring in place yet or the monitoring is not fine grained enough (e.g., aggregating ALL packets on host interfaces).
In pentesting this is a fast and easy method to measure how much traffic/attention your operation produces.

Hands-on Example

Let us assume we want to quickly measure the traffic between the current machine and a remote host.

Initially, we have empty iptables chains:

~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Of course some servers might already have rules attached and we do not want to mess with them. We create a new chain dedicated for our measurements:

~# iptables -N TARGET
~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain TARGET (0 references)
target     prot opt source               destination

Now we attach measurement rules. We want to measure traffic between the current server and the machine 10.11.1.227 in our local network:

~# iptables -A TARGET -d 10.11.1.227
~# iptables -A TARGET -s 10.11.1.227
~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain TARGET (0 references)
target     prot opt source               destination         
           all  --  anywhere             10.11.1.227         
           all  --  10.11.1.227          anywhere

Next, we attach the new chain to input and output chains:

~# iptables -A INPUT -j TARGET
~# iptables -A OUTPUT -j TARGET
~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
TARGET     all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
TARGET     all  --  anywhere             anywhere            

Chain TARGET (2 references)
target     prot opt source               destination         
           all  --  anywhere             10.11.1.227         
           all  --  10.11.1.227          anywhere

Now everything is in place. We can finally zero the packet and byte counters, trigger a command to produce traffic and verify that the traffic was counted:

~# iptables -Z
~# nmap -nA 10.11.1.227
..snip..
~# iptables -L TARGET -n -v -x
Chain TARGET (2 references)
    pkts      bytes target     prot opt in     out     source               destination         
    4663   353156            all  --  *      *       0.0.0.0/0            10.11.1.227         
    3061   214887            all  --  *      *       10.11.1.227          0.0.0.0/0

We clearly see how many packets and bytes were transferred between the current server and 10.11.1.227.

NOTE: Using nmap for port scanning is illegal in most countries. Use it only on networks that you own or for which you have explicit scanning permissions from the owner.

After we are done we can cleanup:

~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
TARGET     all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
TARGET     all  --  anywhere             anywhere            

Chain TARGET (2 references)
target     prot opt source               destination         
           all  --  anywhere             10.11.1.227         
           all  --  10.11.1.227          anywhere            
~# iptables -D INPUT 1
~# iptables -D OUTPUT 1
~# iptables -F TARGET 
~# iptables -X TARGET 
~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Finally, here is a short script for setting up the chains and rules to measure traffic between 2 hosts:

measureTraffic.sh

Jenkins-as-Code Part III | JobDSL

Sat, 09 Feb 2019 10:30:00 +0000

The 3rd part of the Jenkins-as-Code series focuses on the automated creation of job interfaces. That could be achieved with pure groovy scripting as used in the previous part. However, the JobDSL plugin offers a more convenient and clean way, so we will focus on that.

There are already quite a lot of tutorials and code snippets about the JobDSL plugin all over the web, so this will be a rather short post. I post this merely for the sake of completion as without JobDSL, Jenkins is not fully as-code. The complete code can be found in this demo repo.

JobDSL Introduction

JobDSL is a DSL language on top of groovy. It offers an easy way to describe job interfaces as-code. It is very common to have a job interface seeding pipeline, which pulls the jobDSL code from a repository and provisions the job interfaces. Jenkins jobDSL is very easy to understand and write. It is easy to pickup as it comes with a good documentation. However, jobDSL can only provision features of plugins that are already installed in your jenkins instance. So the online documentation might show you features which are not available to your jenkins instance because of missing plugins. You can find all available features documented on your jenkins instance at ${JENKINS_BASE_URL}/plugin/job-dsl/api-viewer/index.html.

Further, there is an online playground for jobDSL scripts. It translates your scripts to the final xml of the job interface, so it is ideal to quickly validate your scripts during jobDSL development.

Multibranch Build Pipelines

After all that praise, now let’s have a look at some code. In the following we create a set of Multibranch Pipelines for project repositories on Github:

multibranchJobs.groovy:

We assume that Jenkins is already hooked up with Github, which is a requirement for a Github Multibranch pipeline to function.

Hooking up your Jenkins instance with Github and setting up credentials for CI users can be done fully as-code as well, as we have seen in the previous part of this series.

A seed pipeline to trigger the jobDSL could look like that:

seedPipeline.groovy:

This is a very simplified seeding pipeline. In the previous part of this series we covered a more mature configuration and seeding pipeline.

Conclusion

This short post gave a quick overview of the JobDSL plugin and how to use it. We created a simple DSL for multibranch pipeline job interfaces of different projects hosted on Github.

Jenkins-as-Code Part II | Configuration

Sat, 12 Jan 2019 14:30:00 +0000

This is the 2nd part of the Jenkins-as-Code series. In this part we will focus on configuring Jenkins through code. The goal is to avoid manual configuration in the UI and instead leverage the configuration as code plugin and configuration scripts in a central Github repository which are executed by a Jenkins pipeline.

In the previous part of this series we created a Jenkins docker image with a baked in configuration pipeline.
We now focus on the components of that pipeline, i.e., running the CasC plugin, executing configuration scripts and seeding job interfaces. The complete code can be found in this demo repo.

Directory Layout

Lets quickly remind ourselves about the layout of our shared library repository.

shared-library
|-- resources
|   |-- config
|   |   |-- configuration-as-code-plugin
|   |   |   `-- jenkins.yaml
|   |   `-- groovy
|   |       |-- timezone.groovy
|   |       |-- triggerConfigurationAsCodePlugin.groovy
|   |       `-- userPublicKeys.groovy
|   |-- init
|   |   `-- ConfigurationAndSeedingPipeline.groovy
|   `-- jobDSL
`-- vars

The majority of Jenkins can be configured using CasC, but there are a few things which do still require groovy system configuration scripts, like setting the timezone.

Configuration as Code Plugin

Configuring Jenkins with groovy scripts can be tedious. You have to go through the plugin’s code, find the constructor and properly use it in groovy code in order to configure it.

However, in 2018 a new approach arised in the form of a plugin. It is called the configuration-as-code plugin (CasC). The goal of this plugin is to describe your jenkins configuration in simple .yaml files.

The following CasC file configures Security, GitHub, OAuth, Slack, Themes, Agents and Authorizations:

jenkins.yaml:

Secrets are described as ${secret}. As of writing this article, secrets in CasC can be loaded from environment variables, files or HashiCorp Vault paths. In this post’s demo repo we use files as secret sources. The path to the secret files is pre-baked in the Dockerfile as ENV SECRETS="/var/jenkins_home/". A secret like ${jenkins-ssh-keys/ssh-agent-access-key} is then loaded from CasC at /var/jenkins_home/jenkins-ssh-keys/ssh-agent-access-key. However, I am a big fan of HashiCorp’s Vault. Whenever possible, I use it as a secret source, but that is out of scope of this blog post. To setup CasC with Vault, consult their README.

Configuration Scripts

CasC does the heavy lifting of our configuration work. However, there are some aspects that still need some groovy scripting magic.

Timezone

Obviously a proper timezone setting is nice in order to not be confused by the build times.

timezone.groovy:

User Public Keys

We can add public keys for users.

user-public-keys.groovy:

Adding a public key to a user is useful if you must interact with jenkins via jenkins-cli. In my experience Jenkins CLI works best when used with the -ssh option.

Others

Over the time I gathered more configuration scripts, e.g., GitHub OAuth or Slack configuration. Thanks to CasC I do not need them anylonger. However, I keep the collection in a gist. Maybe it will still be of help to someone out there.

Summary

We now have a fully configured Jenkins. The configuration is part of our shared library. The configuration is executed via a single pre-baked configuration and seeding pipeline which uses our shared library. We can change a configuration by pushing the change to the shared library and running the pre-baked configuration and seeding pipeline in Jenkins.

In the next part of this series we will have a quick look at Jenkins JobDSL plugin for job interfaces as-code.

Jenkins-as-Code Part I | Initial Setup

Sun, 06 Jan 2019 14:45:00 +0000

This is the beginning of a series about full Jenkins automation. I am calling that approach jenkins-as-code. The goal is to configure every aspect of Jenkins and its pipelines from a central git repository. We will leverage groovy scripting, jobDSL and shared libraries to not only codify the build/deploy pipelines (pipeline-as-code), but to also bootstrap and configure Jenkins from scratch (e.g., credentials, authorization, theme and job setup).

The following demo shows what we have at the end of this series:

Every aspect of what is shown here can be found in full context in the jenkins-as-code Github repository.

It is usually a good idea to have as much as possible setup in an *-as-code approach, because it offers:

clear definitions and automation, i.e., clear state definition and reproducibility
rollouts from central servers (offers audit logs)
common software development workflows, i.e., testing, reviewing and rollouts are easier to manage

Since Jenkins 2.x we can use pipeline-as-code approaches like Travis or CircleCI do. However, the overall (plugin) configuration and setup of job interfaces is not covered by pipeline-as-code. For a complete jenkins-as-code approach we also want our configuration and job interfaces treated as-code and be able to rollout any changes dynamically through a jenkins job interface without the need to restart Jenkins.

In the first part of this series we will discuss the basics:

What are the layers in jenkins-as-code?
What are the relations between those layers?

Next, we will create a Jenkins docker image packaged with a configuration/seeding job interface and pipeline definition, which uses a central shared library from GitHub to rollout configuration and seeding changes. Finally, we will have a closer look at contents of that central shared library.

Layers and Terminology

In order to avoid confusion it is very important to define the different layers and terms which will be used throughout this series.

Jenkins-as-Code

Jenkins-as-code describes an approach to codify and automate every layer of Jenkins. In order to achieve its goal the approach leverages jobDSL, configuration and pipeline definition scripts.

Job Interface

A job interface describes the job in the Jenkins UI. Deep down a job interface is xml on the Jenkins master instance, usually found at ${JENKINS_HOME}/jobs. We use a job interface to declare input parameters and to trigger the job. Preferably, a job interface is created using the Jenkins jobDSL plugin. When a job is triggered it loads the pipeline definition.

Pipeline Definition

A pipeline definition is a groovy script which is loaded by a job interface when a job is triggered. Those groovy scripts are also known as pipeline-as-code - an approach also known from cloud CIs like Travis or CircleCI.

Configuration Script

A configuration script is a system groovy script, which interacts with the Jenkins eco-system to configure Jenkins and its plugins.

CasC

[C]onfiguration [as] [C]ode refers to the Configuration-as-Code Plugin. Rather than writing configuration scripts, this plugin allows to define your configuration in a simple .yaml syntax.

JobDSL Script

A jobDSL script is a groovy script which programmatically describes job interfaces. It is read by the Jenkins jobDSL plugin to create the job interfaces.

Seeding

Seeding refers to creating all job interfaces from a single pipeline (the seeding job). That pipeline executes the jobDSL scripts in order to create all job interfaces.

Layers and their Interaction

Layers in jenkins-as-code:

Interaction between the layers:

Jenkins Docker Image

Now that we have covered some basics, let us begin with the hands-on work and create a docker image for our Jenkins.

Directory Structure

We will use the following directory structure:

docker/
|-- Dockerfile
|-- dsl
|   `-- ConfigurationAndSeedingPipelineDSL.groovy
|-- init.groovy.d
|   `-- init.groovy
`-- plugins.txt

The following sub-sections show the files in detail.

Dockerfile and Plugins

We want to build an image which automatically bootstraps our Jenkins with an initial job interface for configuration and seeding.

Dockerfile:

NOTE: That Dockerfile also installs HashiCorp’s Vault. While it is not needed in this part of the series we install it anyways for later use.

Also, we must ensure that some plugins are installed.

plugins.txt:

Most of those plugins are not necessary at this step of the series, but might come in handy later on.

NOTE: Plugin versions are not pinned here as this is a demo. However, I recommend to pin the plugin versions inside the plugins.txt in order to have deterministic docker image builds.

Initial Pipeline Interface

The seed and configuration job interface is created through a jobDSL script.

ConfigurationAndSeedingPipelineDSL.groovy:

init.groovy.d for initial setup

init.groovy.d is a directory in the jenkins home directory, which contains configuration scripts. Configuration scripts inside that directory are executed in alphabetical order at Jenkins boot time. This is ideal for setting up seeding and configuration job interfaces.

init.groovy:

This script first adds a SSH private deploy key to Jenkins with access permissions to the shared library repository. Further, the script creates a configuration and seed job from a jobDSL script. That job will be used to pull the shared library from git and rollout its configuration and jobDSL definitions.

NOTE: The script assumes that the SSH key is already created and mounted into the docker container. The corresponding public key must be uploaded to GitHub as a deploy key for the shared library repository in order for Jenkins to pull it.

When you build this docker image and start the container, init.groovy.d will add your (mounted) SSH deploy key and create the configuration and seeding job interface. When this interface is triggered, it will load the central shared library and run our pipeline defintions.

Shared Library Repository

Lets have a look now at how our shared jenkins library looks like.

Directory Structure

We will use the following directory structure for our shared library:

shared-library
|-- resources
|   |-- config
|   |   |-- configuration-as-code-plugin
|   |   |   `-- jenkins.yaml
|   |   `-- groovy
|   |       |-- timezone.groovy
|   |       |-- triggerConfigurationAsCodePlugin.groovy
|   |       `-- userPublicKeys.groovy
|   |-- init
|   |   `-- ConfigurationAndSeedingPipeline.groovy
|   `-- jobDSL
`-- vars

The vars directory will contain our job pipeline (build/deploy). Those pipelines will be called from a Jenkinsfile inside one of our projects, but we will focus on that in a later part of this series and can leave it empty. Also the resources/jobDSL directory which is intended for job definitions will be left out for now. We will come back to in a later part of this series. For now we focus on the resources/init and resources/config directories. resources/init holds initial configuration and seeding pipelines which leverage scripts in resources/config and resources/jobDSL to configure and seed jobs.

In addition, we need a configuration and seeding pipeline script in the shared library repository.

ConfigurationAndSeedingPipeline.groovy:

The scripts loaded and executed in this job definition, e.g., triggerConfigurationAsCodePlugin.groovy or timezone.groovy are discussed in the next part of this jenkins-as-code series.

Summary

We now have a pipeline definition for configuration and seeding. That pipeline defintion is triggered by an initial job interface for configuration and seeding. The initial job interface is bootstrapped by our docker image (init.groovy.d).

The next part of the jenkins-as-code series will focus on the configuration scripts, e.g., OAuth authentication, theming, slack and credentials.

My Consul Service ID hunt

Sat, 09 Sep 2017 09:21:22 +0000

This is a rather short post about a recent annoying issue I stumbled across. As the title suggests I wanted to retrieve the ID of a service from a consul node. Easy as this may sound, the documentation is not very clear about what the service ID really looks like or how I can find it. (FYI: At the time of writing I was running consul 0.8.1 and vault 0.7.0)

What actually happened: I accidentally registered my vault server directly with the consul server nodes, but ideally it should be connected to a local consul agent instead. After realizing my mistake, I properly hooked up the vault node with its local consul agent. Besides being registered now on the correct node, the vault service was still also registered on the consul server node and marked as failing there .. On the first view this looks like a simple problem to solve .. Consul offers a HTTP API which you can also use to deregister services on nodes manually. Straight forward - or so it seems..

According to the documentation and assuming I can reach my consul node via localhost I just need to run:

curl -v -X PUT http://localhost:8500/v1/agent/service/deregister/:service_id

But wait - what’s my :service_id??? I figured, that my vault’s :service_id might as well be the name of the service itself: vault. It didn’t work:

$ curl -sX PUT http://localhost:8500/v1/agent/service/deregister/vault
Unknown service "vault"

Seems vault is really just a service name and not a proper service id. My failing vault service is still showing up inside the consul catalog ..

After some reading, I query the service catalog API to get some more meta information:

curl -sX PUT http://localhost:8500/v1/catalog/service/:service_name

The result of this query looks like that:

$ curl -sX PUT http://localhost:8500/v1/catalog/service/vault | jq
[
  {
    "ID": "<uuid>",
    "Node": "kernel-terra-vault-0",
    "Address": "10.5.1.200",
    "TaggedAddresses": {
      "lan": "10.5.1.200",
      "wan": "10.5.1.200"
    },
    "NodeMeta": {},
    "ServiceID": "vault:10.5.1.200:8200",
    "ServiceName": "vault",
    "ServiceTags": [
      "active"
    ],
    "ServiceAddress": "10.5.1.200",
    "ServicePort": 8200,
    "ServiceEnableTagOverride": false,
    "CreateIndex": <some index>,
    "ModifyIndex": <some index>
  },
  {
    "ID": "<uuid>",
    "Node": "kernel-terra-vault-1",
    "Address": "10.5.1.201",
    "TaggedAddresses": {
      "lan": "10.5.1.201",
      "wan": "10.5.1.201"
    },
    "NodeMeta": {},
    "ServiceID": "vault:10.5.1.201:8200",
    "ServiceName": "vault",
    "ServiceTags": [
      "standby"
    ],
    "ServiceAddress": "10.5.1.201",
    "ServicePort": 8200,
    "ServiceEnableTagOverride": false,
    "CreateIndex": <some index>,
    "ModifyIndex": <some index>
  },
  {
    "ID": "<uuid>",
    "Node": "kernel-terra-vault-1",
    "Address": "10.5.1.100",
    "TaggedAddresses": {
      "lan": "10.5.1.100",
      "wan": "10.5.1.100"
    },
    "NodeMeta": {},
    "ServiceID": "vault:10.5.1.100:8200",
    "ServiceName": "vault",
    "ServiceTags": [
      "standby"
    ],
    "ServiceAddress": "10.5.1.100",
    "ServicePort": 8200,
    "ServiceEnableTagOverride": false,
    "CreateIndex": <some index>,
    "ModifyIndex": <some index>
  }
]

Very interesting! It looks like :service_id is actually a combination of the service name, the node’s ip and the service port! This means that for one service across the whole cluster every offering node has a unique :service_id for that service.

In my case, 10.5.1.100 is the consul server node on which I want to deregister vault. I run:

$ curl -sX PUT http://localhost:8500/v1/agent/service/deregister/vault:10.5.1.100:8200

It worked! The faulty vault service got successfully deregistered on server node 10.5.1.100.

Lesson learned: :service_id != :service_name

Note: You should do the mentioned queries against a consul agent with the necessary privileges in case you activated the consul ACL. In any case it should work if you do the query against the agent on which the service is running that you want to deregister, as usually that node must have permissions to register/deregister the service in the first place.

A Docker Build Pipeline as Code with Jenkins

Mon, 21 Nov 2016 19:25:09 +0000

Since quite some time now we are no longer satisfied with only having our App Layer as code in repositories, but we also want every other component of our system codified. After Infrastructure as Code (e.g., Terraform) and Configuration as Code a.k.a. Continuous Configuration Automation (e.g., Ansible and SaltStack) approaches, we also want our automation itself codified: Pipeline as Code! Some of the big advantages of managing system components in code are:

It is easier for teams to work on the same complex components if they are managed in code repositories
Each aspect of your system can be versionized
Codified components can be easier used for further automation

External CI providers such as Travis or Circle CI made the Pipeline as Code principle very popular - especially amongst OSS Projects. Nevertheless, external CI providers may not support all of your Pipeline needs and even more important: A Pipeline may contain information you do not want to share with external providers. Personally, I pretty much trust the guys from Travis and for OSS projects they are way to go (not saying no to free checks and badges!), but if they get compromised the Pipeline data is exposed. Beginning with version 2, Jenkins finally supports the Pipeline as Code approach with the Jenkinsfile, which brings our Pipeline back into our own hands. In this post I want to outline how to build a simple codified Pipeline in Jenkins to containerize a Java project with Docker. I will solely focus on the Pipeline code. No Maven/Java code examples will be given here.

Setup Assumptions

I am assuming that you have a Maven Project which builds target/project.war via mvn clean package -U.

I also assume that you have a running Jenkins 2.x with access to a Docker Daemon (locally or remote). In case you are running Jenkins inside a container, you have to give it access to a Docker Daemon socket, either by sharing the docker hosts socket with the container or by using a remote server.

Jenkins must have read access to your code repository (which can easily be established using deploy keys).

Jenkins must have Maven configured as a global tool referenced as M3.

Further, we need the following Jenkins plugins installed:

The Jenkins Docker Pipeline

For our Java project, let us build a pipeline which handles the following steps:

Checkout a branch from a SCM
Set a branch SNAPSHOT version inside the pom file
Build a .war package for the checked out branch
Build a docker image containing this .war package
Send notifications to a Slack channel

To get started, we first create a build.groovy file inside our Java code repository, ideally on a new branch, which we call jenkins.

NOTE: Some Plugins, such as the Multibranch Pipeline Plugin require that the pipeline code is in a single file called Jenkinsfile in the repositorie’s root.

The reason why we put the build.groovy on a separate branch is to decouple the app code from the pipeline code. Imagine you want to build a feature branch, but something in your pipeline changed. You would need to push a new commit with that pipeline change on the feature branch. This is especially bad, if you want to build a specific commit, e.g., a tag, since that would require you to re-tag. On the other hand this approach adds more complexity to your project, as an extra branch for pipelines may not be as intuitive as keeping the pipeline defintions with the rest of your code in the same branch. A better approach would be to use the Multibranch Pipeline Plugin, also because it is a requirement to get the most out of the new Jenkins UI Blue Ocean. However, in this post we are satisfied with a separate branch. The repositorie’s dir structure may now look something like this:

project/
├── .jenkins
│   └── build.groovy
├── pom.xml
└── src

Initially, the build.groovy script may contain the following for testing purposes:

Now, we create a Jenkins Pipeline job, which checks out the jenkins branch of our project and looks for the pipeline file .jenkins/build.groovy. The configuration may look something like this:

When we run this job, it should print The pipeline started in the job’s console output.

In a next step, we want to build a SNAPSHOT package of a branch. The branch is given as an input variable named BRANCH_NAME. It can be given to the job as a simple String input parameter or by using the Git Parameter Plugin and is referenced inside the pipeline script as env.BRANCH_NAME. We want our branch SNAPSHOTs to have the following version naming pattern: <version>-<branchName>-SNAPSHOT, where <version> is taken from the checked out pom.xml and branchName is given as an input parameter env.BRANCH_NAME.

Let’s extend our pipeline script to checkout and build the specified branch:

This pipeline checks out the specified branch, determines and sets the proper branch snapshot version inside the pom.xml and finally builds the .war file. Next, we move on to packaging our .war inside a Tomcat docker image. We create a docker build context in our project’s jenkins branch, which will give us the following directory structure:

project/
├── .docker
│   └── build
│       ├── docker-entrypoint.sh
│       └── Dockerfile
├── .jenkins
│   └── build.groovy
├── pom.xml
└── src

The Dockerfile and docker-entrypoint.sh may contain the following:

IMPORTANT: By default the resulting container will run as root. This should be avoided if possible, since the root user inside the container has the same uuid as the host’s root user, which is potentially dangerous in case the container gets compromised.

The docker-entrypoint.sh allows our container to be configured via environment variables, making it more flexible to run in different environments. Further, we adjust our build.groovy script to build and push a functional docker image:

Note, that we also adjusted the Checkout stage, to copy the docker build context to another location before we switch branches (the build context might not be available on the new branch). Also, this pipeline does not contain important aspects such as testing or uploading of the .war file to a java package manager like Artifactory.

withDockerServer and withDockerRegistry are built-in wrappers by Jenkins’ docker-plugin and allow us to use a remote host to build the docker image (very handy when running the Jenkins master inside a container itself) and to define a custom private docker registry (most likely you do not store your companie’s docker images on dockerhub).

Finally, let us add some visibility to our build by sending the build status to a slack channel. We slightly modify our build.groovy:

That’s it! We now have a Pipeline which handles our Docker image builds for specific branches and sends the results to a Slack channel. In a future post I will show how to share Groovy libraries between Jenkins jobs to apply the DRY principle to your Pipeline code.

Ansible role structuring

Thu, 02 Jun 2016 15:47:09 +0000

In this short post I want to talk about structuring Ansible roles in a way that OS specific tasks and variables are cleanly separated from the other common logic of the role.

My main sources of inspiration for role structuring are Ansible Galaxy and the DebObs repository and I would like to share my findings here.

A simple ssh role

Let’s first quickly discuss the directory structure of an Ansible role. In general, we have a tasks directory, which contains each task that has to be executed for that role. Further, we got the handlers directory, which contains service handlers which can be called from any task. We also have a vars directory, which as the name already can contain variables. Jinja2 templates and static files are being stored inside the templates and files directory respectively. Another common directory is the defaults directory, which could be used for OS independant variables.

ssh
├── defaults
├── files
├── handlers
├── tasks
├── templates
└── vars

In each directory, except files and templates, the file main.yml gets included automatically when you use the role.

Now, to write our ssh role for Ubuntu 14.04 we would need a template for the sshd configuration.

templates/sshd_config.j2

Next, we need a handler for the sshd service.

handlers/main.yml

Further, we need tasks which ensure that sshd is installed and the ssh config template gets rendered and deployed.

tasks/main.yml

Finally, we could define some common variables.

defaults/main.yml

A more mature role structure

While the above approach does work for Ubuntu 14.04, we still would like to easily extend our role for any other OS too. One easy way to do so, is by separating OS specific tasks and variables in different OS dedicated files. Ansible helps us to automatically identify the currently used distribution with {{ ansible_distribution }} and {{ ansible_distribution_version }}, so we just have to name the OS dedicated yml files accordingly and include them in our main.yml files.

For our ssh role, the dir tree would then look something like that:

ssh
├── defaults
│   └── main.yml
├── handlers
│   ├── main.yml
│   └── Ubuntu14.04.yml
├── tasks
│   ├── main.yml
│   └── Ubuntu14.04.yml
├── templates
│   └── Ubuntu14_04.sshd_config.j2
└── vars
    └── Ubuntu14.04.yml

Here are the gists of the files.

defaults/main.yml

handlers/main.yml

handlers/Ubuntu14.04.yml

tasks/main.yml

tasks/Ubuntu14.04.yml

vars/Ubuntu14.04.yml

In that way we separated OS specific variables in dedicated files inside the vars directory. Also OS specific tasks and handlers can now easily be separated. Ansible detects the OS it operates on and includes the dedicated files.

Testing ansible playbooks with ansible-vault encrypted data using Travis CI

Sat, 02 Apr 2016 16:03:23 +0000

In this post I want to talk about using ansible-vault to encrypt secret variables and templates in your ansible roles. Further, I want to have a look at how to test those playbooks and encrypted files using Travis CI.

The combination of ansible-vault and Travis CI seems a little odd on first sight and I must admit I’ve had trouble finding a proper way of handling private data in a public repository with an external and also public CI environment. Here is some little background information why I still chose such an approach: My use case is an open source project which uses ansible provisioning for its infrastructure. One sweet thing about ansible is that it is highly supported by tools such as Packer and Vagrant (HashiCorp in general does some really awesome stuff - I am also already curious to play around with Otto and Vault (not to be confused with ansible-vault)). I use Vagrant locally to develop my ansible roles and Packer can be used to create images (also docker containers are possible) with these roles/playbooks which basically enables me to port my ansible described architecture to any kind of machine image. I find it tempting to also share infrastructure code, such as ansible scripts publicly, without risking security issues, but of course parts of the infrastructure such as keys, passwords etc. have to be kept secret. Further, when developing open source I can use services such as Travis CI to test my playbooks and roles for free, so I could test the whole setup instead of just single role rollouts. In the end of this post I will talk a little more about some security concerns. I think keeping up such an open approach in a secure way requires more maintenance on my side, but it is also an interesting challenge!

Ansible - A simple example playbook

Lets first create a simple ssh role and playbook. In the next sections we will then step by step encrypt sensitive parts of that role. First, we create a template for the sshd_config file.

roles/ssh/templates/sshd_config.j2

This sshd_config handles access permissions based on the user groups. The group list all_groups, which contains all groups with ssh permissions must be visible for this template to render. Next, we define the task for the role. Basically all we need is to place the sshd_config file and restart the ssh daemon.

roles/ssh/tasks/main.yml

Now we need to define our default variables.

roles/ssh/defaults/main.yml

Please note, that the sshd_groups list could also be defined in a users/groups role, but to keep it simple we just define it for now as a defaults variable in our ssh role. We also need a service handler to restart our service.

roles/ssh/handlers/main.yml

Next, we need a playbook which uses our role.

app.yml

This playbook rolls out the ssh role on all hosts in our inventory with the app tag. The no_log is set to avoid the logging of secret data in Travis later. Of course, while actively working on a playbook/role it might be easier for debugging to turn the logging on again by setting no_log: no, so inside the Travis build we just have to hand over an extra argument to disable the logging --extra-vars='{\"disable_log\": \"yes\"}'. Finally, we need an inventory file for development.

inventories/dev

Note, that the app host needs to be resolvable in order for this to work. You could add it to your /etc/hosts.

We could now run our playbook via:


ansible-playbook -i inventories/dev app.yml

We now have a simple and fully functional playbook. In the next sections I want to encrypt the secrets in our playbook and run tests via Travis CI.

Encrypting secrets with ansible-vault

Now we discuss how we can encrypt secrets that should not be shared with everyone inside your company. First, we will have a look at how to encrypt and use secret variables. Second, we will look at secret config files / templates.

Secret Variables

In Ansible we can use a feature called ansible-vault in order to encrypt our files which contain secrets. Simply use ansible-vault encrypt <file> in order to encrypt an existing file. You can also edit encrypted files (without decrypting them first on the disk) with ansible-vault edit <file> or create a new encrypted file via ansible-vault create <file>. Please consult the documentation for more commands of ansible-vault.

By storing the vault password in a file we can use our playbooks without decrypting the encrypted files directly on the disk. As an example we could use the following command to rollout a playbook with vault encrypted files:


ansible-playbook -i inventories/travis my-playbook.yml --vault-password-file vault_pass_file

Ansible will then decrypt the files in memory and rollout the playbook.

Remember to add the vault_pass_file to your .gitignore, so you do not add the password to your repository by accident.

For our playbook example from the previous section, we could encrypt our default variables, to keep our port and group access settings a secret. First, we generate a complex vault password and store it in vault_pass_file. Then, we run:


ANSIBLE_VAULT_PASSWORD_FILE=vault_pass_file ansible-vault encrypt roles/ssh/defaults/main.yml

We could then rollout the playbook via:


ansible-playbook -i inventories/dev app.yml --vault-password-file vault_pass_file

Secret Templates

In the first place ansible-vault is designed to encrypt files containing secret variables, such as files in the group_vars or host_vars directories, but theoretically we can also use it to encrypt templates. If I want to keep config files such as sshd_config or nginx config files secret, then I can use ansible-vault to encrypt their templates. The problem with encrypting templates is that ansible assumes that templates or files which are copied to the server are already unencrypted. Thus, we cannot use on the fly in memory decryption by simply specifying the --vault-password-file flag, but we need to decrypt the templates/files on disk before we rollout playbooks. There is a discussion on github about enabling this feature and currently there is some work done on the file lookup feature to allow encrypted templates/files.

Lets rename our sshd_config.j2 template to sshd_config.j2.enc and encrypt it by simply running:


ANSIBLE_VAULT_PASSWORD_FILE=vault_pass_file ansible-vault encrypt roles/ssh/templates/sshd_config.j2.enc

We now need to make a small adjustment to our ssh role’s main.yml, so it first locally decrypts the template and after the rollout removes the decrypted template again.

Modified roles/ssh/tasks/main.yml

Note, that I set changed_when: False for decryption and removal, since without it each run would result in changes, which would make idempotency tests very difficult. Also, local_action has problems to source the python virtualenv inside Travis, which is why I handle the decryption inside Travis with a separate script. This does not affect non-Travis rollouts.

In case we are not dealing with templates, but with static encrypted files instead, we could use the Ansible lookup feature together with the content setting of the copy module:

content={{ lookup('pipe', 'ansible-vault --vault-password-file vault_pass_file view secret.file') }}

Travis CI - Testing my playbooks

We now want to test our playbooks with Travis CI.

In general this is very easy to setup, but when using ansible-vault encrypted files we need to do some extra steps in order for Travis to know the vault password.

Simple tests without encryption

First, we create simple ansible tests assuming our files are all unencrypted. We need a Travis CI account (for open source projects you can use Travis for free!). Travis uses a file called .travis.yml in the top directory of your project to read the test configurations. You can use this file to setup and run your tests inside the Travis infrastructure.

.travis.yml

This configuration initially installs ansible inside the Travis VM. It also sets app in /etc/hosts to point to localhost. Next, we run 3 tests in the script section. The first test is a simple syntax check to catch the most obvious errors. Next, we rollout the playbook against the local VM. Finally, we make an idempotence test by rolling out the same playbook against localhost again. In order to succeed this test, there should not be any changes.

Currently we only have one playbook to test, but we can simply test multiple playbooks in parallel by adding them to the env.matrix section. For instance if we had a playbook db.yml, we could also test it in parallel by simply adding it to the test matrix:

env:
  global:
    - ANSIBLE_VERSION=2.0.1.0
  matrix:
    - PLAYBOOK=app.yml
    - PLAYBOOK=db.yml

Tests with encrypted files

We also want to test our playbooks when they reference encrypted variables and in order for this to work we need to give Travis knowledge about how to decrypt them. For each project Travis generates a public / private keypair, which can be used to share secrets. We can then use the Travis CLI to encrypt values which can later be decrypted by the private key inside the Travis infrastructure for your project. Here is the official documentation about file encryption in Travis.

In our case one way to approach this problem is by encrypting our vault_pass_file to vault_pass_file.enc with a complex password via OpenSSL.


openssl aes-256-cbc -k "<my-very-strong-password>" -in vault_pass_file -out vault_pass_file.enc

We could then add the vault_pass_file.enc to our repository and use Travis CLI to give Travis knowledge about the password to be able to decrypt the vault_pass_file.enc. To share the password with travis, we first need to authenticate with Travis CLI.


travis login

Next, we need to define an encrypted environment variable (in our case we name it vault_file_pass) which stores the password.


travis encrypt vault_file_pass=<my-very-strong-password> --add

The --add flag adds the encrypted data to your env map inside the .travis.yml file.

Finally, we need to tell Travis to decrypt vault_pass_file.enc in the before_install step and decrypt the template files for the tests. Your .travis.yml file should now look something like that.

.travis.yml

Note, that after running the tests I run .travis/sensitive_data.sh clean and shred vault_pass_file in the after_script section to remove the decrypted flles and the cleartext vault password from the Travis VM.

As mentioned above, I had trouble using local_action within the Travis environment, since it does not properly source the python virtual environment and thus cannot find the ansible-vault command. I am still investigating this, but in the meanwhile as a workaround I wrote a script that Travis can use to decrypt the templates before running (and also clean them up after the build is done).

Here is an example of what the .travis/sensitive_data.sh could look like:

.travis/sensitive_data.sh

Some security concerns

While testing with Travis CI is quite comfortable, an obvious issue is that Travis needs to be able to decrypt your secrets. Also, you create VMs/Containers inside the Travis infrastructure which contain your decrypted setups. This means, that if Travis gets compromised so are your secrets. To work around that issue you should consider using fake data in your tests for the most important secrets such as private keys (go snakeoil!!!). That way the provisioned VMs/Containers do not hold any critical security relevant keys. What makes this approach difficult is that ansible-vault only allows one vault password for all the files. Thus, we should at least store real private keys in a different directory which is not part of the repository or encrypt them using another password (which means we would need to decrypt them on disk before using them again, since ansible will use the same provided password to decrypt all the files). Your templates and less important secret variables are still visible to Travis, but at least your private keys and other very important secrets are kept private. Another issue is if your tests require valid SSL certificates in order to run (think end-to-end testing). In that case you might need to apply some workarounds in your test scripts to allow insecure certificates. Of course this adds more maintenance work on your part, but the privacy of your private keys should be worth the effort.

Computing Large Binomial Coefficients Modulo Prime / Non-Prime

Thu, 25 Jun 2015 22:15:53 +0000

Not rarely, in combinatoric problems it comes down to calculating the binomial coefficient $n \choose k$ for very large $n$ and/or $k$ modulo a number $m$. In general, the binomial coefficient can be formulated with factorials as ${n \choose k} = \frac{n!}{k!(n-k)!}, 0 \leq k \leq n$. The problem here is that factorials grow extremely fast which makes this formula computationally unsuitable because of quick overflows. For that reason, many problems in that category require the calculation of ${n \choose k} \mod m$. In this post I want to discuss ways to calculate the binomial coefficients for cases in which $m$ is prime and when $m$ is non-prime.

1. First simple approaches for any $m$

The good news is that there are easy ways to compute the binomial coefficient for any modulo $m$ - the bad news is that they are not feasible for very large numbers. However, we will see later that we can break down very large numbers into sub-problems of much smaller numbers, in which case we can re-use techniques that we discuss now.

Using Pascal’s Triangle

The easiest (but also most limited) way to tackle this problem is to use a bottom-up dynamic programming approach by pre-computing Pascal’s Triangle. Pascal’s Triangle can be seen as a lookup table, where $n \choose k$ is the value at the $n$th row in the $k$th column. Pre-computing Pascal’s Triangle is very easy, since the current value is simply the sum of two neighboring cells from the previous row. We know that $(a+b) \mod m =$$ (a \mod m) +$$ (b \mod m) \mod m$, thus we can calculate each cell of Pascal’s Triangle with the modulo to avoid overflows.

Code: Computing Pascal’s Triangle, for any $m$

Computing the triangle takes quadratic time $O(n^2)$, since we compute each possible $k$ for each row ($n$ rows in total). Once the triangle is computed we can simply lookup the binomial coefficient in constant time $O(1)$for different $n$ and $k$. The big problem arises when looking at the space complexity of $O(n^2)$. For some problems $n$ might be small enough, so there won’t be an issue, but for very large $n$ we will not have enough memory to store the table, which means we have to find a better approach for those cases.

There is an easy upgrade in terms of space complexity. Each row from Pascal’s Triangle can be calculated by only knowing the previous row. This means we only have to keep track of the current row, thus the space complexity is linear $O(n)$.

Code: Computing single row in Pascal’s Triangle, for any $m$

Even though this approach allows to calculate larger binomial coefficients due to better space complexity, it still comes with a downside. In the first approach we can calculate the binomial coefficient in $O(1)$, after we have pre-computed Pascal’s Triangle in $O(n^2)$. However in this approach, we need to recalculate the triangle or compute further rows each time a smaller or bigger row $n$ is asked than the currently computed one. Consequently, this approach is expensive when we expect to calculate many different binomial coefficients that are not in the same row of Pascal’s Triangle.

2. Advanced approaches when $m$ is prime

In most related problems we are asked to calculate ${n \choose k} \mod m$, with $m$ being a prime number. The choice of $m$ being a prime number is by far no coincidence, since prime number modulus create a field in which certain Theorems hold.

Using Fermat’s Little Theorem

We remember that the binomial coefficient can be written as $\frac{n!}{k!(n-k)!} = \frac{n \cdot (n-1) \cdot \dots \cdot (n-k+1)}{k!}$. We can easily calculate the numerator of that equation, since $((a \mod m) \cdot (b \mod m))$$ \mod m = (a \cdot b) \mod m$. In order to divide the numerator by the denominator, we have to multiply the numerator with the multiplicative inverse of the denominator. Lucky for us, if $m$ is prime, we can easily apply Fermat’s little theorem. Through this theorem we know that $a^{-1} \equiv a^{p-2} \mod p$, with $p$ being a prime number. Consequently, we can easily find the multiplicative inverse through modular exponentiation of $(k!)^{m-2}$.

Code: Using Fermat’s Theorem, for $m$ prime and $k < m$

Computing the factorial $n!$ takes $O(n)$ and computing one multiplicative inverse takes $O(\log_2 n)$. Note, that we implemented two versions here:

In the first version, we pre-computed the factorials and the corresponding inverses in an array at the cost of $O(n)$ memory. This pre-computation takes $O(n \log_2 n)$ time and gives us the advantage of further lookups of binomial coefficients within the same range at constant time $O(1)$.

In the second version, without pre-computed lookups, we re-compute the values each time and only calculate the multiplicative inverse of one denominator, thus having a linear time $O(n)$ in total each time we want to calculate a binomial coefficient.

Whether we use pre-computed tables or not highly depends on $n$’s expected size. In case we have very large $n$, pre-computing the factorial and inverse tables with Fermat’s theorem (space complexity $O(n)$ and runtime of pre-computation $O(n \log_2 n)$) might still be too expensive. In all other cases, pre-computing is desired since it will save us a lot of time later on. In the following I will only show algorithms without pre-computation, since those are also working for very large $n$, but changing those algorithms to pre-computed versions can be done easily (analogous to the current example), since the core idea is still the same.

Note, that this code is only working as long as $k < m$, since otherwise $k! \mod m = 0$, which means there is no inverse defined. Luckily, with slight modifications we can work around this issue. First we compare the degree of $m$ in numerator and denominator, because the degree in the numerator has to be smaller or equal. Further, we cancel out any occurrence of $m$ in both, numerator and denominator.

Code: Using Fermat’s Theorem, for $m$ prime

We are now able to determine the binomial coefficient in linear time $O(n)$ as long as the modulo is a prime number. We will see in the next sections how we can enhance the computation in terms of time complexity in some scenarios. Further, we will see how we can determine the binomial coefficients in case the modulo is not a prime number. The algorithms involving Fermat and Pascal are very crucial, since all further approaches that we are going to discuss in this post are relying on them to solve the core computation.

Using Lucas’ Theorem

Lucas’ theorem gives us a great way to break down the computation of one large binomial coefficient into the computation of smaller binomial coefficients. In order to do so, we first have to convert the digits of $n$ and $k$ into their base $m$ representation $n = n_0 + n_1 \cdot m^1 + n_2 \cdot m^2 \dots n_x \cdot m^x$ and $k = k_0 + k_1 \cdot m^1 + k_2 \cdot m^2 \dots k_x \cdot m^x$. We can then define sub problems: ${n \choose k} = \prod_{i=0}^x {n_i \choose k_i} (\mod m)$. These problems can then be solved using Fermat’s theorem like we did before.

Code: Using Lucas’ + Fermat’s Theorem, for $m$ prime

A little side fact: Lucas’ theorem gives an efficient way of determining whether our prime $m$ is a divisor of the binomial coefficient, without calculating the coefficient itself. All we have to do is compare the digits of $n$ and $k$ in their base $m$ representation $n = n_0 + n_1 \cdot m^1 + n_2 \cdot m^2…$ and $k = k_0 + k_1 \cdot m^1 + k_2 \cdot m^2 …$ and if in at least one case $k_i > n_i$, then the product of all sub binomial coefficients would be 0, since ${n \choose k} = 0$, when $k > n$.

We used Lucas’ theorem to first split the problem into smaller binomial coefficients that we could then use to find our solution. In order to solve the small coefficients, we used Fermat’s theorem like in the previous section of this post. We could have also used Pascal’s Triangle in order to compute these small coefficients. Also, in case (n) and (k) are smaller than (m), Lucas’ theorem would not have brought any benefits, because the smallest sub problem would then be the original problem itself. However, in that case Lucas’ theorem does only bear the small overhead of bringing $n$ and $k$ in base $m$ representation, which is why in general it is a good approach to first try splitting the problem into smaller sub problems with Lucas’ theorem, then using Pascal’s Triangle or Fermat’s theorem to solve these sub problems.

Remember, in this example we did not use pre-computation, which makes it more efficient when computing only one single very large $n$. Whether we use pre-computed tables or not highly depends on the expected size of $n$ (which when using Lucas’ theorem, is the biggest prime factor in $m$) and whether we expect to calculate many different coefficients. In case our sub problems have still very large $n$, pre-computing the factorial and inverse tables with Fermat’s theorem (space complexity $O(n)$ and initial runtime $O(n \log_2 n)$) might still be too expensive to be feasible. In all other cases, pre-computing is desired since it will save us a lot of time later on.

All in all, now we have efficient ways of computing large binomial coefficients modulo a prime number.

3. $m$’s prime factorization is square-free

What do we do in cases in which $m$ turns out not to be a prime? The answer lies in the prime factorization of $m$. The following method works when $m$’s prime factorization is square-free, e.g. $6 = 2 \cdot 3$, which means $6$ is square-free, whereas $24 = 2^3 \cdot 3$ is not square-free, since the prime $2$ occurs more than once.

Lets assume $m$’s prime factors are $m = p_0 \cdot p_1 \dots \cdot p_r$. For each prime factor $p_i$ of $m$, we can solve ${n \choose k} \mod p_i$, using the approaches we have discussed so far, which leaves us with $r$ results $a_0, a_1, \dots, a_r$. Through the Chinese Remainder Theorem (CRT) we know that we can find a number $x$, which solves the following congruences $x \equiv a_0 \mod p_0$, $x \equiv a_1 \mod p_1$, $\dots$, $x \equiv a_r \mod p_r$.

Let’s define $m_i = \frac{p_0 \cdot p_1 \cdot \dots \cdot p_r}{p_i}$. We know that $m_i$ and $p_i$ must be coprime, then $gcd(m_i,p_i) = 1$, meaning $1 = s_i \cdot m_i + t_i \cdot p_i$. Using the Extended Euclidean Algorithm, we can find such $s_i$ and $t_i$. Finally, we can find the solution that combines all the congruences to $x = \sum_{i=0}^r (a_i \cdot m_i \cdot s_i) (\mod \prod_{i=0}^r m_i)$,

The CRT works as long as $p_0 \dots p_r$ are co-prime. We see, that by applying the CRT to those congruence systems, we can combine them into a single congruence modulo $m$, which solves our problem.

Code: Using Lucas’ + Fermat’s + Chinese Remainder Theorem, for (m) with square-free prime factors

Finding the prime factorization of $m$ is non-trivial and no efficient algorithm is known so far that can solve this problem for large $m$. However, as long as $m$ has a reasonable size (and that is very likely in those combinatoric problems), finding the prime factorization is feasible. You can write your own algorithm or use some online calculators to determine the prime factors. Also remember that our current approach is only valid for $m$ whose prime factors are square-free.

Now we are also able to compute large binomial coefficients modulo numbers whose prime factorization is square-free.

4. $m$ has prime factors with multiplicity higher than 1

Finally, lets have a look at all the remaining possible modulo numbers, namely those containing prime factors with a multiplicity higher than 1. For that purpose we can use Andrew Granville’s generalization of Lucas’ Theorem. This theorem also allows prime powers in the factorization of $m$. After applying the theorem, we could then use our other methods to calculate the sub problems and then like before use the Chinese Remainder Theorem to combine the sub solutions into one solution. With prime powers allowed, we can then factorize any modulus $m$, meaning we can find the binomial coefficients modulo any number $m$. I don’t have an implementation ready yet for the generalization of Lucas’ Theorem, but once I have some time I will try to add it to this post.

5. Quick Summary

We should do the following steps in order to compute large binomial coefficients ${n \choose k} \mod m$:

Find prime factors (and multiplicities) $p_0^{e_0}, \dots p_r^{e_r}$ of $m$
Use (generalized) Lucas’ Theorem to find all sub problems for each ${n \choose k} \mod p_i^{e_i}$
Solve sub problems ${n \choose k} \mod p_i^{e_i}$ with Fermat’s little theorem or Pascal’s Triangle
Use Chinese Remainder Theorem to combine sub results

Remember, if $p_i > n$, then there are no sub problems and we basically just solve the problem at hand using Fermat’s little theorem or Pascal’s Triangle.

That’s it! Now we can compute the binomial coefficients for very large numbers for any modulo $m$.

Another Python BrainF*** Interpreter

Mon, 15 Jun 2015 22:21:12 +0000

BrainF*** (BF) is a famous esoteric programming language. Writing an interpreter for BF is actually easier or should I say at least easier to comprehend than writing an actual BF Hello World! program. Even though BF can theoretically be used to write any program, it is not practical to do so because of its low abstraction level. I guess that’s one reason why it feels like I find more BF interpreters written in all kinds of languages scattered over many tech blogs rather than actual BF programs. Still, since I had a free day, a nice cold drink and the urge to script something I decided to add yet another Python BF interpreter to the web. Though for Python BF there are some nice one-liner code-golf-like versions out there, I chose the simpler approach of writing a structured, easy to comprehend version.

BF Basics

Before we get started on the interpreter code itself, let’s start off with some BF basics.

In BF you have a memory of usually 30000 cells - also referred to as tape - and each cell holds one byte (0-255). You also have exactly one memory pointer initially pointing to the leftmost memory cell. Further, you have an instruction pointer keeping track of which operation to execute next.

BF consists of only the following 8 operations:

>: shift the memory pointer 1 to the right
<: shift the memory pointer 1 to the left
+: increment byte in cell at memory pointer
-: decrement byte in cell at memory pointer
.: output byte from cell at memory pointer
,: accept input and store in cell at memory pointer
[: loop start: if byte in cell at memory pointer is zero, then jump to corresponding closing bracket ]. Otherwise execute the next command.
]: loop end: if byte in cell at memory pointer is nonzero, then jump back to corresponding opening bracket [. Otherwise execute next command.

As you can see the operations are quite simple and easy to comprehend. Apart from that, some additional constraints have to be taken into account:

Memory Borders

As I have already mentioned BF usually has 30000 cells. The question now is what happens when the memory pointer is shifted beyond those limits? One way to handle this is by wrapping the pointer around, meaning when being shifted right from the right-most cell, the pointer goes to the left-most cell and vice versa. Another, simpler approach would be to throw a segmentation fault or some other kind of exception.

Memory Space

You could initialize a memory with the maximum amount of cells. If you only need a very small portion of that memory, than most of the memory was unnecessarily reserved. Another way would be to grow the memory on demand until it reaches its maximum allowed value. Whenever the pointer is moved to the right and the cell is not initialized yet, we then reserve the memory for that cell. The advantage of this method is obviously a more efficient space usage. The clear disadvantage is that each reservation costs time and thus influences the overall speed of the BF interpreter.

Argument Input

There are 2 major ways to get input. The first way is via standard user input via STDIN. It is easy to implement, but for longer repetitive programs quite annoying for the user. Another way is to use an extra file with input arguments.

BF Loops

Most BF operations are very easy to implement and mainly consist of a few pointer movements. Depending on whether we decide to use memory wrapping or separate argument input files as described before, the complexity slightly rises, but is still very easy to comprehend.

The only thing we should have a closer discussion about is the loop implementation. A naive way to implement the loop could be to move the instruction pointer until the corresponding closing or opening bracket is found. We must be aware of possible nested loops, meaning we have to count the occurrences of each bracket while looking for the matching ones - a stack structure might come in handy at that point. However, searching the source code for the corresponding bracket each time we encounter one is rather inefficient. At least we should remember the brackets in a map, so next time we can simply lookup the jump to the next instruction without searching. We could also pre-parse the code in order to make a map of all matching brackets, even before we start executing the first operation.

Finally - Some Code

Finally, let’s have a look at some BF interpreter code!

This BF interpreter has no memory wrapping, so whenever the memory pointer moves beyond the limits, a SegFault exception is thrown. Further, an optional input file can be used to feed the input BF function ,. In case no input file is provided or all bytes have already been read from the input file, the user has to provide input via STDIN.

In the first section we simply added some specific exceptions which might occur (such as SegFaults). In the second section we define a BF process which holds the context of our BF program. This class also implements every allowed BF operation and a code parse function. Finally, in the main routine we remove everything from our source code that’s not BF code syntax, then build a context and execute the program. Prior to the first operation the code is parsed to get a map of the matching loop brackets.

And finally, here is a BF Hello World!.

hello_world.bf

++++++++++[>+++++++>++++++++++>+++>+<<<<-]>++.>+.
+++++++..+++.>++.<<+++++++++++++++.>.+++.------.
--------.>+.>.

When giving this code as an argument to our BF interpreter we should be able to see a Hello World!.

Karl Fischer

Measuring Traffic with iptables

Benefits of using iptables for traffic measurements

Hands-on Example

Jenkins-as-Code Part III | JobDSL

JobDSL Introduction

Multibranch Build Pipelines

Conclusion

Jenkins-as-Code Part II | Configuration

Directory Layout

Configuration as Code Plugin

Configuration Scripts

Timezone

User Public Keys

Others

Summary

Jenkins-as-Code Part I | Initial Setup

Layers and Terminology

Jenkins-as-Code

Job Interface

Pipeline Definition

Configuration Script

CasC

JobDSL Script

Seeding

Layers and their Interaction

Jenkins Docker Image

Directory Structure

Dockerfile and Plugins

Initial Pipeline Interface

init.groovy.d for initial setup

Shared Library Repository

Directory Structure

Summary

My Consul Service ID hunt

A Docker Build Pipeline as Code with Jenkins

Setup Assumptions

The Jenkins Docker Pipeline

Ansible role structuring

A simple ssh role

A more mature role structure

Testing ansible playbooks with ansible-vault encrypted data using Travis CI

Ansible - A simple example playbook

Encrypting secrets with ansible-vault

Secret Variables

Secret Templates

Travis CI - Testing my playbooks

Simple tests without encryption

Tests with encrypted files

Some security concerns

Computing Large Binomial Coefficients Modulo Prime / Non-Prime

1. First simple approaches for any \(m\)

Using Pascal’s Triangle

2. Advanced approaches when \(m\) is prime

Using Fermat’s Little Theorem

Using Lucas’ Theorem

3. \(m\)’s prime factorization is square-free

4. \(m\) has prime factors with multiplicity higher than 1

5. Quick Summary

Another Python BrainF*** Interpreter

BF Basics

BF Loops

Finally - Some Code