Posted in Devops, Information Technology, microservices

Container Orchestrator

With enterprises containerizing their applications and moving them to the cloud, there is a growing demand for container orchestration solutions. While there are many solutions available, some are mere re-distributions of well-established container orchestration tools, enriched with features and, sometimes, with certain limitations in flexibility.

Although not exhaustive, the list below provides a few different container orchestration tools and services available today:

Posted in Devops, Tools

Tutorial Creating Jenkins Job using DSL

This tutorial will walk you through how to create a single job using a DSL script; and then add a few more.

1. Creating the Seed Job

We use a Free-style Jenkins Job as a place to run the DSL scripts. We call this a “Seed Job”. Since it’s a normal Job you’ll get all the standard benefits of Jenkins: history, logs, emails, etc. We further enhance the Seed Job to show which Jobs got created from the DSL script, in each build and on the Seed Job page.

The first step is to create this Job.

  • From the Jenkins main page, select either the “New Job” or “Create new Jobs” link. A new job creation page will be displayed.

[[images/newjob.png|center|frame]]

  • Fill in the name field, e.g. “tutorial-job-dsl-1”
  • Select the “Build a free-style software project” radio button.
  • Click the OK button

[[images/createjob.png|center|frame]]

2. Adding a DSL Script

Now that we have created our empty Seed Job we need to configure it. We’re going to add a build step to execute the Job DSL script. Then we can paste in an example script as follows:

  • On the configure screen, scroll down to the “Build: Add build step” pull down menu

[[images/AddBuildStep.png|center|frame]]

  • From the pull down menu, select “Process Job DSLs”. You should be presented with two radio buttons. The default will be “Use the provided DSL script” and a text input box will be displayed below it.

[[images/AddBuildStepSelected.png|center|frame]]

  • Copy the following DSL Script block into the input box. (Note: The job resulting from this will be called DSL-Tutorial-1-Test. It’ll check a GitHub repo every 15 minutes, then run ‘clean test’ if there’s any changes found.)
job('DSL-Tutorial-1-Test') {
    scm {
        git('git://github.com/quidryan/aws-sdk-test.git')
    }
    triggers {
        scm('H/15 * * * *')
    }
    steps {
        maven('-e clean test')
    }
}

[[images/DslBuildStep.png|center|frame]]

  • Click the “Save” button. You’ll be shown the overview page for the new Seed job you just created.

[[images/EmptySeed.png|center|frame]]

3. Run the Seed Job and Generate the new Jobs from the Script

The Seed Job is now all set up and can be run, generating the Job we just scripted.

(Note: As it stands right now, we didn’t setup any build triggers to run the job automatically but we could have, using the standard Jenkins UI in Step 2.)

Let’s just run it ourselves manually.

  • Click the “Build Now” link/button on the tutorial-job-dsl-1 overview page. It should only take a second to run.

[[images/Build1.png|center|frame]]

  • Look at the build result to see a link to the new Job which has been created by the running of your DSL script in the Seed Job. You should see this in the section called “Generated Jobs”. (If you don’t see it, you probably have Auto-Refresh disabled. Enable it, or just refresh the page and then you’ll see the new job.)
  • Follow this link to your new Job. You can run this new script-generated Job manually or wait the 15 minutes for the scm trigger to kick in.

(Note: if you have a new Jenkins server, you might be missing the Git plugin or a Maven installation which Jenkins knows about. That could cause this job to fail when run. If you need to add these, be sure to re-run the Seed Job to make sure the Scripted Job is configured correctly – it won’t be if you ran without all the necessary plugins installed in Jenkins.)

(Additional Note: if the build still fails with these plugins / config set up, it may be because the new job is using a “default” maven rather than the one you just added.)

4. Adding additional Jobs to the DSL Script

To show some more of the power of the DSL Plugin, let’s create a bunch more Jobs.

  • Go back to the ‘tutorial-job-dsl-1’ Seed Job
  • Click the “Configure” link/button and navigate back down the the “Process Job DSLs” build step.
  • Add the following into the text box, below the script which we added at the beginning.

(Note: The practicality of this block is questionable, but it could be used to shard your tests into different jobs.)

def project = 'quidryan/aws-sdk-test'
def branchApi = new URL("https://api.github.com/repos/${project}/branches")
def branches = new groovy.json.JsonSlurper().parse(branchApi.newReader())
branches.each {
    def branchName = it.name
    def jobName = "${project}-${branchName}".replaceAll('/','-')
    job(jobName) {
        scm {
            git("git://github.com/${project}.git", branchName)
        }
        steps {
            maven("test -Dproject.name=${project}/${branchName}")
        }
    }
}

5. Enjoy the results

That’s it. Now you know how to make Seed Jobs, which can create a multitude of Scripted child Jobs. Take a look at some Real World Examples or jump ahead and read up on the DSL commands in detail for more fun.

Posted in Devops, Software Engineering

VIM

Vim is a very efficient text editor. This reference was made for Vim 8.0.
For shortcut notation, see :help key-notation.

Exiting

:qa Close all files
:qa! Close all files, abandon changes
:w Save
:wq / :x Save and close file
:q Close file
:q! Close file, abandon changes
ZZ Save and quit
ZQ Quit without checking changes
h j k l Arrow keys
<C-U> / <C-D> Page up/page down

Words

b / w Previous/next word
e / ge Previous/next end of word

Line

0 (zero) Start of line
^ Start of line (after whitespace)
$ End of line

Character

fc Go forward to character c
Fc Go backward to character c

Document

gg First line
G Last line
:n Go to line n
nG Go to line n

Window

zz Center this line
H Move to top of screen
M Move to middle of screen
L Move to bottom of screen

Tab pages

:tabedit [file] Edit file in a new tab
:tabfind [file] Open file if exists in new tab
:tabclose Close current tab
:tabs List all tabs
:tabfirst Go to first tab
:tablast Go to last tab
:tabn Go to next tab
:tabp Go to previous tab

Editing

a Append
i Insert
o Next line
O Previous line
s Delete char and insert
S Delete line and insert
C Delete until end of line and insert
r Replace one character
R Enter Replace mode
u Undo changes
<C-R> Redo changes

Exiting insert mode

Esc / <C-[> Exit insert mode
<C-C> Exit insert mode, and abort current command

Clipboard

x Delete character
dd Delete line (Cut)
yy Yank line (Copy)
p Paste
P Paste before

Visual mode

v Enter visual mode
V Enter visual line mode
<C-V> Enter visual block mode

In visual mode

d / x Delete selection
s Replace selection
y Yank selection (Copy)

See Operators for other things you can do.

#Operators

Usage

Operators let you operate in a range of text (defined by motion). These are performed in normal mode.

d w
Operator Motion

Operators list

d Delete
y Yank (copy)
c Change (delete then insert)
> Indent right
< Indent left
g~ Swap case
gU Uppercase
gu Lowercase
! Filter through external program

See :help operator

Examples

Combine operators with motions to use them.

dd (repeat the letter) Delete current line
dw Delete to next word
db Delete to beginning of word
2dd Delete 2 lines
dip Delete a text object (inside paragraph)
(in visual mode) d Delete selection

See: :help motion.txt

#Text objects

Usage

Text objects let you operate (with an operator) in or around text blocks (objects).

v i p
Operator [i]nside or [a]round Text object

Text objects

p Paragraph
w Word
s Sentence
[ ( { < A [], (), or {} block
' " ` A quoted string
b A block [(
B A block in [{
t A XML tag block

Examples

vip Select paragraph
vipipipip Select more
yip Yank inner paragraph
yap Yank paragraph (including newline)
dip Delete inner paragraph
cip Change inner paragraph

See Operators for other things you can do.

Diff

gvimdiff file1 file2 [file3] See differencies between files, in HMI

#Misc

Folds

zo / zO Open
zc / zC Close
za / zA Toggle
zv Open folds for this line
zM Close all
zR Open all
zm Fold more (foldlevel += 1)
zr Fold less (foldlevel -= 1)
zx Update folds

Uppercase ones are recursive (eg, zO is open recursively).

[( [{ [< Previous ( or { or <
]) Next
[m Previous method start
[M Previous method end

Jumping

<C-O> Go back to previous location
<C-I> Go forward
gf Go to file in cursor

Counters

<C-A> Increment number
<C-X> Decrement

Windows

z{height}<Cr> Resize pane to {height} lines tall

Tags

:tag Classname Jump to first definition of Classname
<C-]> Jump to definition
g] See all definitions
<C-T> Go back to last tag
<C-O> <C-I> Back/forward
:tselect Classname Find definitions of Classname
:tjump Classname Find definitions of Classname (auto-select 1st)

Case

~ Toggle case (Case => cASE)
gU Uppercase
gu Lowercase
gUU Uppercase current line (also gUgU)
guu Lowercase current line (also gugu)

Do these in visual or normal mode.

Marks

`^ Last position of cursor in insert mode
`. Last change
`` Last jump
ma Mark this cursor position as a
`a Jump to the cursor position a
'a Jump to the beginning of the line with position a

Misc

. Repeat last command
]p Paste under the current indentation level

Command line

<C-R><C-W> Insert current word into the command line
<C-R>" Paste from “ register
<C-X><C-F> Auto-completion of path in insert mode

Text alignment

:center [width]
:right [width]
:left

See :help formatting

Calculator

<C-R>=128/2 Shows the result of the division : ‘64’

Do this in insert mode.

Exiting with an error

:cq
:cquit

Works like :qa, but throws an error. Great for aborting Git commands.

Spell checking

:set spell spelllang=en_us Turn on US English spell checking
]s Move to next misspelled word after the cursor
[s Move to previous misspelled word before the cursor
z= Suggest spellings for the word under/after the cursor
zg Add word to spell list
zw Mark word as bad/mispelling
zu / C-X (Insert Mode) Suggest words for bad word under cursor from spellfile

See :help spell

#Also see

Posted in Devops, Information Technology

A comparison of tools that help developers build and deploy their apps on Kubernetes

Overview

  • Draft
    – deploy code to k8s cluster (automates build-push-deploy)
    – deploy code in draft-pack supported languages without writing dockerfile or k8s manifests
    – needs draft cli, helm cli, tiller on cluster, local docker, docker registry
  • Gitkube
     deploy code to k8s cluster (automates build-push-deploy)
    – git push to deploy, no dependencies on your local machine
    – needs dockerfile, k8s manifests in the git repo, gitkube on cluster
  • Helm
    – deploy and manage charts (collection of k8s objects defining an application) on a k8s cluster
    – ready made charts for many common applications, like mysql, mediawiki etc.
    – needs helm cli, tiller on cluster, chart definition locally or from a repo
  • Ksonnet
    – define k8s manifests in jsonnet, deploy them to k8s cluster
    – reusable components for common patterns and stacks, like deployment+service, redis
    – needs jsonnet knowledge, ksonnet cli
  • Metaparticle
    – deploy your code in metaparticle supported languages to k8s (automates build-push-deploy)
    – define containerizing and deploying to k8s in the language itself, in an idiomatic way, without writing dockerfile or k8s yaml
    – needs metaparticle library for language, local docker
  • Skaffold
    – deploy code to k8s cluster (automates build-push-deploy)
    – watches source code and triggers build-push-deploy when change happens, configurable pipeline
    – needs skaffold cli, dockerfile, k8s manifests, skaffold manifest in folder, local docker, docker registry

Want to know more? Read ahead.

Kubernetes is super popular nowadays and people are looking for more ways and workflows to deploy applications to a Kubernetes cluster. kubectl itself has become like a low-level tool, with people looking for even easier workflows. Draft, Gitkube, Helm, Ksonnet, Metaparticle and Skaffold are some of the tools around that help developers build and deploy their apps on Kubernetes.

Draft, Gitkube and Skaffold ease the developer effort, when you are building an application, to get it running on a Kubernetes cluster as quickly as possible. Helm and Ksonnet help the deployment process once your app is built and ready to ship, by defining applications, handling rollout of new versions, handling different clusters etc. Metaparticle is an odd one out here, since it combines everything into your code — yaml, dockerfile, all in the code itself.

So, what should you use for your use case?

Let’s discuss.

 

Draft

Simple app development & deployment — on to any Kubernetes cluster.

As the name suggests, Draft makes developing apps that run on Kubernetes clusters easier. The official statement says that Draft is a tool for developing applications that run on Kubernetes, not for deploying them. Helm is the recommended way of deploying applications as per Draft documentation.

The goal is to get the current code on the developer’s machine to a Kubernetes cluster, while the developer is still hacking on it, before it is committed to version control. Once the developer is satisfied by the changes made and deployed using Draft, the code is committed to version control.

Draft is not supposed to be used for production deployments as it is purely intended to be used for a very quick development workflow when writing applications for Kubernetes. But it integrates very well with Helm, as it internally uses Helm to deploy the changes.

Architecture

Draft: architecture diagram

As we can see from the diagram, draft CLI is a key component. It can detect language used from the source code and then use an appropriate pack from a repo. A pack is a combination of Dockerfile and Helm chart which together defines the environment for an application. Packs can be defined and distributed in repos. Users can define their own packs and repos as they’re present as files in the local system or a git repo.

Any directory with source code can be deployed if there is a pack for that stack. Once the directory is setup using draft create(this adds dockerfile, Helm chart and draft.toml), draft up can build the docker image, push it to a registry and rollout the app using Helm chart (provided Helm is installed). Every time a change is made, executing the command again will result in a new build being deployed.

There is a draft connect command which can port forward connections to your local system as well as stream logs from the container. It can also integrate with nginx-ingress to provide domain names to each app it deploys.

From zero to k8s

Here are the steps required to get a python app working on a k8s cluster using Draft. (See docs for a more detailed guide)

Prerequisites:

  • k8s cluster (hence kubectl)
  • helm CLI
  • draft CLI
  • docker
  • docker repository to store images
$ helm init

Use case

  • Developing apps that run on Kubernetes
  • Used in “inner loop”, before code is committed onto version control
  • Pre-CI: Once development is complete using draft, CI/CD takes over
  • Not to be used in production deployment

More details here.

Gitkube

Build and deploy Docker images to Kubernetes using git push

Gitkube is a tool that takes care of building and deploying your Docker images on Kubernetes, using git push. Unlike draft, gitkube has no CLI and runs exclusively on the cluster.

Any source code repo with a dockerfile can be deployed using gitkube. Once gitkube is installed and exposed on the cluster, developer can create a remote custom resource which gives a git remote url. The developer can then push to the given url and docker build-kubectl rollout will happen on the cluster. The actual application manifests can be created using any tool (kubectl, helm etc.)

Focus is on plug and play installation & usage of existing well known tools (git and kubectl). No assumptions are made about about the repo to be deployed. The docker build context and dockerfile path, along with deployments to be updated are configurable. Authentication to the git remote is based on SSH public key. Whenever any change in code is made, committing and pushing it using git will trigger a build and rollout.

Architecture

Gitkube: architecture diagram

There are 3 components on the cluster, a remote CRD which defines what should happen when a push is made on a remote url, gitkubed which builds docker images and updates deployments, and a gitkube-controller which is watching on the CRD to configure gitkubed.

Once these objects are created on the cluster, a developer can create their own applications, using kubectl. Next step is to create a remote object which tells gitkube what has to happen when a git push is made to a particular remote. Gitkube writes the remote url back to the status field of the remote object.

From zero to k8s

Prerequisites:

  • k8s cluster (kubectl)
  • git
  • gitkube installed on the cluster (kubectl create)

Here are the steps required to get you application on Kubernetes, including installation of gitkube:

$ git clone https://github.com/hasura/gitkube-example

$ cd gitkube-example

$ kubectl create -f k8s.yaml$ cat ~/.ssh/id_rsa.pub | awk '$0="  - "$0' >> "remote.yaml"

$ kubectl create -f remote.yaml

$ kubectl get remote example -o json | jq -r '.status.remoteUrl'

$ git remote add example [remoteUrl]

$ git push example master

## edit code
## commit and push

Use case

  • Easy deployments using git, without docker builds
  • Developing apps on Kubernetes
  • While development, WIP branch can be pushed multiple times to see immediate results

More details here.

Helm

The package manager for Kubernetes

As the tag suggests, Helm is a tool to manage applications on Kubernetes, in the form of Charts. Helm takes care of creating the Kubernetes manifests and versioning them so that rollbacks can be performed across all kind of objects, not just deployments. A chart can have deployment, service, configmap etc. It is also templated so that variables can be easily changed. It can be used to define complex applications with dependencies.

Helm is primarily intended as a tool to deploy manifests and manage them in a production environment. In contrast to Draft or Gitkube, Helm is not for developing applications, but to deploy them. There are a wide variety of pre-built charts ready to be used with Helm.

Architecture

Helm: architecture diagram

Let’s look at Charts first. As we mentioned earlier, a chart a bundle of information necessary to create an instance of a Kubernetes application. It can have deployments, services, configmaps, secrets, ingress etc. all defined as yaml files, which in turn are templates. Developers can also define certain charts as dependencies for other charts, or nest charts inside another one. Charts can be published or collated together in a Chart repo.

Helm has two major components, the Helm CLI and Tiller Server. The cli helps in managing charts and repos and it interacts with the Tiller server to deploy and manage these charts.

Tiller is a component running on the cluster, talking to k8s API server to create and manage actual objects. It also renders the chart to build a release. When the developer does a helm install <chart-name> the cli contacts tiller with the name of the chart and tiller will get the chart, compile the template and deploy it on the cluster.

Helm does not handle your source code. You need use some sort of CI/CD system to build your image and then use Helm to deploy the correct image.

From zero to k8s

Prerequisites:

  • k8s cluster
  • helm CLI

Here is an example of deploying WordPress blog onto a k8s cluster using Helm:

$ helm init
$ helm repo update
$ helm install stable/wordpress
## make new version
$ helm upgrade [release-name] [chart-name]

Use case

  • Packaging: Complex applications (many k8s objects) can be packaged together
  • Reusable chart repo
  • Easy deployments to multiple environments
  • Nesting of charts — dependencies
  • Templates — changing parameters is easy
  • Distribution and reusability
  • Last mile deployment: Continuous delivery
  • Deploy an image that is already built
  • Upgrades and rollbacks of multiple k8s objects together — lifecycle management

More details here.

Ksonnet

A CLI-supported framework for extensible Kubernetes configurations

Ksonnet is an alternate way of defining application configuration for Kubernetes. It uses Jsonnet, a JSON templating language instead of the default yaml files to define k8s manifests. The ksonnet CLI renders the final yaml files and then applies it on the cluster.

It is intended to be used for defining reusable components and incrementally using them to build an application.

Architecture

Ksonnet: overview

The basic building blocks are called parts which can be mixed and matched to create prototypes. A prototype along with parameters becomes a component and components can be grouped together as an application. An application can be deployed to multiple environments.

The basic workflow is to create an application directory using ks init, auto-generate a manifest (or write your own) for a component using ks generate, deploy this application on a cluster/environment using ks apply <env>. You can manage different environments using ks env command.

In short, Ksonnet helps you define and manage applications as collection of components using Jsonnet and then deploy them on different Kubernetes clusters.

Like Helm, Ksonnet does not handle source code, it is a tool for defining applications for Kubernetes, using Jsonnet.

From zero to k8s

Prerequisites:

  • k8s cluster
  • ksonnet CLI

Here’s a guestbook example:

$ ks init
$ ks generate deployed-service guestbook-ui \     --image gcr.io/heptio-images/ks-guestbook-demo:0.1 \     --type ClusterIP
$ ks apply default
## make changes
$ ks apply default

Use case

  • Flexibility in writing configuration using Jsonnet
  • Packaging: Complex configurations can be built as mixing and matching components
  • Reusable component and prototype library: avoid duplication
  • Easy deployments to multiple environments
  • Last-mile deployment: CD step

More details here.

Metaparticle

Cloud native standard library for Containers and Kubernetes

Positioning itself as the standard library for cloud native applications, Metaparticle helps developers to easily adopt proven patterns for distributed system development through primitives via programming language interfaces.

It provides idiomatic language interfaces which helps you build systems that can containerize and deploy your application to Kubernetes, develop replicated load balanced services and a lot more. You never define a Dockerfile or a Kubernetes manifest. Everything is handled through idioms native to the programming language that you use.

For example, for a Python web application, you add a decorator called containerize (imported from the metaparticle package) to your main function. When you execute the python code, the docker image is built and deployed on to your kubernetes cluster as per parameters mentioned in the decorator. The default kubectl context is used to connect to cluster. So, switching environments means changing the current context.

Similar primitives are available for NodeJS, Java and .NET and support for more languages are a work in progress.

Architecture

The metaparticle library for the corresponding language have required primitives and bindings for building the code as a docker image, pushing it to a registry, creating k8s yaml files and deploying it onto a cluster.

The Metaparticle Package contains these language idiomatic bindings for building containers. Metaparticle Sync is a library within Metaparticle for synchronization across multiple containers running on different machines.

JavaScript/NodeJS, Python, Java and .NET are supported during the time of writing.

From zero to k8s

Prerequisites:

  • k8s cluster
  • metaparticle library for the supported language
  • docker
  • docker repository to store images

A python example (only relevant portion) for building docker image with the code and deploying to a k8s cluster:

@containerize(
    'docker.io/your-docker-user-goes-here',
    options={
        'ports': [8080],
        'replicas': 4,
        'runner': 'metaparticle',
        'name': 'my-image',
        'publish': True
    })
def main():
    Handler = MyHandler
    httpd = SocketServer.TCPServer(("", port), Handler)
    httpd.serve_forever()

Use case

  • Develop applications without worrying about k8s yaml or dockerfile
  • Developers no longer need to master multiple tools and file formats to harness the power of containers and Kubernetes.
  • Quickly develop replicated, load-balanced services
  • Handle synchronization like locking and master election between distributed replicas
  • Easily develop cloud-native patterns like sharded-systems

More details here.

Skaffold

Easy and Repeatable Kubernetes Development

Skaffold handles the workflow of building, pushing and deploying an application to Kubernetes. Like Gitkube, any directory with a dockerfile can be deployed to a k8s cluster with Skaffold.

Skaffold will build the docker image locally, push it to a registry and rollout the deployment using the skaffold CLI tool. It also watches the directory so that rebuilds and redeploys happen whenever the code inside the directory changes. It also streams logs from containers.

The build, push, deploy pipelines are configurable using a yaml file, so that developers can mix and match tools that like for these steps: e.g. docker build vs google container builder, kubectl vs helm for rollout etc.

Architecture

Skaffold: overview

Skaffold CLI does all the work here. It looks at a file called skaffold.yaml which defines what has to be done. A typical example is to build the docker image using dockerfile in the directory where skaffold dev is run, tag it with its sha256, push the image, set that image in the k8s manifest pointed to in the yaml file, apply that manifests on the cluster. This is run continuously in a loop triggering for every change in the directory. Logs from the deployed container is streamed to the same watch window.

Skaffold is very similar to Draft and Gitkube, but more flexible, as it can manage different build-push-deploy pipelines, like the one shown above.

From zero to k8s

Prerequisites:

  • k8s cluster
  • skaffold CLI
  • docker
  • docker repository to store images

Here are the steps to deploy an example Go application that prints hello-world:

$ git clone https://github.com/GoogleCloudPlatform/skaffold
$ cd examples/getting-started
## edit skaffold.yaml to add docker repo
$ skaffold dev
## open new terminal: edit code

Use case

  • Easy deployments
  • Iterative builds — Continuous build-deploy loop
  • Developing apps on Kubernetes
  • Define build-push-deploy pipelines in CI/CD flow

More details here.


Let me know in the comments if I missed something out or got something wrong. I have not mentioned tools like Ksync and Telepresence, as I plan to write another post about them soon. But, if there are other tools that fit into the category of those mentioned here, please drop a note in the comments.

You can find the discussion on Hacker News here.

We’ve been trying to solve similar problems around deployment with the Hasura platform and our current approach is to:

  1. Bootstrap a monorepo of microservices that contains Dockerfiles and Kubernetes specs
  2. git push to build and deploy all the microservices in one shot

This method helps you get started without knowing about Dockerfiles and Kubernetes (using boilerplates) but keeps them around in case you need access to them. It also has a very simple (git push based) deployment method.

Posted in Devops, Information Technology

Bash Regex Cheat Sheet

Following are the few usefull bash regex

Regexp Matching

Use conditions with doubled [] and the =~ operator. Ensure not to quote the regular expression. Only BRE are allowed. If the regexp has whitespaces put it in a variable first.

if [[ $string =~ ^[0-9]+$ ]]; then 
    echo "Is a number"
fi

Regexp Match Extraction

Variant #1: You can do this with grouping in bash. Despite only BRE being supported grouping works also. Note how you need to set the regexp into a variable because you must not quote it in the if condition!

REGEXP="2013:06:23 ([0-9]+):([0-9]+)"
if [[ $string =~ $REGEXP ]]; then
    echo "Hour ${BASH_REMATCH[1]} Minute ${BASH_REMATCH[2]}"
fi

Variant #2: Actually using “expr” can much simpler especially when only on value is to be extracted:

hour=$(expr match "$string" '2013:06:23 \([0-9]\+\)')

Validate IPs

If you need to validate an IP try the following function

function validate_ip {
        local net=$1
        [[ $net =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/[0-9]{1,2}$ ]] || return 1
        [[ ${net#*/} -le 32 ]] || return 1
        local ip=${net%/*}
        local -a oc=(${ip//\./ })
        [[ ${oc[0]} -le 255 && ${oc[1]} -le 255 && ${oc[2]} -le 255 && ${oc[3]} -le 255 ]] || return 1
        return 0
}

 

Posted in Devops, Information Technology

Shell-Scripting Cheat Sheet

Date Handling

Convert Date To Unix Timestamp

date -d "$date" +%s

Note that this only works for American style dates. European “25.06.2014” like dates are not supported. The simple solution is to convert them first to “2014-06-25” for example with

sed 's/\([0-9]*\)\.\([0-9]*\)\.([0-9]*\)/\3-\2-\1/'

Convert From Unix Timestamp

date -d "1970-01-01 1234567890 sec GMT"

Calculate Last Day of Month

Found here:

cal $(date "+%M %y") | grep -v ^$ | tail -1 | sed 's/^.* \([0-9]*\)$/\1/'

Lock Files

Using “flock”:

flock /tmp/myapp.lock <some command>
flock -w 10 /tmp/myapp.lock <some command>

Using “lockfile-*” commands:

lockfile-create /tmp/myapp.lock
lockfile-touch  /tmp/myapp.lock
lockfile-remove /tmp/myapp.lock

Parameter Handling

getopt

getopt is a standalone command, supporting GNU style long parameters and parameters mixed with options and can be used like this

PARAMS=`getopt -o a::bc: --long arga::,argb,argc: -n 'myscript.sh' -- "$@"`
eval set -- "$PARAMS"

while true ; do
    case "$1" in
        -a|--arga)
            case "$2" in
                "") ARG_A='some default value' ; shift 2 ;;
                *) ARG_A=$2 ; shift 2 ;;
            esac ;;
        -b|--argb) ARG_B=1 ; shift ;;
        -c|--argc)
            case "$2" in
                "") shift 2 ;;
                *) ARG_C=$2 ; shift 2 ;;
            esac ;;
        --) shift ; break ;;
        *) echo "Unknown option!" ; exit 1 ;;
    esac
done

getopts

getopts is shell-builtin

while getopts ":ap:" opt; do
  case $opt in
    a)
      echo "Option -a ist set"
      ;;
    p)
      echo "Parameter -p is given with value '$OPTARG'"
      ;;
    \?)
      echo "Unknown option: -$OPTARG"
      ;;
  esac
done

shflags – portable getotps

If you ever need to port between different Unix derivates use shflags a Google library providing standard parameter handling. Example:

source shflags

DEFINE_string 'value' '0' 'an example value to pass with default value "0"' 'v'

FLAGS "$@" || exit $?
eval set -- "${FLAGS_ARGV}"

echo "${FLAGS_value}!"

Other Topics

Posted in Devops, Information Technology

Redis Cheat Sheet

When you encounter a Redis instance and you quickly want to learn about the setup you just need a few simple commands to peak into the setup. Of course it doesn’t hurt to look at the official full command documentation, but below is a listing just for sysadmins.

Accessing Redis

CLI

First thing to know is that you can use “telnet” (usually on Redis default port 6379)

telnet localhost 6379

or the Redis CLI client

redis-cli

to connect to Redis. The advantage of redis-cli is that you have a help interface and command line history.

CLI Queries

Here is a short list of some basic data extraction commands:

+--------------------------------------+--------------------------------------+
| Type                                 | Syntax and Explanation               |
+======================================+======================================+
| Tracing                              | Watch current live commands. Use     |
|                                      | this with care on production. Cancel |
|                                      | with Ctrl-C.                         |
|                                      |     monitor                          |
+--------------------------------------+--------------------------------------+
| Slow Queries                         |     slowlog get 25      # print top  |
|                                      | 25 slow queries                      |
|                                      |     slowlog len                      |
|                                      |     slowlog reset                    |
+--------------------------------------+--------------------------------------+
| Search Keys                          |     keys pattern        # Find key m |
|                                      | atching exactly                      |
|                                      |     keys pattern*       # Find keys  |
|                                      | matching in back                     |
|                                      |     keys *pattern*      # Find keys  |
|                                      | matching somewhere                   |
|                                      |     keys pattern*       # Find keys  |
|                                      | matching in front                    |
|                                      |                                      |
|                                      | On production servers use "KEYS"     |
|                                      | with care as it causes a full scan   |
|                                      | of all keys!                         |
+--------------------------------------+--------------------------------------+
| Generic                              |     del <key>                        |
|                                      |     dump <key>       # Serialize key |
|                                      |     exists <key>                     |
|                                      |     expire <key> <seconds>           |
+--------------------------------------+--------------------------------------+
| Scalars                              |     get <key>                        |
|                                      |     set <key> <value>                |
|                                      |     setnx <key> <value>   # Set key  |
|                                      | value only if key does not exist     |
|                                      |                                      |
|                                      | Batch commands:                      |
|                                      |     mget <key> <key> ...             |
|                                      |     mset <key> <value> <key> <value> |
|                                      |  ...                                 |
|                                      |                                      |
|                                      | Counter commands:                    |
|                                      |     incr <key>                       |
|                                      |     decr <key>                       |
+--------------------------------------+--------------------------------------+
| Lists                                |     lrange <key> <start> <stop>      |
|                                      |     lrange mylist 0 -1      # Get al |
|                                      | l of a list                          |
|                                      |     lindex mylist 5         # Get by |
|                                      |  index                               |
|                                      |     llen mylist         # Get length |
|                                      |                                      |
|                                      |     lpush mylist "value"             |
|                                      |     lpush mylist 5                   |
|                                      |     rpush mylist "value"             |
|                                      |                                      |
|                                      |     lpushx mylist 6         # Only p |
|                                      | ush in mylist exists                 |
|                                      |     rpushx mylist 0                  |
|                                      |                                      |
|                                      |     lpop mylist                      |
|                                      |     rpop mylist                      |
|                                      |                                      |
|                                      |     lrem mylist 1 "value"       # Re |
|                                      | move 'value' count times             |
|                                      |     lset mylist 2 6         # mylist |
|                                      | [2] = 6                              |
|                                      |     ltrim <key> <start> <stop>       |
+--------------------------------------+--------------------------------------+
| Hashes                               |     hexists myhash field1       # Ch |
|                                      | eck if hash key exists               |
|                                      |                                      |
|                                      |     hget myhash field1               |
|                                      |     hdel myhash field2               |
|                                      |     hset myhash field1 "value"       |
|                                      |     hsetnx myhash field1 "value"     |
|                                      |                                      |
|                                      |     hgetall myhash                   |
|                                      |     hkeys myhash                     |
|                                      |     hlen myhash                      |
|                                      |                                      |
|                                      | Batch commands:                      |
|                                      |     hmget <key> <key> ...            |
|                                      |     hmset <key> <value> <key> <value |
|                                      | > ...                                |
|                                      |                                      |
|                                      | Counter commands                     |
|                                      |     hincrby myhash field1 1          |
|                                      |     hincrby myhash field1 5          |
|                                      |     hincrby myhash field1 -1         |
|                                      |                                      |
|                                      |     hincrbrfloat myhash field2 1.123 |
|                                      | 445                                  |
+--------------------------------------+--------------------------------------+
| Sets                                 | FIXME                                |
+--------------------------------------+--------------------------------------+
| Sorted Sets                          | FIXME                                |
+--------------------------------------+--------------------------------------+

CLI Scripting

For scripting just pass commands to “redis-cli”. For example:

$ redis-cli INFO | grep connected
connected_clients:2
connected_slaves:0
$

Server Statistics

The statistics command is “INFO” and will give you an output as following.

$ redis-cli INFO
redis_version:2.2.12
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
process_id:8353
uptime_in_seconds:2592232
uptime_in_days:30
lru_clock:809325
used_cpu_sys:199.20
used_cpu_user:309.26
used_cpu_sys_children:12.04
used_cpu_user_children:1.47
connected_clients:2         # <---- connection count
connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:6596112
used_memory_human:6.29M         # <---- memory usage
used_memory_rss:17571840
mem_fragmentation_ratio:2.66
use_tcmalloc:0
loading:0
aof_enabled:0
changes_since_last_save:0
bgsave_in_progress:0
last_save_time:1371241671
bgrewriteaof_in_progress:0
total_connections_received:118
total_commands_processed:1091
expired_keys:441
evicted_keys:0
keyspace_hits:6
keyspace_misses:1070
hash_max_zipmap_entries:512
hash_max_zipmap_value:64
pubsub_channels:0
pubsub_patterns:0
vm_enabled:0
role:master             # <---- master/slave in replication setup
db0:keys=91,expires=88

Changing Runtime Configuration

The command

CONFIG GET *

gives you a list of all active configuration variables you can change. The output might look like this:

redis 127.0.0.1:6379> CONFIG GET *
 1) "dir"
 2) "/var/lib/redis"
 3) "dbfilename"
 4) "dump.rdb"
 5) "requirepass"
 6) (nil)
 7) "masterauth"
 8) (nil)
 9) "maxmemory"
10) "0"
11) "maxmemory-policy"
12) "volatile-lru"
13) "maxmemory-samples"
14) "3"
15) "timeout"
16) "300"
17) "appendonly"
18) "no"
19) "no-appendfsync-on-rewrite"
20) "no"
21) "appendfsync"
22) "everysec"              # <---- how often fsync() is called
23) "save"
24) "900 1 300 10 60 10000"     # <---- how often Redis dumps in background
25) "slave-serve-stale-data"
26) "yes"
27) "hash-max-zipmap-entries"
28) "512"
29) "hash-max-zipmap-value"
30) "64"
31) "list-max-ziplist-entries"
32) "512"
33) "list-max-ziplist-value"
34) "64"
35) "set-max-intset-entries"
36) "512"
37) "slowlog-log-slower-than"
38) "10000"
39) "slowlog-max-len"
40) "64"

Note that keys and values are alternating and you can change each key by issuing a “CONFIG SET” command like:

CONFIG SET timeout 900

Such a change will be effective instantly. When changing values consider also updating the redis configuration file.

Databases

Multiple Databases

Redis has a concept of separated namespaces called “databases”. You can select the database number you want to use with “SELECT”. By default the database with index 0 is used. So issuing

redis 127.0.0.1:6379> SELECT 1
OK
redis 127.0.0.1:6379[1]>

switches to the second database. Note how the prompt changed and now has a “[1]” to indicate the database selection. To find out how many databases there are you might want to run redis-cli from the shell:

$ redis-cli INFO | grep ^db
db0:keys=91,expires=88
db1:keys=1,expires=0

Dropping Databases

To drop the currently selected database run

FLUSHDB

to drop all databases at once run

FLUSHALL

Replication

Checking for Replication

To see if the instance is a replication slave or master issue

redis 127.0.0.1:6379> INFO
[...]
role:master

and watch for the “role” line which shows either “master” or “slave”. Starting with version 2.8 the “INFO” command also gives you per slave replication status looking like this

slave0:ip=127.0.0.1,port=6380,state=online,offset=281,lag=0

Setting up Replication

If you quickly need to set up replication just issue

SLAVEOF <IP> <port>

on a machine that you want to become slave of the given IP. It will immediately get values from the master. Note that this instance will still be writable. If you want it to be read-only change the redis config file (only available in most recent version, e.g. not on Debian). To revert the slave setting run

SLAVEOF NO ONE


Performance Testing

Benchmark

Install the Redis tools and run the provided benchmarking tool

redis-benchmark -h <host> [-p <port>]

If you are migrating from/to memcached protocol check out how to run the same benchmark for any key value store with memcached protocol.

Debugging Latency

First measure system latency on your Redis server with

redis-cli --intrinsic-latency 100

and then sample from your Redis clients with

redis-cli --latency -h <host> -p <port>

If you have problems with high latency check if transparent huge pages are disabled. Disable it with

echo never > /sys/kernel/mm/transparent_hugepage/enabled

Dump Database Backup

As Redis allows RDB database dumps in background, you can issue a dump at any time. Just run:

BGSAVE

When running this command Redis will fork and the new process will dump into the “dbfilename” configured in the Redis configuration without the original process being blocked. Of course the fork itself might cause an interruption. Use “LASTSAVE” to check when the dump file was last updated. For a simple backup solution just backup the dump file. If you need a synchronous save run “SAVE” instead of “BGSAVE”.

Listing Connections

Starting with version 2.4 you can list connections with

CLIENT LIST

and you can terminate connections with

CLIENT KILL <IP>:<port>

Monitoring Traffic

The propably most useful command compared to memcached where you need to trace network traffic is the “MONITOR” command which will dump incoming commands in real time.

redis 127.0.0.1:6379> MONITOR
OK
1371241093.375324 "monitor"
1371241109.735725 "keys" "*"
1371241152.344504 "set" "testkey" "1"
1371241165.169184 "get" "testkey"

additionally use “SLOWLOG” to track the slowest queries in an interval. For example

SLOWLOG RESET
# wait for some time
SLOWLOG GET 25

and get the 25 slowest command during this time.

Sharding with proxies

There are two major proxy solutions

  • Twemproxy (aka nutcracker, by Twitter)
  • Codis
Posted in Devops, Information Technology

Java Cheat Sheet

Heapsize calculation

You can print the effective heap size and RAM settions by using -XX:+PrintFlagsFinal. Below is an example for a 8GB host of which Java per default 1/4 (MaxRAMFraction) uses 2GB:

java -XX:+PrintFlagsFinal $MY_PARAMS -version | grep -Ei "maxheapsize|maxram"
  size_t MaxHeapSize                              = 2061500416                                {product} {ergonomic}
uint64_t MaxRAM                                   = 137438953472                           {pd product} {default}
   uintx MaxRAMFraction                           = 4                                         {product} {default}
  double MaxRAMPercentage                         = 25.000000                                 {product} {default}

Java RAM and containers

When running Java in containers you need to ensure Java see the real amount of RAM it has available. Before Java 11 it usually sees the total amount of RAM available to the host system. Basing usage on this amount often causes OOM kills.

Java Version Solution
Java <8u131 Calculate correct memory size and set using -Xms/-Xmx
Java 8u131+, Java 9 -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap
Java 10+ -XX:+UseContainerSupport -XX:MaxRAMPercentage

JDBC Problems

Oracle JDBC hanging and timing out when run on VMs: this can indicate missing entropy Workaround: Note that this reduces security! Use urandom as RNG by adding the following JVM option

-Djava.security.egd=file:///dev/urandom

Check if you are using Oracle JDK (and need a valid license)

pgrep -fl java | grep -q "+UnlockCommercialFeatures"

Default Keystore Location

readlink -e $(dirname $(readlink -e $(which keytool)))/../lib/security/cacerts

JMX Remote JConsole via SSH tunnel

Enable JMX and JConsole:

  -Dcom.sun.management.jmxremote
      -Dcom.sun.management.jmxremote.access.file=/usr/local/tomcat/conf/jmxremote.access
      -Dcom.sun.management.jmxremote.authenticate=true
      -Dcom.sun.management.jmxremote.local.only=false
      -Dcom.sun.management.jmxremote.password.file=/usr/local/tomcat/conf/jmxremote.password
      -Dcom.sun.management.jmxremote.port=3333
      -Dcom.sun.management.jmxremote.rmi.port=3334
      -Dcom.sun.management.jmxremote.ssl=false

And connect jconsole to remote localhost:3333 e.g. via SSH port forwarding

ssh targethost -L 3334:localhost:3334 -f -N
ssh targethost -L 3333:localhost:3333 -f -N

Setting defaults from environment

When you want to merge user passed settings with some defaults use JAVA_TOOL_OPTIONS. Options from the JVM CLI overrule any options also specified in JAVA_TOOL_OPTIONS.

Posted in Devops, Information Technology

MySQL Cheat Sheet

Below you find a unordered list of solutions by tasks useful for a MySQL DBA:

Live Monitoring of MySQL

There are two useful tools:

  • mytop
  • innotop

with “mytop” being an own Debian package, while “innotop” is included in the “mysql-client” package. From both innotop has the more advanced functionality. Both need to be called with credentials to connect to the database:

mytop -u <user> -p<password>
innotop -u <user> -p<password>

Alternatively you can provide a .mytop file to provide the credentials automatically.

Show MySQL Status

You can get a very simple status by just entering “\s” in the “mysql” command line client prompt:

mysql> \s

You can show the replication status using

SHOW SLAVE STATUS \G
SHOW MASTER STATUS \G

Note that the “\G” instead of “;” just makes the output more readable. If you have configured slaves to report names you can list them on the master with:

SHOW SLAVE HOSTS;

Check InnoDB status

show /*!50000 ENGINE*/ INNODB STATUS;

List Databases/Tables/Colums

You can either use the “mysqlshow” tool:

mysqlshow                         # List all databases
mysqlshow <database>              # List all tables of the given database
mysqlshow <database> <table>      # List all columns of the given table in the given DB

And you can also do it using queries:

SHOW DATABASES;

USE <database>;
SHOW TABLES;
DESCRIBE <table>;

Check and Change Live Configuration Parameters

Note that you cannot change all existing parameters. Some like innodb_pool_buffer require a DB restart.

show variables;                          # List all configuration settings
show variables like 'key_buffer_size';   # List a specific parameter

set global key_buffer_size=100000000;    # Set a specific parameter

# Finally ensure to edit my.cnf to make the change persistent

MySQL Parameter Optimization

You can check MySQL parameters of a running instance using tools like

Also have a look at this MySQL config parameter explanation.

Remote MySQL Dump and Import

The following command allows dumping a database from one source host that doesn’t see the target host when executed on a third host that can access both. If both hosts can see each other and one has SSH access to the other you can simply drop one of the ssh calls.

ssh <user@source host> "mysqldump --single-transaction -u root --password=<DB root pwd> <DB name>" | ssh <user@target host> "mysql -u root --password=<DB root pwd> <DB name>"

Troubleshooting

Connection refused

mysqladmin flush-hosts

How to solve: Could not find target log during relay log initialization

Happens on corrupted/missing relay logs. To get the DB working

  • Stop MySQL
  • Remove /var/lib/mysql/relay-log-index.*
  • Remove all relay log files
  • Remove relog log file index
  • Start MySQL

mysqldump: Error 2013: Lost connection to MySQL server during query when dumping table

This is caused by timeouts when copying overly large database tables. The default network timeouts are very short per-default. So you can workaround this by increasing network timeouts

set global net_write_timeout = 100000;
set global net_read_timeout = 100000;

Dump Skip Event Table

If your MySQL backup tool or self-written script complains about an event table than you have run into an issue caused by newer MySQL versions (>5.5.30) that introduced a new table “events” in the internal schema. If you run into this you need to decide wether you want to include or exclude the new events table when dumping your database. To skip: Due to a MySQL bug #68376 you have two choices. You can check documentation and add the logical option

--skip-events

which will cause the event table not to be exported. But the warning won’t go away. To also get rid of the warning you need to use this instead:

--events --ignore-table=mysql.events

And of course you can also choose just to dump the events table: Add the option

--events

to your “mysqldump” invocation. If you use a tool that invokes “mysqldump” indirectly check if the tool allows to inject additional parameters.

Forgotten root Password

# 1. Stop MySQL and start without grant checks

/usr/bin/mysqld_safe --skip-grant-tables &
mysql --user=root mysql

# 2. Change root password
UPDATE user SET password=PASSWORD('xxxxx') WHERE user = 'root';

Import a CSV file into MySQL

LOAD DATA IN '<CSV filename>' INTO TABLE <table name> FIELDS TERMINATED BY ',' (<name of column #1>,<<name of column #2>,<...>);

MySQL Pager – Output Handling

Using “PAGER” or \P you can control output handling. Instead of having 10k lines scrolling by you can write everything to a file or use “less” to scroll through it for example. To use less issue

pager less

Page output into a script

pager /home/joe/myscript.sh

Or if you have Percona installed get a tree-like “EXPLAIN” output with

pager mk-visual-explain

and then run the “EXPLAIN” query.

MySQL – Check Query Cache

# Check if enabled
SHOW VARIABLES LIKE 'have_query_cache';

# Statistics
SHOW STATUS LIKE 'Qcache%';

Check for currently running MySQL queries

show processlist;
show full processlist;

Filter items in process list by setting grep as a pager. The following example will only print replication connections:

\P grep system
show processlist;

Find top long running queries

SELECT id,host,state,command,time,left(replace(info,'\n','<lf>'),120)
FROM information_schema.processlist
WHERE command <> 'Sleep' 
AND info NOT LIKE '%PROCESSLIST%'
ORDER BY time DESC LIMIT 25;

To abort/terminate a statement determine it’s id and kill it:

kill <id>;    # Kill running queries by id from process listing

Show Recent Commands

SHOW BINLOG EVENTS;
SHOW BINLOG EVENTS IN '<some bin file name>';

Inspect a MySQL binlog file

There is an extra tool to inspect bin logs:

mysqlbinlog <binary log file>

Skip one statement on replication issue HA_ERR_FOUND_DUPP_KEY

If replication stops with “HA_ERR_FOUND_DUPP_KEY” you can skip the current statement and continue with the next one by running:

STOP SLAVE;
 SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1;
START SLAVE;

Changing Replication Format

When you want to change the replication format of a running setup you might want to follow this steps:

  1. Ensure you have a database backup
  2. Make master read-only by running
    FLUSH TABLES WITH READ LOCK;
    
  3. Wait until all slaves do catch up
  4. Stop all slaves (shutdown MySQL)
  5. On master:
    FLUSH LOGS;
    SET GLOBAL binlog_format='xxxxxx';
    FLUSH LOGS;
    UNLOCK TABLES;
    

    (ensure to replace ‘xxxxxx’ with for example ‘ROW’)

  6. Start all slaves
  7. Ensure to put the new binlog_format in all /etc/mysql/my.cnf

Note: the second “FLUSH LOGS;” ensures that the a new binary log is opened on the master with the new binlog_format. The stopping of the slaves ensures that they open a new relay log matching the new binlog_format.

Munin MySQL Plugin Setup on Debian

apt-get install libcache-cache-perl

for i in `./mysql_ suggest`
do
   do ln -sf /usr/share/munin/plugins/mysql_ $i;
done

/etc/init.d/munin-node reload

Fix Slow Replication

When replication is slow check the status of the replication connection. If it is too often in “invalidating query cache” status you need to decrease your query cache size. You might even consider disabling query cache for the moment if the DB load does allow it:

set global query_cache_size=0;

Debug DB Response Time

There is generic TCP response analysis tool developed by Percona called tcprstat. Download the binary from Percona, make it executable and run it like

tcprstat -p 3306 -t 1 -n 0

to get continuous statistics on the response time. This is helpful each time some developer claims the DB doesn’t respond fast enough!

 

Posted in Devops, Information Technology

Kafka Cheat Sheet

CLI Commands for Kafka Topics

List existing topics

/usr/bin/kafka-topics --zookeeper <zookeeper host:port> --list

Create a new topic

/usr/bin/kafka-topics --create --zookeeper <zookeeper host:port>/kafka --replication-factor 2 --partitions 4 --topic <name>

Try to read from a topic

/usr/bin/kafka-console-consumer --zookeeper  <zookeeper host:port> --topic <name>

Try to write to a topic on one or more brokers

echo "Test data" | /usr/bin/kafka-console-producer --broker-list <broker1 host:port>,<broker2 host:port> --topic <name>

Checking Kafka Consumers

Show consumer group configuration

/usr/bin/kafka-consumer-groups --zookeeper <zookeeper host:port> --describe --group <group name>

To read messages read/written including lag per consumer in a consumer group

/usr/bin/kafka-consumer-offset-checker --group <group name> --topic <name> --zookeeper <zookeeper host:port>

Syslog to Kafka

Monitoring Tools