What is the name of this Blog?

instead of notepad

Oracle JDK 7 on CentOS

When it comes to which java to use, Hadoop is a bit picky. It prefers the Oracle jdk. See the list of apache tested JDKs.

Lately all the google references which use wget or curl are broken.

Silent install

For the impatient here is the one-liner.

1
2
3
curl -LO 'http://download.oracle.com/otn-pub/java/jdk/7u51-b13/jdk-7u51-linux-x64.rpm' \
-H 'Cookie: oraclelicense=accept-securebackup-cookie'
rpm -i jdk-7u51-linux-x64.rpm

or if you really want it in one single line

1
curl -LO 'http://download.oracle.com/otn-pub/java/jdk/7u51-b13/jdk-7u51-linux-x64.rpm' -H 'Cookie: oraclelicense=accept-securebackup-cookie' && rpm -i jdk-7u51-linux-x64.rpm

tl;dr

What I did was: - opened Chrome Developer Toolbar - went to the official oracle download page - clicked Accept License Agreement - clicked on jdk-7u51-linux-x64.rpm - as the download has started, I switched to the Network tab of the Developer Toolbar, and right-clicked on the rpm, and choos Copy as cURL

Voila! Of cours it was a huge line, but it can be trimmed down to just one singe required cookie: oraclelicense=accept-securebackup-cookie

Meanwhile it appeared on stackowerflow with a fatty +50 reputation bounty.

Fix a Broken Mac Terminal

Sometimes my Terminal gets broken by either displaying binary files, or sometimes caused by a broken ssh connection.

You can fix it by copy pasting the following line:

1
2
3
tput init
tput reset
reset

You might not see the charaters even after pasted to the terminal, but keep on press ENTER, and suddenly your terminal is back to normal.

Install Docker Binary

Docker can be installed as binary from get.docker.io as described in the docs: Get the docker binary

Linux

1
2
wget https://get.docker.io/builds/Linux/x86_64/docker-latest -O docker
chmod +x docker

Mac

The mac binary installation is described in a different documentation page: Docker OS X Client

1
2
curl -o docker https://get.docker.io/builds/Darwin/x86_64/docker-latest
chmod +x docker

Access Docker Containers via Internal Ip

Whenn you run a lot of containers inside of a docker host, you can get lost of the forwarded ports. Let’s say you have 3 instances all exposing 3 ports: 22, 80 and 443. Docker will automatically assign ‘random’ port numbers like:

1
2
3
4
ID            ...  PORTS
656eb7831dda  ...  49162->22, 49163->80, 49164->443   
ff2141eb218d  ...  49159->22, 49160->80, 49161->443   
54b9dfd5400c  ...  49156->22, 49157->80, 49158->443   

Now its hard to remember all those ports, and is confusing that if you want to ssh into the first one, you use ssh root@docker -p 49162 and in the browser you would use: http://docker:49163.

There is an even bigger problem with random ports. Wehn your http server sometimes redirects you to https, you need the standard port numbers. The webserver running inside of a docker container, will redirect you to port 443 which will not be correct port (49164 in our case)

So it would be much natural for the docker instances to use their ip address, with straight port numbers. But those ip addresses are only valid inside of the docker host. How to do the magic?

SSH is yor best friend

There is complete SOCKS proxy hidden indide of ssh. To get it alive you just have to toss a -D 1099 parameter, and boom, it will listen on your localhost’s 1099 port and act as a proxy. So here is a how you create an ssh based tunnel :

1
ssh -qTfN2 -D 1099 docker

For the curious the parameter meanings:

  • q :- be very quite
  • T :- Do not allocate a pseudo tty
  • f :- move the ssh process to background
  • N :- Do not execute remote command.
  • 2 :- Forces ssh to try protocol version 2 only.

docker internal IPs

How can you get the internal ip of a container? You can get all the dirty details by: sudo docker inspect XXXXX. For the moment the only interresting this is the IP addresses. So the oneliner is:

1
for i in $(sudo docker ps -q); do sudo docker inspect $i| grep IPA; done

it will create the following list:

1
2
3
    "IPAddress": "172.17.0.12",
    "IPAddress": "172.17.0.11",
    "IPAddress": "172.17.0.10",

SOCKS proxy

Now all the programs which can use a SOCKS proxy can access the docker containers. For example let’s poke the first containers webserver with curl

1
curl --proxy socks5h://localhost:1099  172.17.0.10

for a java process ypu can specify it by system properties: -DsocksProxyHost=127.0.0.1 -DsocksProxyPort=1099

SOCKS proxy for the browser

You have probably already uses an http proxy in your browser. In that case all trafic goes throught the http proxy. For our usecase what we wan is:

  • accessing docker containers via SOCKS proxy
  • all other shoud be accessed directly

Luckily thus problem is already solved by the Proxy auto-config or shortly PAC. Its a JavaScript function which returns with the connection type: PROXY/SOCKS/DIRECT. See the sample below:

1
2
3
4
5
6
7
8
9
10
11
function FindProxyForURL(url, host) {
 
        // URLs within this network are accessed through
        // socks proxy provided by ssh -D
        if (isInNet(host, "172.17.0.0", "255.255.0.0"))
        {
                return "SOCKS 127.0.0.1:1099";
        }
 
        return "DIRECT";
}

SSH doesn’t need a SOCKS proxy

We use ssh to create a SOCKS proxy, which can be used by other programs java/browser/curl and so on. But ssh is clever enough that if there is no SOCKS proxy opened, it can create it on demand:

1
ssh -o ProxyCommand='ssh user@10.253.129.151 nc %h %p' user@targetsshserver.example.com

This tells ssh that instead of connecting directly to targetsshserver.example.com’s port 22, use a ProxyCommand to create the connection.

In this case the command itself is an ssh: ssh user@docker nc %h %p which means start a NetCat (nc host port) via ssh.

Debug Ansible

If you reaaly want to see the ctual scripts ansible is running use the ANSIBLE_KEEP_REMOTE_FILES=1 to prevent ansible from deleting remote files.

Fixing Startup on Docker

When working in docker container, you need to start services differently. Let’s say you want to start mysql. You have 3 choices:

  • mysqld_safe & which is a quick and dirty solution but its not idempotent.
  • /etc/init.d/mysql restart its the old school way
  • service mysql start that’s the new school way

Normally you would prefer the 3. option service mysql start but it needs upstart running which is normally the /sbin/init process with the special pid of 1.

Unfortunately inside of a docker instance the special 1 pid is rserved for the command started by docker. In most case its either /bin/bash, or sshd, ot whatever service you are running in the container.

when you try to start mysql with then you will most probably face the following message:

1
 Failed to connect to socket /com/ubuntu/upstart: Connection refused

thats because upstart is not running, or phrasing differently: 1 pid is not /sbin/init

upstart workaround

There is a workaround for supressing this messages:

1
2
dpkg-divert --local --rename --add /sbin/initctl
ln -s /bin/true /sbin/initctl

Of course you stil can’t use service mysql start, but at least you wont get those error messages.

related issues / pull-reqs

I can see that the docker issues/pull requests are disscussing it. But im still not sure the proper way to go. But here are the references:

  • issue 223: the original Run /sbin/init issue
  • issue 1024: the Container cannot connect to Upstart issue where i found the workaround
  • issue 1311 : Production-ready process monitoring
  • pull-req 898: Inject dockerinit at /.dockerinit instead of overwriting /sbin/init
  • pull-req 1209: Upstart improvements
  • pull-req 1883: /usr/sbin/policy-rc.d is added since the vast majority of docker users are surprised when they apt-get install or apt-get upgrade and services fail to start and that sometimes makes the install fail

update

I just read the docker integration into supervisor-s () is planned for the upcomming 0.7 version

For docker to be fully usable in production, it needs to cleanly integrate with the host machine’s process supervisor of choice. Whether it’s sysV-init, upstart, systemd, runit or supervisord, we want to make sure docker plays nice with your existing system. This will be a major focus of the 0.7 release.

Install Sshpass on Mac

I’m experimenting with ansibe, an awesome IT orchestration engine. It deservse its own blog entry, but in short: It’s an alternative of puppet and chef, but its much easier to learn, and dowsn’t require any client side agent installation.

passwordless ssh

Its higly recommended to use public key authentication, but sometimes you have to deal with passwords. Let’s say a fresh virtual environment is always created with a default password.

When I tried to call ansible with -k password, meaning ask for password, i faced the following message:

1
to use the 'ssh' connection type with passwords, you must install the sshpass program

installing sshpass

Installing sshpass on linux is as easy as calling apt-get or yum, whatever package manager its using. Unfortunately homebrwe, my favourite package manager for mac doesn’t supports this package officially.

Actually it was created, but the team refused to take it as its a bad practice to put passwords into scripts.

However there is an unofficial brew package:

1
brew install https://raw.github.com/eugeneoden/homebrew/eca9de1/Library/Formula/sshpass.rb

Spring RooBot Workaround

I’m doing all sorts of trainings inside Epam, and one of them is Spring Roo. Roo is a rapid application development tool, you can create maven based SpringMVC applications which serves hibernate JPA entities backed by mysql It’s not limited to this framework mixture, although this is one of the most popular.

Spring Roo is modular, you can install new features with the install addon command. The actual list of addons is collected by a process called RooBot. For some reason RooBot is unable to di his work since late June. Some explanation can be found at the ROO-3184 Jira Issue.

When you try to install any addon, or even list them you get the folowing:

1
2
roo> addon list 
No add-ons known. Are you online? Try the 'download status' command

For the impatient

when you see that, just simple type the following (using the symbolic name of your missing addon)

1
2
3
4
osgi obr url add --url http://spring-roo-repository.springsource.org/repository.xml

osgi obr deploy --bundleSymbolicName org.springframework.roo.addon.cloud.foundry
osgi obr deploy --bundleSymbolicName org.springframework.roo.addon.git

if you are interested in the whole story just read on …

Reload Tiles Definitions in SpringMVC

If you have a springMVC application, normally you want some layout templating framewrok so you don’t have to repeat header/menu/footer in each jsp. One of the most popular choice is tiles. Spring roo generates mvc apps use them as well.

While developing you often want to create a new page, which needs a new tiles definition in one of the tiles definition xml (views.xml). But the application needs a redeploy to se those changes.

For impatiens

If you don’t care about the explanation, here is the solution: add 2 line to your webmvc-config.xml

changes needed in webmvc-config.xml
1
2
3
4
5
6
7
 <bean class="org.springframework.web.servlet.view.tiles2.TilesConfigurer"
       id="tilesConfigurer" >
+  <property name="useMutableTilesContainer">true</property>
+  <property name="checkRefresh">true</property>
   <property name="definitions">
     <list>
       <value>/WEB-INF/layouts/layouts.xml</value>

If you do care about details read on …

Starting Tomcat in Debug Mode From Maven

If you want to start tomcat from maven you just simply run:

1
mvn tomcat:run

if you want to run it in debug mode, in order to be able to remote debug it, set the MAVEN_OPTS to:

1
export MAVEN_OPTS="-Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9000"

Connect from Eclipse

To connect to this process from eclipse/sts open the debug/debug configurations dialog, and choose: Remote Java Application (even if it can be misleading as its running on localhost)