hbase not-so-quick start

I wanted to play with Apache HBase so I downloaded v0.94.2 to a ubuntu VirtualBox and followed the quick start.  But it didn’t start at all.  The log file had exceptions similar to the following:

2012-11-15 09:37:28,728 INFO org.apache.hadoop.ipc.HBaseRPC: Server at localhost/ could not be reached after 1 tries, giving up.
 2012-11-15 09:37:28,732 WARN org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of -ROOT-,,0.70236052 to localhost,40408,1352990244709, trying to assign elsewhere instead; retry=0
 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy interface org.apache.hadoop.hbase.ipc.HRegionInterface to localhost/ after attempts=1
 at org.apache.hadoop.hbase.ipc.HBaseRPC.handleConnectionException(HBaseRPC.java:291)
 at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:259)
 at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1313)
 at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1269)
 at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1256)
 at org.apache.hadoop.hbase.master.ServerManager.getServerConnection(ServerManager.java:550)
 at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:483)
 at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1640)
 at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1363)
 at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1338)
 at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1333)
 at org.apache.hadoop.hbase.master.AssignmentManager.assignRoot(AssignmentManager.java:2212)
 at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:632)
 at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:529)
 at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:344)
 at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.run(HMasterCommandLine.java:220)
 at java.lang.Thread.run(Thread.java:722)
 Caused by: java.net.ConnectException: Connection refused
 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
 at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
 at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
 at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupConnection(HBaseClient.java:416)
 at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:462)
 at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1150)
 at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1000)
 at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
 at $Proxy12.getProtocolVersion(Unknown Source)
 at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:183)
 at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:335)
 at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:312)
 at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:364)
 at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236)
 ... 15 more
 2012-11-15 09:37:28,732 WARN org.apache.hadoop.hbase.master.AssignmentManager: Unable to find a viable location to assign region -ROOT-,,0.70236052

After trying a few different things, I found this blog post.  The post was using Cloudera releases, but the symptoms were very similar to what I experienced.  It presented two approaches to solving the issue — either commenting out the “ COMPNAME” line in /etc/hosts, or adding a property “-Djava.net.preferIPv4Stack=true” to HADOOP_OPTS in hadoop-env.sh.  I tried the first approach by commenting out all the COMPNAME lines in /etc/hosts.  In my case, there were 2 such lines, one resolved to localhost and the other resolved to evabuntu (the name I gave to my virtual machine).

When I tried to start HBase again, I got a different exception:

2012-11-15 10:06:22,062 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMasterevabuntu
at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:134)
at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:197)
at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:147)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:140)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1806)
Caused by: java.net.UnknownHostException: evabuntu: evabuntu
at java.net.InetAddress.getLocalHost(InetAddress.java:1438)
at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:185)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:241)
at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.<init>(HMasterCommandLine.java:215)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:131)
... 7 more
Caused by: java.net.UnknownHostException: evabuntu
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:866)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1258)
at java.net.InetAddress.getLocalHost(InetAddress.java:1434)
... 15 more

This time, the exception was much easier to understand.  HBase picked up the name “evabuntu” from /etc/hostname but wasn’t able to resolve the IPv6 because there was no entry for it in /etc/hosts.  So all I had to do was add an IPv6 entry for it in /etc/hosts.

First use ifconfig to find the IPv6

> /sbin/ifconfig

eth0      Link encap:Ethernet  HWaddr 08:00:27:f4:8f:83
inet addr:  Bcast:  Mask:
inet6 addr: fe80::a00:27ff:fef4:8f83/64 Scope:Link
RX packets:66784 errors:0 dropped:0 overruns:0 frame:0
TX packets:38494 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:86804874 (86.8 MB)  TX bytes:2845363 (2.8 MB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:df:4d:e0
inet addr:  Bcast:  Mask:
inet6 addr: fe80::a00:27ff:fedf:4de0/64 Scope:Link
RX packets:93 errors:0 dropped:0 overruns:0 frame:0
TX packets:92 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:11166 (11.1 KB)  TX bytes:12582 (12.5 KB)

lo        Link encap:Local Loopback
inet addr:  Mask:
inet6 addr: ::1/128 Scope:Host
RX packets:9652 errors:0 dropped:0 overruns:0 frame:0
TX packets:9652 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:971333 (971.3 KB)  TX bytes:971333 (971.3 KB)

in my case, I wanted to use eth1, but you may pick the IPv6 from any interface.  So I copied and pasted the inet6 address fe80::a00:27ff:fedf:4de0 to /etc/hosts

> cat /etc/hosts

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
fe80::a00:27ff:fedf:4de0 evabuntu

and HBase started.

Then I tried the second approach outlined in that blog post, which was to force Java to use IPv4.  The example in the post added the property to HADOOP_OPTS in conf/hadoop-env.sh, but in the case of Apache HBase, it needed to be added to HBASE_OPTS in conf/hbase-env.sh.  But this alone wasn’t enough because as I mentioned earlier, the /etc/hostname says evabuntu, so I also needed to add an entry for evabuntu in /etc/hosts.  This would be the same drill as above, except this time, you pick out the IPv4 assigned to each network interface instead of IPv6.



Step by step instructions on self-signed certificate and Tomcat over SSL

Creating a self-signed certificate to test Tomcat https is easy, and this article gives you the step-by-step instructions on the following parts:

1. Create a self-signed host certificate using openSSL

2. Create a PKCS12 keystore and convert it to java keystore

3. Configure Tomcat 6 to use https, and redirect http to https

4. Create a Java client to talk to the Tomcat over SSL with the self-signed certificates

Part 1.  Create a self-signed host certificate using openSSL

There are different ways of creating a self-signed certificate, such as using Java keytool.  But I prefer openSSL because the keys and certificates generated this way are more standardized and can be used for other purposes.  The openSSL HOWTO page gives you a lot of details and other information.

1.1 Create a pair of PKI keys

PKI stands for Public Key Infrastructure, which is also known as Asymmetric key pair, where you have a private key and a public key.  The private key is a secret you guard with your honor and life, and the public key is something you give out freely.  Messages encrypted with one can be decrypted with the other.  While generally speaking, given one key, it should be infeasible to derive the other.  However, openSSL makes it so that given a private key, you can easily derive the public key (but not vice versa, otherwise the security is broken).  For this reason, when you generate a key using openSSL, it only gives you a private key.

As a side note, the word asymmetric is really a poor choice.  Once, a security expert was giving a presentation to a roomful of students on PKI, and one of his slides was supposed to have the title “Asymmetric key scheme”, but perhaps it was the fonts he used, or perhaps he made a last-minute typo,  it looked like there was a space between the letter “A” and the rest of the letter.  After that presentation, quite a few naive students began to think that  PKI is a symmetric (WRONG!) key scheme where it should be exactly the opposite — this is probably a less forgivable mistake than blowing up the chemistry lab because someone thinks inflammable means not flammable.

1.1.1 Create a host private key using openSSL

openssl genrsa -out HOSTNAME-private.pem 2048

This private key is 2048 bits long, generated using RSA algorithm, and we choose not to protect it with an additional passphrase because the key will be used with a server certificate.  The name of the private key is HOSTNAME-private.pem where HOSTNAME should be replaced by the name of the machine you intend to host Tomcat.

1.1.2 Derive the public key using openSSL.  This step is not necessary, unless  you want to distribute the public key to others.

openssl rsa -in HOSTNAME-private.pem -pubout  > HOSTNAME-public.pem

1.2 Create a self-signed X509 certificate

openssl req -new -x509 -key HOSTNAME-private.pem -out HOSTNAME-certificate.pem -days 365

Then you will be prompted to enter a few pieces of information, use “.” if you wish to leave the field blank

Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Indiana
Locality Name (eg, city) []:Bloomington
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Cool Org
Organizational Unit Name (eg, section) []:Cool IT
Common Name (eg, YOUR name) []:Cool Node
Email Address []:.

You will now see your host certificate file HOSTNAME-certificate.pem

UPDATE: The field Common Name is quite important here.  It is the hostname of the machine you are trying to certify with the certificate, which is the name in the DNS entry corresponding to your machine IP.

If your machine does not have a valid DNS entry (in other words, doing a nslookup on the IP of your machine doesn’t give you anything), the host certificate probably won’t work too well for you.  If you are only doing some very minimalistic https connection using only the HttpsURLConnection provided by Java, you can probably get by by disabling the certificate validation as outline towards the end of this article; however, if you use other third-party software packages, you will probably get an exception look like the following:

java.io.IOException: HTTPS hostname wrong:  should be <xxx.yyy.zzz>

This is because many security packages would check for things such as URL Spoofing, and when they do a reverse lookup of the machine IP,  but do not yield the same hostname as what is in the certificate, they think something is fishy and throws the exception.

Part 2. Create a PKCS12 keystore and convert it to a Java keystore

Java keytool does not allow the direct import of x509 certificates with an existing private key, and here is a Java import key utility Agent Bob created to get around that.  However, we can still get it to work even without this utility.  The trick is to import the certificate into a PKCS12 keystore, which Java keytool also supports, and then convert it to the Java keystore format

2.1 Create a PKCS12 keystore and import (or export depending on how you look at it) the host certificate we just created

openssl pkcs12 -export -out keystore.pkcs12 -in HOSTNAME-certificate.pem -inkey HOSTNAME-private.pem

It will ask you for the export password, and it is recommended to provide a password.

2.2 Convert the PKCS12 keystore to Java keystore using Java keytool.

keytool -importkeystore -srckeystore keystore.pkcs12 -srcstoretype PKCS12 -destkeystore keystore.jks -deststoretype JKS

Keytool will first ask you for the new password for the JKS keystore twice, and it will also ask you for the password you set for the PKCS12 keystore created earlier.

Enter destination keystore password:
Re-enter new password:
Enter source keystore password:
Entry for alias 1 successfully imported.
Import command completed: 1 entries successfully imported, 0 entries failed or cancelled

It will output the number of entries successfully imported, failed, and cancelled.  If nothing went wrong, you should have another keystore file: keystore.jks

Part 3. Configure Tomcat to use HTTPS

With the keystore in place, we can now configure Tomcat to communicate via SSL using the certificate.

3.1 Configure Tomcat HTTPS Connector.

Edit CATALINA_HOME/conf/server.xml, where CATALINA_HOME is the base directory of Tomcat.  By default, the HTTPS Connector configuration is commented out.  We can search for “8443” which is the default port number for HTTPS connector, and then either replace the configuration block, or add another block just below.  We are going to use the Coyote blocking connector:

 <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
 maxThreads="150" scheme="https" secure="true"
 clientAuth="false" sslProtocol="TLS" />

<Connector port="8443" protocol="org.apache.coyote.http11.Http11Protocol" SSLEnabled="true" maxThreads="150" secure="true" scheme="https" keystoreFile="PATH/TO/keystore.jks" keystorePass="JKS_KEYSTORE_PASSWORD" clientAuth="false" sslProtocol="TLS" />

In the snippet above, PATH/TO/keystore.jks is the path to the Java Keystore we created earlier, and I recommend using the absolute path to eliminate any confusion.  Also provide the keystore password – it is in plain text, so protect server.xml using the correct permission (700).

The Tomcat SSL configuration instruction is a bit misleading and may let us believe both blocking and non-blocking should be configured.  This is not true because the port number can only be used by one connector type.

This configuration enables Tomcat to communicate HTTPS on port 8443.  At this point, it is a good idea to fire up Tomcat and make sure the configuration works using a web browser.


And point your web browser to https://HOSTNAME:8443 to see if Tomcat’s front page shows up.  Since we are using a self-signed certificate, your browser may complain about the certificate being not secure.  Accept the certificate so your browser can display the page.

3.2 Configure Tomcat to redirect HTTP to HTTPS

However, so far, Tomcat still supports HTTP (default port is 8443, but it may have been changed in your situation).  It would be desirable to automatically redirect any requests to the HTTP over to the HTTPS.  The first thing to do is edit CATALINA_HOME/conf/server.xml again, and this time, locate the Connector configuration for HTTP, and modify it so that the “redirectPort” attribute points to the HTTPS port (8443 by default).

<Connector port="8080" protocol="HTTP/1.1"
 redirectPort="8443" />

Now save server.xml, and edit web.xml, and add the following block to the end of the file, just before the </web-app> tag (in other words, the security-constraint section must be added AFTER the servlet-mapping sections:

<web-app ...>


      <web-resource-name>All Apps</web-resource-name>

Save this file, restart Tomcat again. This time, open a browser and enter the URL to the normal HTTP port, and see if Tomcat redirects to the HTTPS port.

Part 4. Create a test Java client to talk to Tomcat over SSL

Since we created our own self-signed certificate, if we just use a Java HttpsURLConnection client trying to connect to the Tomcat over SSL, it will not honor the certificate and throw an exception like the following:

Exception in thread "main" javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1731)
 at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:241)
 at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:235)
 at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1206)
 at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:136)
 at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:593)
 at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:529)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:925)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1170)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1197)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1181)
 at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:434)
 at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:166)
 at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:133)
 at test.SClient.main(SClient.java:97)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:323)
 at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:217)
 at sun.security.validator.Validator.validate(Validator.java:218)
 at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:126)
 at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:209)
 at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:249)
 at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1185)
 ... 11 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:174)
 at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:238)
 at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:318)
 ... 17 more

To circumvent this problem, we should disable to the certificate validation in the client so we can move forward with the testing.  Add the following block of code in your Java client before creating any HttpsURLConnection (credits of this block of code goes to http://www.exampledepot.com/egs/javax.net.ssl/trustall.html:

// this block of code turns off the certificate validation so the client can talk to an SSL
// server that uses a self-signed certificate
// !!!! WARNING make sure NOT to do this against a production site
// this block of code owes thanks to http://www.exampledepot.com/egs/javax.net.ssl/trustall.html

TrustManager[] trustAllCerts = new TrustManager[] {
    new X509TrustManager() {
        public java.security.cert.X509Certificate[] getAcceptedIssuers() {
            return null;

        public void checkClientTrusted(java.security.cert.X509Certificate[] certs, String authType){}

        public void checkServerTrusted(java.security.cert.X509Certificate[] certs, String authType){}

SSLContext sslContext = SSLContext.getInstance("SSL");
sslContext.init(null, trustAllCerts, new java.security.SecureRandom());

// end of block of code that turns off certificate validation
// ////////////////////////////////////////////////////////////////////////////////////