Site Sponsors:
Hadoop: Outloading Support for User-Defined Functions in Java 
So there we are, trying to locate the Java-Archive (JAR) files required for creating a User-Defined Function (UDF) for Hive. (Pig works much the same.)

Perhaps one is even using the Hortornworks Sandbox. No matter how we access Hadoop's Distributed File System (HDFS) however, many have broken a sword or three trying to locate the proper jar-file support.

Shell Games...

One of the reasons for the breakage is that we folks in the Java world will insist on packaging classes in different archives over time. -A worthy rant in itself, the sad truth is that - while classes and packages will remain the same - wee Java folks always need to scramble things about so as to make them logically easier to find.

Speaking as someone who has been known to migrate a class between archives from time to time, perhaps a JSR-someone needs to create what might be a simple meta-tracking system - a packaging idea well outside of git, and perhaps a tag closer to @Depricated.

The need is to help us keep our sanity when playing the class shell game.... but I digress!


Yes, I needed a new term, so I coined one: Unlike "uploading" or "downloading," the idea here is that we need to post something from INSIDE the system, so as to be available from OUTside of the same... so we can downLOAD it! (Remember the term "DLL Hell?" -I needed a new term when writing for BYTE Magazine in 1992, as well! =)

While the Internet has worked so as to make locating properly-named jar files allot easier now than it was back in 1992... it still ain't easy. Because the exact same package and class can migrate between archives so much in Java, maybe we should call the little "shell game" we all must eventually play "Jar Hell"?

So whilst we have used "hadoop-core" -> "hadoop-common" on HUE (HDP 2) -> Ambari (HDP 2.4), after the requesite Internet search (sigh) I have discovered that the best way to locate the Jar we need (i.e. nowhere else to be found!) would be to use a simple find command on the target platform, itself. --The JAR we need to create UDF's is simply to be found no-where else!


On the present incarnation of the Hortonworks VM (again, the truly marvelous HDP 2.4 - Gotta love Ambari!) here is how to locate + post the archive(s) we need to create UDF's ... to the HDFS:

find / -type f -name "hive-exec*core.jar" -exec hadoop fs -put "{}" hdfs:/hadoop9000 ";"

(Please note that I obviously had created /hadoop9000 in the HDFS beforehand...) Ultimately, note also that the default incarnation of the fs command will not replace an existing file.

A similar exercise will also work (if required) so as to outload "hadoop-common" (or hadoop-core!) via HDFS.

Enjoy the Journey!


[ view entry ] ( 784 views )   |  permalink
Cloud, Client, & Browser Concepts 
For those who are entering the software and / or software testing world, here is an excerpt from our new "Testing Success" training offering.

I have been informed that there are even some new ideas for experienced software developers / testers, as well ... :)



[ view entry ] ( 604 views )   |  permalink  |  related link
CentOS, Tomcat, MySQL, and Java Hosting 
NOTE: GODADDY Cloud Servers GONE as of December 31, 2017

In as much as it is now extremely affordable to toss server installs around the cloud, we decided to set up a Linux, Tomcat & MySQL (LTM) stack on CentOS.

Bitter Experiences

In an earlier post I mentioned that - while capable - that AWS was a box of frogs. While we might wonder aloud as we hear-tell of what is bumping around inside, once the 'pandora has been braved one will surely crave a far more native-English way of getting things done!

So it was with no little trepidation that I created a series of cloud servers on GoDaddy. While the prohibition on the static IP address remains, just like when creating cloud-servers elsewhere there are indeed DNS work-arounds.

Java Web Hosting

Setting up your first LTM stack can be a bit of a challenge.

Indeed, if RedHat is your favorite flavor of Linux, even lots of old-hands have been challenged by the server-management changes between CentOS 5, CentOS 6, & CentOS 7.

So while using Ubuntu on Godaddy's OpenStack cloud is allot easier, by way of documentation I also created a CentOS 7 based cloud server today.

Executive Summary

Please allow me to report that - for a mere $5 a month - that from starting & stopping to using SSH to access our Servers, that GoDaddy's cloud experience is presently far, far easier for software developers to work with. So easy that I feel no need to document it!

Once we have installed & updated a CentOS 7 Server - either in the cloud or elsewhere - installing the SCL is usually a solid place to start any LTM-installation process:
    sudo bash
yum update
yum install centos-release-SCL
yum update

rpm -Uvh
yum -y install mysql-community-server
yum update
/usr/bin/systemctl enable mysqld
/usr/bin/systemctl start mysqld

yum -y install tomcat tomcat-webapps tomcat-admin-webapps
yum update
/usr/bin/systemctl enable tomcat
/usr/bin/systemctl start tomcat
If you are familiar with using CentOS 6 or earlier, then after typing in the above it might seem that RedHat is doing 'devo on the SCL mantra here...

After reviewing the above, can't we almost hear the sound of millions of web pages dying, as well as 500+ technical tomes hitting the trash can?

Yet I would have not been able to write a the most comprehensive Java Server set-up post ever if creating an LTM stack on CentOS 7 was easy!

Tomcat Users

Once all is running on 8080, then:
/usr/bin/systemctl stop tomcat
vi /etc/tomcat/tomcat-users.xml
~ then add ~
<user name="YourSecretName" password="YourSecretPassword" roles="admin,manager,admin-gui,admin-script,manager-gui,manager-script,manager-jmx,manager-status" />
followed by the
/usr/bin/systemctl start tomcat
-once again.

From 80 to 8080...

Sadly, the easiest way to get Tomcat to show up on port 80 is still via port forwarding.

Even more sadly, a heckofalotta things have to be gutted & installed to get our ports working. So if we see nothing when attempting to `cat` /etc/sysconfig/iptables, then one must:
systemctl stop firewalld
systemctl mask firewalld
yum install iptables-services
systemctl enable iptables

iptables -A INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 8080 -j ACCEPT
iptables -A OUTPUT -o eth0 -p tcp --dport 80 -j ACCEPT
iptables -A OUTPUT -o eth0 -p tcp --dport 8080 -j ACCEPT
iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080

iptables-save > /etc/sysconfig/iptables
Note that adding any of these port-mappings to the bottom of /etc/rc.d/rc.local will not work. --The prohibition is an order-of-IPL type of thing.

Also note that if we see any under-desirable REJECTS when we do a:
iptables -L
-that we will have to remove those, as well.

I had two such port rejects.

To remove an offensive blockage from an iptables chain, we can either edit the iptables file directly (discussed later,) or use something like
iptables -D INPUT 1'sbn
iptables -D FORWARD 1'sbn
- where "1'sbn" is the 1's based item-number shown from the -L command.


Note that the:
    /usr/bin/systemctl enable mysqld
/usr/bin/systemctl enable tomcat
ensured that both MySQL and Tomcat would ever be there whenever that infamous 00.01 is subtracted from that 100% remainder of the work-weeks (!) 365.25. (i.e. 99.99? ;o)

Yet to verify that our on-boot iptables settings will be just as robust, then be sure to:
cat /etc/sysconfig/iptables
-We may also manually edit the same before a
systemctl restart iptables

Strawman - NOT!

Whenever one makes a change to the iptables using the command line interface, unless we want our port-mappings to go-away then don't forget to:
iptables-save > /etc/sysconfig/iptables
-or one dark & stormy night even a scheduled, maintenance reboot might suddenly begin to make one's after(work)life disappear.

MySQL Connector

Note also that yes, we can still:
yum -y install mysql-connector-java
Be sure to place the Connector either into your project's Java library bundle, and / or into the $CATALINA_HOME/lib wherever Tomcat will be used. -Never provided with Tomcat, we will need the MySQL Connector wherever we will be connecting from our application (R&D, Testing, Production (et al)), to an MySQL Server : port.

In my own "army of one" research, given the choice between modifying conf/server.xml or conf/context.xml in the $CATALINA_HOME, we wanted to share the resource. Because bouncing the 'tcat is also no longer much of a problem, we decided to use the former:
  <Resource name="jdbc/TestDB" auth="Container" type="javax.sql.DataSource"
maxActive="100" maxIdle="30" maxWait="10000"
username="javauser" password="javadude" driverClassName="com.mysql.jdbc.Driver"
Such told however, note that if the Resource definition is exactly duplicated, that creating multiple definitions in both files are presently little more than an annoying maintenance problem.

Next, as recommended by the Tomcat docs, we temporarily granted universal access & created a simple database:
sudo bash
GRANT ALL PRIVILEGES ON *.* TO 'javauser'@'localhost'
create database javatest;
use javatest;
create table testdata (
id int not null auto_increment primary key,
foo varchar(25),
bar int);
insert into testdata values(null, 'hello', 12345);
If you are attaching to a remote MySQL, be sure to replace javauser@localhost with the user@IP of your R&D machine.

Please note that putting quotes around the user@server identifier - as shown above - is also a well kept secret. -A secret that will make any dotted-quad session identification quite impossible without them.

Note also that - unlike any form of rational English usage of the concept "IDENTIFIED BY" - that javadude is the MySQL password that Mr. or Miss javauser must use.

Beware the bind-address!

If you are connecting okay on your localhost, yet see "error 2003 (hy000)" and / or need to tunnel through ssh (:22) elsewhere, then you will have to find / -type f ... your my.cnf file.

On CentOS, the my.cnf file is usually /etc/my.cnf. Ubuntu, /etc/mysql/my.cnf.... but your mileage may vary.

Yet no matter where the mysqld configuration file is located, we will want to comment-out that bind-address:

# bind-address=
# Disabling symbolic-links is recommended to prevent assorted security risks

--You will now be able to create a classic Java Connection String without using strange (and slow!) SSH connection-tunnelling strategies.

Remember that to keep your brain fully rooted inside of it's proper casing to be sure to finish up with a
service mysqld restart

after saving that my.cnf edit!

Ultimately, note that if & when all else fails (it has been known :) that we can read the MySQL connection troubleshooting scroll for ourselves.

The Java Project

Turning our attention to the WEB-INF/web.xml in our Java project, we added the obligatory
<web-app xmlns=""
<description>MySQL Test App</description>
<description>DB Connection</description>
When working remotely (who should not?) we also added the exact same <Resource name="jdbc/TestDB" ... (as used above) to our project's META-INF/context.xml file.


If the above to-do retelling did not rocket your project into the inter-app hyperspace of cloud 'nerdvanna, then you will surely want to read more about either:

(1) Installing Tomcat 7

(2) Installing MySQL on CentOS 7

(3) Re-using those VERY nicely-demonstrative Tomcat 7 MySQL + other database example walk-throughs.

(4) Or perhaps even reviewing the official list of MySQL Connectors, as you may insist ...

Sharing is caring!


[ view entry ] ( 1087 views )   |  permalink  |  related link

<<First <Back | 1 | 2 | 3 | 4 | Next> Last>>