Site Sponsors:
Hadoop: Outloading Support for User-Defined Functions in Java 
So there we are, trying to locate the Java-Archive (JAR) files required for creating a User-Defined Function (UDF) for Hive. (Pig works much the same.)

Perhaps one is even using the Hortornworks Sandbox. No matter how we access Hadoop's Distributed File System (HDFS) however, many have broken a sword or three trying to locate the proper jar-file support.

Shell Games...


One of the reasons for the breakage is that we folks in the Java world will insist on packaging classes in different archives over time. -A worthy rant in itself, the sad truth is that - while classes and packages will remain the same - wee Java folks always need to scramble things about so as to make them logically easier to find.

Speaking as someone who has been known to migrate a class between archives from time to time, perhaps a JSR-someone needs to create what might be a simple meta-tracking system - a packaging idea well outside of git, and perhaps a tag closer to @Depricated.



The need is to help us keep our sanity when playing the class shell game.... but I digress!

Outloading?


Yes, I needed a new term, so I coined one: Unlike "uploading" or "downloading," the idea here is that we need to post something from INSIDE the system, so as to be available from OUTside of the same... so we can downLOAD it! (Remember the term "DLL Hell?" -I needed a new term when writing for BYTE Magazine in 1992, as well! =)

While the Internet has worked so as to make locating properly-named jar files allot easier now than it was back in 1992... it still ain't easy. Because the exact same package and class can migrate between archives so much in Java, maybe we should call the little "shell game" we all must eventually play "Jar Hell"?

So whilst we have used "hadoop-core" -> "hadoop-common" on HUE (HDP 2) -> Ambari (HDP 2.4), after the requesite Internet search (sigh) I have discovered that the best way to locate the Jar we need (i.e. nowhere else to be found!) would be to use a simple find command on the target platform, itself. --The JAR we need to create UDF's is simply to be found no-where else!

Punt!


On the present incarnation of the Hortonworks VM (again, the truly marvelous HDP 2.4 - Gotta love Ambari!) here is how to locate + post the archive(s) we need to create UDF's ... to the HDFS:

find / -type f -name "hive-exec*core.jar" -exec hadoop fs -put "{}" hdfs:/hadoop9000 ";"

(Please note that I obviously had created /hadoop9000 in the HDFS beforehand...) Ultimately, note also that the default incarnation of the fs command will not replace an existing file.


A similar exercise will also work (if required) so as to outload "hadoop-common" (or hadoop-core!) via HDFS.



Enjoy the Journey!

-Rn



[ add comment ] ( 93 views )   |  permalink
New Project Alert: Nagy's CalOMine 
Happy Memorial Day!

While I like to take some time off, my usual way to celebrate a Holiday is to get up at 4AM to write something that can help us for the rest of the year.

Today was not different: Submitted for your approval, I hope that you will enjoy our new CalOMine Project.



Designed for embarrassingly stark simplicity, one of the key advantages to CalOMine is the ability to make allot of noise (or not!) when a certain time arrives. Soon to be the subject of an entire training video, one can also use other programs to schedule events and alarms for use by CalOMine.



Hope you find it useful, but be sure to turn your volume down before trying out the alarm feature... our cat is still recovering from this morning's sound-test.


[ add comment ] ( 87 views )   |  permalink  |  related link

<<First <Back | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | Next> Last>>