Site Sponsors:
Hadoop 2.2.0 in the Hortonworks Sandbox (HDP 2) 
For the benefit of my fellow Hadoop-ies, as I work thru the evolving coolness of the Hadoop User Experience (HUE) under HDP 2 I wanted to pass in a few observations by-value.

Use Java 6

While we could use Java 1.7, note that the following message means that we had better not use the same as the default compatibility mode:

The google-candy here is:

Java MapReduce
MaxTemperature : Unsupported major.minor version 51.0

What the run time is trying to tell us is to use Java 1.6.

Logging into the VM will - of course - verify as much:

The solution is to simply re-deploy as a JDK 1.6 rendition. Even the Job Designer's "Job Design" need not change.

Moving the source code from the 3rd edition of O'Reilly's "Hadoop: The Definitive Guide" by Tom White to Hadoop 2.2.0, I am also bumping into allot of compatibility warnings and issues.

Stay tuned!!

[ view entry ] ( 50719 views )   |  permalink
PC Signaling & IPC Efficiencies 
Linkedin: Observations on Qt Signalling:

Qt Design Lamentations

I still do not understand what was wrong with simply using structs... Indeed, even when it comes to IPC, savvy consultants have always liked marshalling such things into XDR/RPC packages.... Allot like "Bundles" or "Parsables" on Android, once a standards-based out-of-process marshalling has been ensured, data can pretty much go everywhere. The only benefit to not having so many struct-version compile-time changes is to ensure that any time-frozen stringy-application will break someday?

Data Interopt

When it comes to IPC, savvy consultants have also always liked XDR/RPC. The cornerstone of CORBA, many are gratified that - years after Microsoft upset the apple-cart with their COM/DCOM nightmare - that places like Google's GWT are finally abandoning the fat-and-chatty XML (and even JSON!) world for things like good 'old XDR.


Why did we ever get into this nightmare world of passing strings? Because - at the time - Microsoft did not like things like XDR because the transmission favoured Motorola (natural order / big-endian) format over that strange INTEL (little-endian) format.
(click to enlarge)

Many of my friends at MS felt like they were being picked on. Today however, with AMD / INTEL computing speeds being what they are in multicore, who the heck cares?


Looking at messages more as octets than double-byte characters results in transmissions that are ever-so marvellously faster, as well as often laughingly more efficient & secure than these "string things." While understanding protocols from the wire-up is tougher on the commodity-developer, taking the time to learn about things like binary efficiencies is invariably massively easier on the computer. --Many feel that weighing system performance (as well as genuine security!) over developer-ease and industry convention (homogeneity is - by very definition - insecure) is one of many long forgotten absolute necessities of designing larger-scale enterprise systems.

Lean, Mean ... and Green?

Of course, better performance means allot less switches, routers, network traffic, and computers... Such effective weight loss surely spells less rack-space, lower utility costs ... and is therefore allot easier on this spaceship that we call earth, too?

While knowledge is indeed power, perhaps wisdom might yet save our planet? :-)

Indeed, when it comes to why-the-heck use anything other than structs or XDR under QT, I remember wondering why folks ever used such silly & unsafe string-mechanism as that Qt signal tomfoolery. --For decades, marshaling structs between things like CORBA skeletons / signatures have been doing much the same thing, yet providing full-speed end-to-end type safety via delivery systems such as RPC. When it comes to stark type-safety & efficiency, why not use it between process, as well as between computers? While a bit overkill for local signal-signatures, rather than being so string-generic even a humble struct surely might be used?

CORBA Problems

Of course, with the CORBA type safety comes a problem of routing / scaling / and using generic turnstiles such as what we see in Spring Integration. Resolving those problems are what the Internet Inter-ORB Protocol (IIOP) and CORBA Services are all about.

Olde-School Bestfficiencies

Yet, come to discussing Java in a C/C++ forum (bad idea?), it is fun to note that both generic (common message header -w- size fields (etc)) - as well as all competent security (bit-level field encoding, etc.) remains ever in the realm of the absolute efficiency folded into those 1970-someting transactions... -For more secured transactions, even today it seems that only a blast to that transactional-past can be allot more readily 'rollingly obfuscated... as well as a whole lot more efficient?

[ view entry ] ( 1848 views )   |  permalink  |  related link
Hadoop, Ununtu, and Maven ... Oh My! 

Light Musings

While I am indeed gratified that my blog helps others out, from time to time one cannot help but to wonder if it is more about self-documentation ... One never knows when the same path must be trod!

Oh well, no matter if others have staunchly refused to upgrade their 'Gnome or not - For me & for now the task for the day was to get Hadoop working with the latest edition of Hortonworks, Windows, Ubuntu, Eclipse Kepler, and the code from that latest copy of "Hadoop: The Definitive Guide, 3rd Edition."

Here are the key dependancies:

Hadoop 2.2.0 (finally here!)
Hortonworks 2.0
The 3rd Edition Code

Honourable Mention:
Protocol Buffers - Google's data interchange format
Copyright 2008 Google Inc.

Upgrading to Maven 3

When it came to the mainstream, the maven2 under Ubuntu was simply a no-go. Version 3.0.* was required for the book. Higher is also required for creating Hadoop.

Following the thread of ideas found here, after removing maven2, here is what worked to get that first pass at those POMs a 'poppin:
sudo -H gedit /etc/apt/sources.list

Add the following line the sources.list file:

deb precise main
deb-src precise main

Then we can:
sudo apt-get update && sudo apt-get install maven3
sudo ln -s /usr/share/maven3/bin/mvn /usr/bin/mvn
sudo ln -s /usr/bin/mvn3 /usr/bin/mvn

Other Obsolescence

Next was the need to get a few of those c-libs up to spec. That bit of sage advice came from here.

Since we need to install manually, we will need to have g++ installed:
sudo apt-get install g++

Next, we will need to get the source code:
wget ... 5.0.tar.gz
tar xzvf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
make check
sudo make install
sudo ldconfig

Finally, since the version was indeed the killer here, be sure to re-check the above newly-installed version:
$ protoc --version
libprotoc 2.5.0

So far all is well (i.e. the locus is still building as I type this) ...


'Da Nag.

[ view entry ] ( 1627 views )   |  permalink  |  related link

<<First <Back | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | Next> Last>>