English
README
README for “Hadoop Data Warehousing with Hive”
Strata + Hadoop World 2012 Tutorial Exercises
Dean Wampler academy@thinkbiganalytics.com
@thinkBigAWelcome! Please follow these instructions to download the tutorial presentation and exercises.
About this Hive Tutorial
This Hive Tutorial is adapted from a longer Think Big Academy course on Hive. (The Academy is the education arm of Think Big Analytics.) We offer various public and private courses on Hadoop programming, Hive, Pig, etc. We also provide consulting on Big Data problems and their solutions, especially using Hadoop. If you want to learn more, visit thinkbiganalytics.com or send us email.
We’ll log into Amazon Elastic MapReduce (EMR) clusters[1] to do the exercises. Feel free to pair program with a neighbor, if you want.
NOTE: The exercises should work with any version of Hive, v0.7.1 or later.
Getting Started
Download the following zip file that contains a PDF of the tutorial presentation, the exercises, the data used for the exercises, and a Hive cheat sheet:
Unzip the
tutorial.zipin a convenient place on your laptop.If you are on Windows, you’ll need the
sshclient application putty to log into the EMR servers. You can download and install it from here:Manifest for Tutorial Zip File
Item Whazzat? README.htmlWhat you’re reading! ThinkBigAcademy-Hive-Tutorial.pdfThe tutorial presentation. exercisesThe exercises we’ll use. They are also installed on the clusters, but you’ll open them “locally” in an editor, then use copy and paste. dataThe data files we’ll use. They are here only for your reference later. We’ll use the copies already on the clusters. HiveCheatSheat.htmlA Hive cheat sheet. exercises/.hivercDrop this file in the home directory on any machines where you will normally run the hivecommand-line interface (CLI). Hive will run the commands it contains when it starts. This file is a great place to put commands you always run on startup, such as property settings. Already on the cluster.Log into one of the Amazon Elastic MapReduce Clusters
We have several EMR clusters running and you’ll log into one of them according to the first one or two letters of your last name, using the following table[2]:
Letters Server Name JobFlow ID Aec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKBa - Bhec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKBi - Bzec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKCa - Chec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKCi - Czec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKDec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKE - Fec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKGec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKHec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKI - Jec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKK - Lec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKMa - Mhec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKMi - Mzec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKN - Pec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKQ - Rec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKSa - Shec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKSi - Szec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKT - Vec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKWa - Whec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBKWi - Zec2-50-19-185-170.compute-1.amazonaws.comj-1R3E26P0T3IBK(We’ll explain the JobFlow ID later.)
Once you have picked the correct server, use the following
sshcommand, for Linux, Mac OSX, or use the equivalentputtycommand to log into your server. You’ll be userhadoop:ssh hadoop@ec2-NN-NN-NNN-NNN.compute-1.amazonaws.comThe password is:
strataFinally, since you are sharing the primary user account on the cluster, create a personal work directory using
mkdirfor any file editing that you’ll do today. Pick a name for the directory without spaces, i.e., like a typical user name. You will use that same name for another purpose shortly, as we’ll see. After creating it, change to that directory with thecdcommand:mkdir myusername cd myusernamePlease don’t break anything! ;^) Remember, you’re sharing this cluster.
Feel free to snoop around if you’re waiting for others. Note that all the Hadoop software is installed in the
hadoopuser’s$HOMEdirectory,/home/hadoop.Quick Cheat Sheet on Linux Shell Commands
If you’re not accustomed to the Linux or Mac OSX
bashshell, here are a few hints[3]:Print your current working directory
pwdList the contents of a directory
Add the
-loption to show a longer listing with more information. If you omit the directory, the current directory is used:ls some-directory ls -l some-directoryChange to a different directory
Four variants; using i) an absolute path, ii) a subdirectory of the current directory, iii) the parent directory of the current directory, and iv) your home directory:
cd /home/hadoop cd exercises cd .. cd ~Page through the contents of a file.
Hit the space bar to page,
qto quit:more some-fileDump the contents without paging
I.e., “concatenate” or “cat” the file:
cat some-fileFor More Information
For more information on Amazon Elastic MapReduce commands, see the Quick Reference Guide and the Developer Guide.
For more details on Hive, see Programming Hive or the Hive Wiki.
Visit The AWS EMR Page and the EMR Documentation page for more information about EMR. ↩
I used the following information to determine a good distribution of users across these clusters. Note that these EMR clusters will only be available during the time of the tutorial. ↩
You should learn how to use
bashif you want to use Hadoop. ↩
