{"id":11706,"date":"2019-11-02T17:58:22","date_gmt":"2019-11-02T17:58:22","guid":{"rendered":"https:\/\/ittutorial.org\/?p=11706"},"modified":"2019-11-10T10:07:26","modified_gmt":"2019-11-10T10:07:26","slug":"big-data-import-csv-to-hive","status":"publish","type":"post","link":"https:\/\/ittutorial.org\/big-data-import-csv-to-hive\/","title":{"rendered":"Big Data – Import .csv to Hive"},"content":{"rendered":"

Hi everyone, \u00a0In this article we will see how to add a dataset we downloaded from kaggle as a Hive table.<\/span><\/p>\n

Hive is not a database. \u00a0This is to make use of SQL capabilities by defining a metadata to the files in HDFS. \u00a0Long story short, it brings the possibility to query the hdfs file.<\/span><\/p>\n

 <\/p>\n

\"hive<\/p>\n

 <\/p>\n

I’m doing it on the virtual machine I downloaded from Coudera’s site.\u00a0https:\/\/www.cloudera.com\/downloads\/quickstart_vms\/5-13.html<\/a><\/span><\/p>\n

First, download a dataset from kaggle.<\/span><\/p>\n

Let’s try this one,<\/p>\n

\"\"<\/p>\n

 <\/p>\n

Move the downloaded data set to the virtual machine with a program such as WinSCP or FileZilla.<\/span><\/p>\n

\"\"<\/p>\n

Connect to the virtual machine and start our operations. \u00a0As you can see in the picture, we see the file we threw into the virtual machine.<\/span><\/p>\n

Let’s transfer this file to the hadoop file system.<\/span><\/p>\n

\"\"<\/p>\n

 <\/p>\n

hadoop fs -copyFromLocal african_crises.csv data\/\r\n\r\nhadoop fs -ls \/data<\/pre>\n
\"\"<\/p>\n
Now we will export this csv file to a table we will create.<\/span><\/p>\n
You can do this via “hive shell” or “hue”<\/span>. \u00a0You’ll be doing the same thing in both processes<\/span>.<\/p>\n
To make the text look more beautiful, let’s perform this process over Hue<\/strong>.<\/span><\/p>\n
After reaching the hue via the web interface, you must open the location indicated by the arrow.<\/span><\/p>\n
\"\"<\/p>\n
 <\/p>\n
I took the column names of the table from the csv file and set the data types.<\/span><\/p>\n
\"\"<\/p>\n
 <\/p>\n
Ready to export our data from the csv file to the table.<\/span><\/p>\n
load data local inpath '\/home\/cloudera\/african_crises.csv' overwrite into table african_crises;<\/pre>\n
\"\"<\/p>\n
 <\/p>\n
Check the table,<\/p>\n
 <\/p>\n
\"\"<\/p>\n
Everything looks fine.<\/p>\n
 <\/p>\n
See you in next article..<\/p>\n","protected":false},"excerpt":{"rendered":"
Hi everyone, \u00a0In this article we will see how to add a dataset we downloaded from kaggle as a Hive table. Hive is not a database. \u00a0This is to make use of SQL capabilities by defining a metadata to the files in HDFS. \u00a0Long story short, it brings the possibility to query the hdfs file. …<\/p>\n","protected":false},"author":67,"featured_media":11707,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[6674],"tags":[],"class_list":["post-11706","post","type-post","status-publish","format-standard","has-post-thumbnail","","category-big-data"],"aioseo_notices":[],"jetpack_featured_media_url":"https:\/\/ittutorial.org\/wp-content\/uploads\/2019\/11\/1_BMO8Je_iwGEMQo26m6GhQw.png","jetpack_sharing_enabled":true,"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/ittutorial.org\/wp-json\/wp\/v2\/posts\/11706","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ittutorial.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ittutorial.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ittutorial.org\/wp-json\/wp\/v2\/users\/67"}],"replies":[{"embeddable":true,"href":"https:\/\/ittutorial.org\/wp-json\/wp\/v2\/comments?post=11706"}],"version-history":[{"count":1,"href":"https:\/\/ittutorial.org\/wp-json\/wp\/v2\/posts\/11706\/revisions"}],"predecessor-version":[{"id":11716,"href":"https:\/\/ittutorial.org\/wp-json\/wp\/v2\/posts\/11706\/revisions\/11716"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ittutorial.org\/wp-json\/wp\/v2\/media\/11707"}],"wp:attachment":[{"href":"https:\/\/ittutorial.org\/wp-json\/wp\/v2\/media?parent=11706"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ittutorial.org\/wp-json\/wp\/v2\/categories?post=11706"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ittutorial.org\/wp-json\/wp\/v2\/tags?post=11706"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}