{"id":563,"date":"2016-06-06T09:16:14","date_gmt":"2016-06-06T00:16:14","guid":{"rendered":"https:\/\/52.185.133.111\/?p=563"},"modified":"2019-04-02T13:40:45","modified_gmt":"2019-04-02T04:40:45","slug":"%e3%83%96%e3%82%bf%e3%81%a8%e5%b7%a3%e7%ae%b1%e3%81%a7%e5%bf%ab%e9%81%a9-hadoop-life%ef%bc%81%ef%bc%9f-apache-spark-%e3%81%a8-apache-pig-%ef%bc%86-apache-hive","status":"publish","type":"post","link":"https:\/\/tech-lab.sios.jp\/archives\/563","title":{"rendered":"\u30d6\u30bf\u3068\u5de3\u7bb1\u3067\u5feb\u9069 Hadoop Life\uff01\uff1f – Apache Spark \u3068 Apache Pig \uff06 Apache Hive –"},"content":{"rendered":"
\n

Hadoop \u30a8\u30b3\u30b7\u30b9\u30c6\u30e0\u306e\u4e2d\u3067\u3001\u3088\u304f\u7d44\u307f\u5408\u308f\u305b\u3066\u7528\u3044\u3089\u308c\u308b\u306e\u304c Pig (Apache Pig) \u3068 Hive (Apache Hive) \u3067\u3059\u3002\u76f4\u8a33\u3059\u308b\u3068\u30d6\u30bf\u3068\u5de3\u7bb1 \uff08\u7b11\uff09\u3002\u30d6\u30bf\u3055\u3093\u304c\u30be\u30a6\u3055\u3093\u306e\u4ef2\u9593\u3060\u3063\u305f\u3068\u306f\u30fb\u30fb\u30fb\u4e16\u9593\u306f\u5de3\u7bb1\u306e\u4e2d\u306e\u3088\u3046\u306b\u72ed\u3044\u3067\u3059\u306d\u3002<\/p>\n<\/div>\n

\u3053\u308c\u304b\u3089\u306e Hadoop \u306f Spark \u304c\u4e3b\u6d41\u3068\u8a00\u308f\u308c\u3066\u3044\u307e\u3059\u304c\u30fb\u30fb\u30fb<\/h2>\n

Hadoop 1.x \u304c MapReduce<\/strong> \u3092\u5b9f\u88c5\u3057\u3001\u3053\u306e\u4e16\u306b\u767b\u5834\u3057\u3066\u306f\u308410\u5e74\u304c\u7d4c\u3061\u307e\u3057\u305f\u3002\u305d\u306e\u5f8c YARN \u4e0a\u3067 MapReduce \u3092\u7a3c\u50cd\u3055\u305b\u308b Hadoop 2.x \u304c\u51fa\u73fe\u3057\u3001\u305d\u306e\u9803\u304b\u3089 Hadoop \u30a8\u30b3\u30b7\u30b9\u30c6\u30e0\u306e\u958b\u767a\u304c\u6025\u901f\u306b\u9032\u307f\u59cb\u3081\u307e\u3057\u305f\u3002<\/p>\n

\u305d\u3057\u3066\u4eca\u6ce8\u76ee\u3055\u308c\u3066\u3044\u308b\u306e\u306f Spark<\/strong> <\/span>\u3067\u3059\u3002 \uff08\u304a\u3084\uff1f\u30bf\u30a4\u30c8\u30eb\u3068\u9055\u3046\u305e\uff1f\uff1f\uff09
\nSpark<\/strong> \u306f Hadoop \u306e MapReduce \u90e8\u5206\u3092\u7f6e\u304d\u63db\u3048\u308b\u3053\u3068\u3092\u76ee\u6a19\u3068\u3057\u3066\u958b\u767a\u3055\u308c\u305f\u30a4\u30f3\u30e1\u30e2\u30ea\u306e\u5206\u6563\u30d5\u30ec\u30fc\u30e0\u30ef\u30fc\u30af\u00a0<\/strong>\u3067\u3059\u3002<\/p>\n

\u5404 Hadoop \u30c7\u30a3\u30b9\u30c8\u30ea\u30d3\u30e5\u30fc\u30bf\u304b\u3089\u306f\u3001\u4eca\u5f8c\u306f Spark \u3078\u6ce8\u529b<\/span>\u00a0<\/strong>\u3057\u3066\u3044\u304f\u3068\u3044\u3046\u8868\u660e\u304c\u306a\u3055\u308c\u3066\u3044\u307e\u3059\u3002\u305d\u308c\u306b\u4f34\u3044 Spark \u5bfe\u5fdc\u306e Hadoop \u30a8\u30b3\u30b7\u30b9\u30c6\u30e0\u3082\u3053\u308c\u304b\u3089\u5145\u5b9f\u3057\u3066\u3044\u304f\u3053\u3068\u3067\u3057\u3087\u3046\u3002 \u00a0\uff08\u304a\u3084\uff1f\u30bf\u30a4\u30c8\u30eb\u3068\u9055\u3046\u305e\uff1f\uff1f\uff09<\/p>\n

\u3057\u304b\u3057\u3001Spark \u306f\u00a0\u30e1\u30e2\u30ea\u4e0a\u3067\u30c7\u30fc\u30bf\u51e6\u7406\u00a0<\/strong>\u3092\u884c\u3046\u305f\u3081\u3001\u30b5\u30fc\u30d0\u306b\u5927\u91cf\u306e\u30e1\u30e2\u30ea\u3092\u642d\u8f09\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002\u3067\u3082\u5927\u5bb9\u91cf\u30e1\u30e2\u30ea\u306b\u5bfe\u5fdc\u3057\u305f\u30cf\u30fc\u30c9\u30a6\u30a7\u30a2\u306f\u307e\u3060\u9ad8\u4fa1\u3067\u3059\u3002\u307e\u305f\u3001\u5bfe\u5fdc\u3059\u308b\u30a8\u30b3\u30b7\u30b9\u30c6\u30e0\u3082\u307e\u3060\u51fa\u59cb\u3081\u305f\u3070\u304b\u308a\u306a\u306e\u3067\u3001\u52d5\u4f5c\u306e\u5b89\u5b9a\u6027\u3084Hadoop\u74b0\u5883\u3078\u306e\u9069\u5408\u6027\u306a\u3069\u3001\u4e0d\u5b89\u8981\u7d20\u304c\u307e\u3060\u307e\u3060\u3042\u308a\u307e\u3059\u3002
\n\u305d\u306e\u70b9\u3001\u65e7<\/strong><\/span>\uff08\u5931\u793c\uff01\uff09Hadoop \u30a8\u30b3\u30b7\u30b9\u30c6\u30e0<\/span>\u00a0<\/strong>\u306f\u3001\u30b3\u30e2\u30c7\u30a3\u30c6\u30a3\u30cf\u30fc\u30c9\u30a6\u30a7\u30a2\u3067\u4f7f\u3048\u308b\u00a0<\/strong>\u306e\u304c\u5229\u70b9\u3067\u3059\u3002\u3055\u3089\u306b\u3001\u5229\u7528\u8005\u3092\u652f\u63f4\u3059\u308b\u00a0\u30b3\u30df\u30e5\u30cb\u30c6\u30a3\u3082\u5145\u5b9f\u00a0<\/strong>\u3057\u3066\u304a\u308a\u3001\u30d0\u30b0\u306a\u3069\u306e\u5bfe\u5fdc\u3082\u8fc5\u901f\u306b\u884c\u308f\u308c\u3066\u3044\u307e\u3059\u3002\u307e\u3060\u307e\u3060\u9700\u8981\u306f\u3042\u308a\u305d\u3046\u3067\u3059\u3002<\/p>\n

\u305d\u3053\u3067\u4eca\u56de\u306f\u3001\u6539\u3081\u3066 \u65e7\uff08\u518d\u3073\u5931\u793c\uff01\uff09Hadoop \u30a8\u30b3\u30b7\u30b9\u30c6\u30e0\u306e\u826f\u3055\u3092\u518d\u8a8d\u8b58\u3057\u3066\u307f\u3088\u3046\u3068\u3044\u3046\u3053\u3068\u3067\u3001\u30d6\u30bf\u3055\u3093\uff08Pig\uff09<\/strong><\/span> \u3068 \u5de3\u7bb1\uff08Hive\uff09<\/strong><\/span>\u306b\u767b\u5834\u3057\u3066\u3044\u305f\u3060\u304d\u307e\u3059\u3002<\/p>\n

\u306a\u305c\u3053\u306e\u30da\u30a2\u304c\u53d6\u308a\u4e0a\u3052\u3089\u308c\u3066\u3044\u308b\u306e\u304b\uff1f\u3068\u3044\u3046\u7406\u7531\u306f\u3001\u3053\u306e\u5f8c\u3067\u8aac\u660e\u3059\u308b \u30c7\u30fc\u30bf\u306e\u53d6\u308a\u8fbc\u307f\u304b\u3089\u5206\u6790\u3057\u305f\u7d50\u679c\u3092\u5f97\u308b\u307e\u3067\u306e\u30d7\u30ed\u30bb\u30b9\u00a0<\/strong>\u3092\u3053\u306e\uff12\u3064\u3067 \u307b\u3068\u3093\u3069\u30ab\u30d0\u30fc\u00a0<\/strong>\u3057\u3066\u3057\u307e\u3046\u307b\u3069\u00a0\u5f37\u529b\u306a\u30c4\u30fc\u30eb\u00a0<\/strong>\u3067\u3042\u308b\u304b\u3089\u306a\u3093\u3067\u3059\u3002<\/p>\n

\"spark\"\u4f55\u3084\u3089\u6700\u8fd1\u306e\u6ce8\u76ee\u682a\u3089\u3057\u3044\u30fb\u30fb\u30fb\uff08https:\/\/www.mapr.com\/products\/open-source-engines<\/a>\uff09<\/p>\n

<\/h2>\n

 <\/p>\n

Pig \u3068\u306f\uff1f<\/h2>\n

Pig<\/strong> <\/span>\u306f\u7c73Yahoo!\u793e\u306b\u3088\u308a\u516c\u958b\u3055\u308c\u305f\u30aa\u30fc\u30d7\u30f3\u30bd\u30fc\u30b9\u30bd\u30d5\u30c8\u30a6\u30a8\u30a2\u3067\u3059\u304c\u3001\u73fe\u5728\u3067\u306f Apache\u30d7\u30ed\u30b8\u30a7\u30af\u30c8\u306e\u4e00\u90e8\u00a0<\/strong>\u306b\u306a\u3063\u3066\u3044\u307e\u3059\u3002MapReduce \u3092\u3088\u308a\u7c21\u5358\u306b\u8a18\u8ff0\u00a0<\/strong>\u3067\u304d\u308b\u3088\u3046\u306b\u958b\u767a\u3055\u308c\u307e\u3057\u305f\u3002
\nPig \u306f\u00a0\u30c7\u30fc\u30bf\u3092\u63a1\u53d6\u3057\u3066\u51e6\u7406\u3057\u3001\u6a19\u6e96\u5316<\/strong> \u3059\u308b\u3068\u3044\u3046\u30c7\u30fc\u30bf\u30d5\u30ed\u30fc\u3092\u8a18\u8ff0\u3059\u308b\u306e\u306b\u5411\u3044\u3066\u3044\u307e\u3059\u3002\u305d\u3057\u3066\u3001\u305d\u306e\u7279\u5fb4\u306f\u300cPig Philosophy\u300d<\/strong><\/span>\u3068\u3057\u3066\u8ff0\u3079\u3089\u308c\u3066\u3044\u307e\u3059\u3002<\/p>\n

\u7d9a\u304d\u306f\u3053\u3061\u3089\u304b\u3089\u30fb\u30fb\u30fb<\/a><\/p>\n

\n

\u672c\u8a18\u4e8b\u306f\u3001\u65e5\u672c\u30b5\u30fc\u30c9\u30fb\u30d1\u30fc\u30c6\u30a3\u682a\u5f0f\u4f1a\u793e\uff08JTP\uff09\u306b\u3066\u3001\u57f7\u7b46\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n

JTP\u306f\u7d0430\u5e74\u306b\u6e21\u308a\u3001\u69d8\u3005\u306a\u30d9\u30f3\u30c0\u30fc\u306e\u30b5\u30dd\u30fc\u30c8\u3092\u884c\u3046\u4f01\u696d\u3067\u3059\u3002 \u8a2d\u8a08\u3001\u69cb\u7bc9\u3001\u958b\u767a\u3001\u904b\u7528\u3001\u30d8\u30eb\u30d7\u30c7\u30b9\u30af\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306a\u3069\u3001IT\u306e\u30e9\u30a4\u30d5\u30b5\u30a4\u30af\u30eb\u3092\u901a\u3057\u3066\u6280\u8853\u30b5\u30fc\u30d3\u30b9\u3092\u63d0\u4f9b\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n

\u4e2d\u3067\u3082\u3001JTP \u306e IT\u6559\u80b2\u30b5\u30fc\u30d3\u30b9\u3067\u306f\u3001\u30af\u30e9\u30a6\u30c9\u3001Hadoop\u95a2\u9023\u6280\u8853\u306a\u3069\u3001OSS \u306e\u6700\u65b0\u6280\u8853 \u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u6570\u591a\u304f\u5b9f\u65bd\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n

JTP\u3067\u306f\u3001\u7d4c\u9a13\u8c4a\u5bcc\u306a\u30a8\u30f3\u30b8\u30cb\u30a2\u3001\u8b1b\u5e2b\u9663\u306b\u3088\u308a\u3001\u591a\u304f\u306e\u6280\u8853\u8a18\u4e8b\u3092\u516c\u958b\u3057\u3066\u304a\u308a\u307e\u3059\u3002<\/p>\n

 <\/p>\n

\u25bcJTP\u306e\u60c5\u5831\u30e1\u30c7\u30a3\u30a2\u300cJTP Technology Port\u300d\u306f\u3053\u3061\u3089\u304b\u3089\uff01<\/p>\n

https:\/\/www.jtp.co.jp\/techport\/<\/a><\/p>\n<\/div>\n\n

\u3054\u89a7\u3044\u305f\u3060\u304d\u3042\u308a\u304c\u3068\u3046\u3054\u3056\u3044\u307e\u3059\uff01\r\n\r\n\u3053\u306e\u6295\u7a3f\u306f\u304a\u5f79\u306b\u7acb\u3061\u307e\u3057\u305f\u304b\uff1f<\/span>\r\n
\r\n
\r\n\u5f79\u306b\u7acb\u3063\u305f<\/a>\r\n\u5f79\u306b\u7acb\u305f\u306a\u304b\u3063\u305f<\/a>\r\n
\r\n
\r\n0\u4eba\u304c\u3053\u306e\u6295\u7a3f\u306f\u5f79\u306b\u7acb\u3063\u305f\u3068\u8a00\u3063\u3066\u3044\u307e\u3059\u3002<\/span>\n\n<\/div>\n","protected":false},"excerpt":{"rendered":"

Hadoop \u30a8\u30b3\u30b7\u30b9\u30c6\u30e0\u306e\u4e2d\u3067\u3001\u3088\u304f\u7d44\u307f\u5408\u308f\u305b\u3066\u7528\u3044\u3089\u308c\u308b\u306e\u304c Pig (Apache Pig) \u3068 Hive (Apache Hive) \u3067\u3059\u3002\u76f4\u8a33\u3059\u308b\u3068\u30d6\u30bf\u3068\u5de3\u7bb1 \uff08\u7b11\uff09\u3002\u30d6\u30bf\u3055\u3093\u304c\u30be\u30a6\u3055\u3093\u306e\u4ef2\u9593\u3060\u3063\u305f\u3068\u306f\u30fb\u30fb\u30fb\u4e16 … <\/p>\n","protected":false},"author":5,"featured_media":583,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[298,12],"tags":[25,65,66,67],"class_list":{"0":"post-563","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-oss","8":"category-outside","9":"tag-hadoop","10":"tag-apache-pig","11":"tag-apache-spark","12":"tag-apache-hive","13":"entry"},"aioseo_notices":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/tech-lab.sios.jp\/wp-content\/uploads\/2016\/06\/datapipeline.jpg?fit=833%2C374&ssl=1","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/tech-lab.sios.jp\/wp-json\/wp\/v2\/posts\/563","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tech-lab.sios.jp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tech-lab.sios.jp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tech-lab.sios.jp\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/tech-lab.sios.jp\/wp-json\/wp\/v2\/comments?post=563"}],"version-history":[{"count":62,"href":"https:\/\/tech-lab.sios.jp\/wp-json\/wp\/v2\/posts\/563\/revisions"}],"predecessor-version":[{"id":13653,"href":"https:\/\/tech-lab.sios.jp\/wp-json\/wp\/v2\/posts\/563\/revisions\/13653"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tech-lab.sios.jp\/wp-json\/wp\/v2\/media\/583"}],"wp:attachment":[{"href":"https:\/\/tech-lab.sios.jp\/wp-json\/wp\/v2\/media?parent=563"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tech-lab.sios.jp\/wp-json\/wp\/v2\/categories?post=563"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tech-lab.sios.jp\/wp-json\/wp\/v2\/tags?post=563"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}