Hadoop Nodes to Jessie? (CDH has released Jessie debs) +1 (maybe with CDH upgrade?).

Based on https://wikitech.wikimedia.org/wiki/Analytics/Cluster , CDH may stand for https://www.cloudera.com/products/apache-hadoop/key-cdh-components.html

In Analytics Ops meeting today, we decided we should upgrade to CDH 5.10 now that it is out, even though it doesn't have Spark 2.x like we had hoped.

• Mediawiki History reconstruction can use Spark 1.6
• The NodeManager mem leak bug is fixed in newer CDH
• We can upgrade to Debian.

This will also be good to time with the order of new Hadoop nodes, and to get done before we replace stat1002/stat1003, so we can install those as Jessie too.

Previous CDH 5.5 upgrade task: T119646

etherpad process for that upgrade: https://etherpad.wikimedia.org/p/analytics-cdh5.5
etherpad for this one: https://etherpad.wikimedia.org/p/analytics-cdh5.10

Debian Jessie vs CDH upgrade plan

1. Upgrade whole cluster to CDH 5.10 as is.
2. Get new Hadoop nodes, install those as Debian Jessie with CDH 5.10
3. Incrementally reinstall current cluster nodes as Debian Jessie.

Testing steps include loading data on labs & upgrade & testing refinery jobs before starting cluster migration

Did the upgrade in labs today:

• Went smoothly.
• Except I broke Hue. I think this was not caused by the upgrade though. Will investigate more.
• I was able to run a Jessie worker node on CDH 5.10 alongside all of the Trusty ones.
• A webrequest refine job worked fine both before and after the upgrade.

Change 336906 had a related patch set uploaded (by Ottomata):
Set hue allowed_hosts=* to work around bug http://community.cloudera.com/t5/Web-UI-Hue-Beeswax/New-Cloudera-installation-Hue-Bad-Request-400/td-p/50344/page/5

https://gerrit.wikimedia.org/r/336906

Ah ha! Hue did break because of a change. Had to do: https://gerrit.wikimedia.org/r/#/c/336906/1/templates/hue/hue.ini.erb

So, with that, everything looks good! Time to schedule...

So, we briefly talked about doing this on a weekend..buut I don't really have a free weekend day until March 4. I suppose this can wait that long. Thoughts?

I think it can wait, the advantage of doing it in a weekend would be less hassle for ourselves and users, but if you prefer to do it during a weekday thet would be fine too.

OOok, let's do this on March 4th then! Will send email.

Wait , that is the same weekend than the visualization hackathon, correct?

Current plan: do this February Tues 28th. I will send out announcement and schedule downtime.

Post upgrade manually: remove cdh5.5* packages from apt thirdparty. Use grep-dctrl line from reprepro updates to figure out what to remove:

grep-dctrl -e -S '^zookeeper$|^hadoop$|^hadoop-0.20-mapreduce$|^bigtop-jsvc$|^bigtop-utils$|^sqoop$|^hbase$|^pig$|^pig-udf-datafu$|^hive$|^oozie$|^hue$|^bigtop-tomcat$|^spark$|^avro-libs$|^parquet$|^parquet-format$|^spark-core$|^spark-history-server$|^spark-master$|^spark-python$|^spark-worker$|^mahout$|^kite$|^solr$|^sentry$|^impala$|^impala-catalog$|^impala-server$|^impala-shell$|^impa

Change 336906 merged by Ottomata:
Set hue allowed_hosts=* to work around bug http://community.cloudera.com/t5/Web-UI-Hue-Beeswax/New-Cloudera-installation-Hue-Bad-Request-400/td-p/50344/page/5

https://gerrit.wikimedia.org/r/336906