{"id":50844,"date":"2016-01-07T13:00:26","date_gmt":"2016-01-07T11:00:26","guid":{"rendered":"http:\/\/www.javacodegeeks.com\/?p=50844"},"modified":"2016-01-06T23:08:59","modified_gmt":"2016-01-06T21:08:59","slug":"will-2016-apache-spark-kafka-drill","status":"publish","type":"post","link":"https:\/\/www.javacodegeeks.com\/2016\/01\/will-2016-apache-spark-kafka-drill.html","title":{"rendered":"What Will You Do in 2016? Apache Spark, Kafka, Drill and More"},"content":{"rendered":"

Let\u2019s have some fun.<\/p>\n

It\u2019s the start of a new year — we\u2019re on the threshold of something new — so let\u2019s look forward to what you\u2019re likely to be doing in 2016. Now I know the riskiness of making predictions \u2013 especially ones on record \u2013 but I\u2019m happy for you to drop back by in a year from now and see how my projections for 2016 pan out.<\/p>\n

What will you do in 2016?<\/h2>\n

Before I make my 2016 predictions, first think more generally (and playfully) about the challenge of speculating on what the future will be. Do we work from data and models? From observation and hunch? And remember that the accuracy of describing the future lies in part on how long into the future the target is.<\/p>\n

It\u2019s tradition to project distantly future views of what people\u2019s lives will be. Sometimes they are accurate and more often hilariously wrong. Looking back at what the future was supposed to be is an entertainment I call \u201cremembering the future\u201d.<\/p>\n

For example, the year 2000 caught people\u2019s imagination for many years. I came across an essay published in the year 1900 in Ladies Home Journal<\/i> with predictions for what our lives would be like in 2000. Among the predictions that were roughly correct were that motor cars would become numerous, photographs could be telegraphed from distant countries such that they could be printed in newspapers within an hour and that the US population would reach over 350 million including territories (the 2000 census put the US population at 282 million, a bit short). Much less accurate were predictions that there would be no more flies or mosquitos, through-traffic in cities would be underground or overhead such that cities would be \u201cfree from noise\u201d and that we would have ceased to use the letters C, X or Q.<\/p>\n

The future did not turn out as described, but partly because we often solve the same problems in a different way than predicted: today traffic is shunted through cities on freeways, but — alas — that does not eliminate noise. And instead of normalizing spelling by \u201cfiring\u201d certain consonants, we rely on automated spell-correct systems (sometimes with laughable results).<\/p>\n

Back to Big Data<\/h2>\n

This idea of \u201cremembering the future\u201d was a theme picked up in a lively presentation by Ted Dunning on current and future big data trends at a Big Data Singapore meet-up during the week of the Strata Hadoop World conference in that city. Also touching on the idea of where big data systems are going in the near future was another presenter, Hadoop founder Doug Cutting.<\/p>\n

\"apachespark-kafka-blog-img1\"<\/a><\/p>\n

Doug talked about the evolution of the Hadoop ecosystem particularly with regard to analytics. Batch-based computation is giving way in many cases to in-memory micro-batched computational capabilities, hence the widespread and growing interest in Apache Spark.<\/a><\/p>\n

Ted first entertained people with cultural trends that did not turn out as predicted before he described a successful, forward looking, big data project – an open source project from the 19th century that made good use ocean and wind data to build navigational charts for sailing. Jumping to the present day, Ted explained the current big data trend toward simplification in machine learning projects such that they deliver practical value. Ted also talked about the need for more streamlined ways to handle complex data to avoid having to build hundreds of tables (as was true with traditional relational systems) and he showed the advantage of leveraging the flexibility of SQL engine Apache Drill in these situations.

<\/div> <\/div><\/p>\n

Six Predictions for 2016<\/h2>\n

Inspired by others who have described big data trends, now I\u2019ll stick out my neck to make my own predictions (purely opinion) about what you\u2019ll be doing in 2016. After all, it\u2019s only one year in the future\u2026<\/p>\n

Streaming Data<\/h4>\n

I feel confident that throughout 2016 there will be explosive interest in streaming data and streaming analytics<\/a>. Streaming data will be used by many more organizations than previously and in new ways. Increasing volume in IoT sensor data is just one of the sources of streaming data. Series of events \u2013 such as clickstream data from web traffic or machine log files \u2013 will increasingly be analyzed as streams, using near-real time processing with Apache Spark or actual real time analytics with a newer tool, Apache Flink<\/a>.<\/p>\n

One of the big shifts will be a different way to think of the architecture that best supports these applications: The message queue will become a central focus in designing these systems.<\/u> The messaging layer will be much more than just a safety buffer in the workflow of a streaming analytic program. Done right, the message queue becomes a re-playable, immutable persistent log that serves multiple masters such as real time analytics applications, databases or search documents. For these reasons, I predict a greatly increased use of the already popular messaging tool Apache Kafka as well as strong interest in new MapR Streams<\/a>, an integrated messaging technology that supports the Kafka API.<\/p>\n

Shorter Time-to-Value<\/h4>\n

Businesses want practical ways to get to value faster, so I believe you are likely to try out Apache Drill<\/a> some time in 2016 if your business has any need for SQL<\/a>. With more frequent releases, Drill\u2019s capabilities continue to expand, but it\u2019s already a highly performant, highly scalable and extremely flexible query engine that uses standard SQL. This makes it equally attractive to people coming to big data from a traditional background as well as veterans of the Hadoop and NoSQL world who want a query engine that easily handles a wider variety of unstructured and nested data types, such as JSON and Parquet.<\/p>\n

Perhaps the characteristic of Drill that is most likely to make you want to try it is its ability to query data with little or no preparation \u2013 that can cut hours or days off the time needed to go from data to insights. With less time needed before you start your query, with Drill you\u2019re able to quickly build a second query based on what you\u2019ve learned from the first one. Faster development, faster insights, shorter time-to-value.<\/p>\n

\"apacespark-kafka-blog-img2\"<\/a><\/p>\n

Centralization<\/h4>\n

Increasingly people will think of the big data platform as a central part of their overall organization rather than a special purpose project. A big data platform<\/a> such as an Hadoop<\/a> and NoSQL based system<\/a> will need to be easily connected to traditional technologies such as an enterprise data warehouse, relational data base or BI tools.<\/p>\n

A paradoxical aspect of centralization for a global organization is the need to globally distribute data. Different parts of your organization need access to unified data sets. As you break down unwanted silos within or between geographically distinct centers, you\u2019ll want to avoid propagation delays. There may be legal issues that require localization of data. For these reasons, I predict many organizations will want a system with secure and reliable ways to maintain multiple data centers that can be quickly synchronized.<\/p>\n

Special Topics: Healthcare<\/h4>\n

I think the use of big data in the health care industry<\/a> is poised for rapid expansion in 2016. People are recognizing the power of using data to reduce fraud and improve healthcare delivery through the use of electronic patient histories, long-term maintenance records for machinery, and the flow of sensor information. Excellent data security and governance will of course be important for these use cases<\/a>.<\/p>\n

Special Topics: Telecommunications<\/h4>\n

Another area that will increasingly stand out in the big data space in 2016 is telecommunications<\/a>. Telecom companies already have excellent use cases for big data: offloading the strain of ETL to Hadoop while maintaining complex billing on enterprise warehouses; using anomaly detection on data to and from cell towers to discover and quickly respond to sudden usage shifts, and employing real time analytics to respond quickly users after a dropped call in order to improve experience and reduce churn.<\/p>\n

Expansions in streaming data architectures and technologies (described above) will benefit telecoms. But even if you are not working with a telecom yourself, this special case may still affect you. More non-phone applications are making use of telecom networks. Sensors in cars, for instance, often send data via a telecommunications network. Taking all that together, I predict you will likely be combining advanced telecommunications with big data in 2016.<\/p>\n

The Best Prediction: You\u2019ll Surprise Me<\/h4>\n

And my best prediction for 2016 is that you will come up with some innovative way to put big data to use that has not yet occurred to me. Maybe it will solve a problem I\u2019m already aware of but in a novel way. Or maybe it will be something entirely new. Either way, by January 2017 I\u2019ll be \u201cremembering the future\u201d as I saw it and be surprised by something new, even if my other five predictions turn out to be accurate.<\/p>\n

Other Resources<\/h2>\n

For related content by the author see these free resources:<\/p>\n