{"id":17305,"date":"2013-09-16T01:00:50","date_gmt":"2013-09-15T22:00:50","guid":{"rendered":"http:\/\/www.javacodegeeks.com\/?p=17305"},"modified":"2013-09-13T09:37:29","modified_gmt":"2013-09-13T06:37:29","slug":"configuring-hadoop-with-guava-mapsplitters","status":"publish","type":"post","link":"https:\/\/www.javacodegeeks.com\/2013\/09\/configuring-hadoop-with-guava-mapsplitters.html","title":{"rendered":"Configuring Hadoop with Guava MapSplitters"},"content":{"rendered":"

In this post we are going to provide a new twist on passing configuration parameters to a Hadoop Mapper via the Context object. Typically, we set configuration parameters as key\/value pairs on the Context object when starting a map-reduce job. Then in the Mapper we use the key(s) to retrieve the value(s) to use for our configuration needs. The twist is we will set a specially formatted string on the Context<\/code> object and when retrieving the value in the Mapper, use a Guava MapSplitter<\/code> <\/a> to convert the formatted string into a HashMap<\/code> that will be used for obtaining configuration parameters. We may be asking ourselves why go to this trouble? By doing configuration this way we are able to pass multiple parameters to a Mapper with a single key-value pair set on the Context object. To illustrate one possible usage, we will revisit the last post<\/a>, where we covered how to perform reduce-side joins. There are two problems with the proposed solution in that post. First, we are assuming the key to join on is always the first value in a delimited string from a file. Second, we assume that the same delimiter is used for each file. What if we want to join data from files where the key is located in different locations per file and some files use different delimiters? Additionally we want to use the same delimiter (if any) for all of the data we output regardless of the delimiter used in any of the input files. While this is admittedly a contrived situation, it will serve well for demonstration purposes. First let\u2019s go over what the MapSplitter<\/code> class is and how we can use it.<\/p>\n

MapSplitter<\/h2>\n

The MapSplitter<\/code> <\/a> is a nested class in the Splitter<\/code><\/a> class. The Spitter<\/code> takes a String and splits it into parts with a given delimiter. The MapSplitter goes a step further by creating a Map<String,String> from a string with key-value pairs separated with one delimiter and the pairs themselves separated with another delimiter altogether. Let\u2019s take a look at an example:<\/p>\n

Map<String,String> configParams = Splitter.splitOn(\"#\")\r\n                                   .withKeyValueSeparator(\"=\")\r\n                                   .split(\"6=June#7=July#8=August\");<\/pre>\n
In the example above the String \"6=June#7=July#8=August\"<\/code> would be converted into a Map with keys 6,7 and 8 mapped to June, July and August respectively. The MapSplitter<\/code> is a very simple, but powerful class. Now we know how the MapSplitter<\/code> works, let\u2019s take a look at how we can use it to help us set configuration parameters for our map-reduce jobs<\/p>\n
Configuration using the MapSplitter<\/h2>\n
Previously we set the index position of our join key and the delimiter to split on as the same for all files by setting the values in the Context<\/code> object for the map-reduce job. Now we want to be able to set those on a per input file basis as needed. We will still have default values as needed. In order to accomplish this change we will create a properties file that will have the name of the file as the key and the value will be a string formatted to be used by the MapSplitter<\/code>. Our properties file will look like this:
 
<\/div> <\/div><\/p>\n
oneToManyEmployer2.txt=keyIndex=1&separator=|\r\noneToManyVehicles2.txt=keyIndex=1&separator=#<\/pre>\n
Here we are indicating the the file oneToManyEmployer2.txt has our join key at index position 1 and the delimiter is a \u201c|\u201d pipe character and the oneToManyVehicles2.txt file has the join key at index position 1 and uses a \u201c,\u201d comma as a delimiter. We are going to make a few changes to our driver class. First we are going to load the properties file (assuming we have placed the file in a directory located relative to where we invoke hadoop).<\/p>\n
InputStream inputStream = new FileInputStream(new File(\".\/jobs\/join-config.properties\"));\r\nProperties properties = new Properties();\r\nproperties.load(inputStream);<\/pre>\n
First we define a regular Splitter<\/code> object that will split the file names on a forward slash \u2018\/\u2019. Next as we loop through the file names we get the base name of the file by calling Iterables.getLast<\/code> on the Iterable<\/code> object returned from the Splitter.split<\/code> method call. Then we attempt to retrieve a configured property string for each file in the Properties.getProperty<\/code> method. Take note we also pass the defaultMapConfig<\/code> variable that supplies default values if the property is not found for the file. We\u2019ve also added some additional configuration keys and values; the delimiter to use when joining values together and the join order for the file, which is determined by it\u2019s position in the arguments provided to the program. Then we simply place the formatted string into the Context<\/code> object using the file name as the key.<\/p>\n
String defaultMapConfig = \"keyIndex=0&separator=,\";\r\nSplitter splitter = Splitter.on('\/');\r\nfor (int i = 0; i < args.length - 1; i++) {\r\n    String fileName = Iterables.getLast(splitter.split(args[i]));\r\n    String mapConfig = properties.getProperty(fileName, defaultMapConfig);\r\n    builder.append(mapConfig).append(\"&joinDelimiter=,&joinOrder=\").append(i + 1);\r\n    config.set(fileName, builder.toString());\r\n    builder.setLength(0);\r\n    filePaths.append(args[i]).append(\",\");\r\n}<\/pre>\n
Using the configuration values<\/h2>\n
To use our config values we will first have to retrieve the HashMap<\/code> stored as a String that contains our configuration parameters<\/p>\n
private Splitter.MapSplitter mapSplitter = Splitter.on(\"&\").withKeyValueSeparator(\"=\");\r\n.......\r\nprivate Map<String,String> getConfigurationMap(Context context){\r\n        FileSplit fileSplit = (FileSplit)context.getInputSplit();\r\n        String configString = context.getConfiguration().get(fileSplit.getPath().getName());\r\n        return mapSplitter.split(configString);\r\n    }<\/pre>\n
Here we are using a MapSplitter<\/code> instance variable and to create our HashMap<\/code> by retrieving the formatted string stored in the Context<\/code> with the file name used by this Mapper. Now we can simply pull the needed configuration parameters out from the map as shown here in the setup<\/code> method:<\/p>\n
protected void setup(Context context) throws IOException, InterruptedException {\r\n        Map<String,String> configMap = getConfigurationMap(context);\r\n        keyIndex = Integer.parseInt(configMap.get(\"keyIndex\"));\r\n        String separator = configMap.get(\"separator\");\r\n        splitter = Splitter.on(separator).trimResults();\r\n        String joinDelimiter = configMap.get(\"joinDelimiter\");\r\n        joiner = Joiner.on(joinDelimiter);\r\n        joinOrder = Integer.parseInt(configMap.get(\"joinOrder\"));\r\n    }<\/pre>\n
The code in the map<\/code> method remains the same as it was from our previous post<\/a> Now we have completely configurable settings per file and we aren’t restricted to having the join key in one position or having to use the same delimiter per file. Of course this is just one example, but the approach outlined here could be used to configure many other settings and it only requires one key in the Context<\/code> object.<\/p>\n
Results<\/h2>\n
Initially our data looked like this:<\/p>\n
oneToManyEmployer2.txt:<\/h4>\n
\r\nCreative Wealth|cdd8dde3-0349-4f0d-b97a-7ae84b687f9c\r\nSusie's Casuals|81a43486-07e1-4b92-b92b-03d0caa87b5f\r\nSuper Saver Foods|aef52cf1-f565-4124-bf18-47acdac47a0e\r\n.....\r\n<\/pre>\n
oneToManyVehicles2.txt:<\/h4>\n
\r\n2003 Holden Cruze#cdd8dde3-0349-4f0d-b97a-7ae84b687f9c\r\n2012 Volkswagen T5#81a43486-07e1-4b92-b92b-03d0caa87b5f\r\n2009 Renault Trafic#aef52cf1-f565-4124-bf18-47acdac47a0e\r\n.....\r\n<\/pre>\n
singlePersonRecords.txt:<\/h4>\n
\r\ncdd8dde3-0349-4f0d-b97a-7ae84b687f9c,Esther,Garner,4071 Haven Lane,Okemos,MI\r\n81a43486-07e1-4b92-b92b-03d0caa87b5f,Timothy,Duncan,753 Stadium Drive,Taunton,MA\r\naef52cf1-f565-4124-bf18-47acdac47a0e,Brett,Ramsey,4985 Shinn Street,New York,NY\r\n......\r\n<\/pre>\n
After running our map-reduce job our results look exactly like we want them:<\/p>\n
\r\n08db7c55-22ae-4199-8826-c67a5689f838,John,Gregory,258 Khale Street,Florence,SC,2010 Nissan Titan,Ellman's Catalog Showrooms\r\n0c521380-f868-438c-9916-4ab4ea76d316,Robert,Eversole,518 Stratford Court,Fayetteville,NC,2002 Toyota Highlander,Specialty Restaurant Group\r\n1303e8a6-0085-45b1-8ea5-26c809635da1,Joe,Nagy,3438 Woodstock Drive,El Monte,CA,2011 Hyundai ix35,Eagle Food Centers\r\n15360125-38d6-4f1e-a584-6ab9d1985ab8,Sherri,Hanks,4082 Old House Drive,Alexandria,OH,2003 Toyota Solara,Odyssey Records & Tapes\r\n......\r\n<\/pre>\n
Resources<\/h2>\n