{"id":651,"date":"2011-10-03T21:58:00","date_gmt":"2011-10-03T21:58:00","guid":{"rendered":"http:\/\/www.javacodegeeks.com\/2012\/10\/scala-tutorial-iteration-for-expressions-yield-map-filter-count.html"},"modified":"2012-10-21T20:33:04","modified_gmt":"2012-10-21T20:33:04","slug":"scala-tutorial-iteration-for","status":"publish","type":"post","link":"https:\/\/www.javacodegeeks.com\/2011\/10\/scala-tutorial-iteration-for.html","title":{"rendered":"Scala Tutorial – iteration, for expressions, yield, map, filter, count"},"content":{"rendered":"
\n

Preface<\/h2>\n

This is part 4 of tutorials for first-time programmers getting into Scala. Other posts are on this blog, and you can get links to those and other resources on the links page of the Computational Linguistics course<\/a> I\u2019m creating these for. Additionally you can find this and other tutorial series on the JCG Java Tutorials<\/a> page.<\/p>\n

This tutorial departs from the very beginner nature of the previous three, so this may be of more interest to readers who already have some programming experience in another language. (Though also, see the section on using matching in Scala in Part 3<\/a>.)<\/p>\n

Iteration, the Scala way(s)<\/h2>\n

Up to now, we have (mostly) accessed individual items on a list by using their indices. But one of the most natural things to do with a list is to repeat some action for each item on the list, for example: \u201cFor each word in the given list of words: print it\u201d. Here is how to say this in Scala.<\/p>\n

scala> val animals = List(\"newt\", \"armadillo\", \"cat\", \"guppy\")\r\nanimals: List[java.lang.String] = List(newt, armadillo, cat, guppy)\r\n \r\nscala> animals.foreach(println)\r\nnewt\r\narmadillo\r\ncat\r\nguppy\r\n<\/pre>\n
This says to take each element of the list (indicated by foreach<\/strong>) and apply a function (in this case, println<\/strong>) to it, in order. There is some underspecification going on in that we aren\u2019t providing a variable to name elements. This works in some cases, such as above, but won\u2019t always be possible. Here\u2019s is how it looks in full, with a variable naming the element.<\/p>\n
scala> animals.foreach(animal => println(animal))\r\nnewt\r\narmadillo\r\ncat\r\nguppy\r\n<\/pre>\n
This is useful when you need to do a bit more, such as concatenating a String element with another String.<\/p>\n
scala> animals.foreach(animal => println(\"She turned me into a \" + animal))\r\nShe turned me into a newt\r\nShe turned me into a armadillo\r\nShe turned me into a cat\r\nShe turned me into a guppy\r\n<\/pre>\n
Or, if you are performing a computation with it, like outputing the length of each element in a list of strings.<\/p>\n
scala> animals.foreach(animal => println(animal.length))\r\n4\r\n9\r\n3\r\n5\r\n<\/pre>\n
We can obtain the same result as foreach<\/strong> using a for<\/strong> expression.<\/p>\n
scala> for (animal <- animals) println(animal.length)\r\n4\r\n9\r\n3\r\n5\r\n<\/pre>\n
With what we have been doing so far, these two ways of expressing the pattern of iterating over the elements of a List are equivalent. However, they are different: a for expression<\/strong> returns a value, whereas foreach<\/strong> simply performs some function on every element of the list. This latter kind of use is  termed a side-effect<\/em>: by printing out each element, we are not creating new values, we are just performing an action on each element. With for expressions<\/strong>, we can yield values that create transformed Lists. For example, contrast using println<\/strong> with the following.<\/p>\n
scala> val lengths = for (animal <- animals) yield animal.length\r\nlengths: List[Int] = List(4, 9, 3, 5)\r\n<\/pre>\n
The result is a new list that contains the lengths (number of characters) of each of the elements of the animals<\/em> list. (You can of course print its contents now by doing lengths.foreach(println)<\/strong>, but typically we want to do other, usually more interesting, things with such values.)<\/p>\n
What we just did was map the values of animals<\/em> into a new set of values in a one-to-one manner, using the function length<\/strong>. Lists have another function called map<\/strong> that does this directly.<\/p>\n
scala> val lengthsMapped = animals.map(animal => animal.length)\r\nlengthsMapped: List[Int] = List(4, 9, 3, 5)\r\n<\/pre>\n
So, the for-yield expression<\/strong> and the map<\/strong> method achieve the same output, and in many cases they are pretty much equivalent. Using map<\/strong>, however, is often more convenient because you can easily chain a series of operations together. For example, let\u2019s say you want to add 1 to a List of numbers and then get the square of that, so turning List(1,2,3) into List(2,3,4) into List(4,9,16). You can do that quite easily using map.<\/p>\n
nums.map(x=>x+1).map(x=>x*x)\r\n<\/pre>\n
Some readers will be puzzled by what was just done. Here it is more explicitly, using an intermediate variable nums2<\/em> to store the add-one list.<\/p>\n
scala> val nums2 = nums.map(x=>x+1)\r\nnums2: List[Int] = List(2, 3, 4)\r\n \r\nscala> nums2.map(x=>x*x)\r\nres9: List[Int] = List(4, 9, 16)\r\n<\/pre>\n
Since nums.map(x=>x+1)<\/strong> returns a List, we don\u2019t have to name it to a variable to use it \u2014 we can just immediately use it, including doing another map function on it. (Of course, one could do this computation in a single go, e.g. map((x+1)*(x+1)), but often one is using a series of built-in functions, or functions one has predefined already).<\/p>\n
You can keep on mapping to your heart\u2019s content, including mapping from Ints to Strings.<\/p>\n
scala> nums.map(x=>x+1).map(x=>x*x).map(x=>x-1).map(x=>x*(-1)).map(x=>\"The answer is: \" + x)\r\nres12: List[java.lang.String] = List(The answer is: -3, The answer is: -8, The answer is: -15)\r\n<\/pre>\n
Note: the use of x<\/em> in all these cases is not important. They could have been named x, y, z<\/em> and turlingdromes42<\/em> \u2014 any valid variable name.<\/p>\n
         Iterating through multiple lists<\/h2>\n
Sometimes you have two lists that are paired up and you need to do something to elements from each list simultaneously. For example, let\u2019s say you have a list of word tokens and another list with their parts-of-speech. (See the previous tutorial<\/a> for discussion of parts-of-speech.)<\/p>\n
scala> val tokens = List(\"the\", \"program\", \"halted\")\r\ntokens: List[java.lang.String] = List(the, program, halted)\r\n \r\nscala> val tags = List(\"DT\",\"NN\",\"VB\")\r\ntags: List[java.lang.String] = List(DT, NN, VB)\r\n<\/pre>\n
Now, let\u2019s say we want to output these as the following string:<\/p>\n
the\/DT program\/NN halted\/VB
 
<\/div> <\/div><\/p>\n
Initially, we\u2019ll do it a step at a time, and then show how it can be done all in one line.<\/p>\n
First, we use the zip<\/strong> function to bring two lists together and get a new list of pairs of elements from each list.<\/p>\n
scala> val tokenTagPairs = tokens.zip(tags)\r\ntokenTagPairs: List[(java.lang.String, java.lang.String)] = List((the,DT), (program,NN), (halted,VB))\r\n \r\nZipping two lists together in this way is a common pattern used for iterating over two lists.\r\n \r\nNow we have a list of token-tag pairs we can use a for expression to turn it into a List of strings.\r\n \r\n1\r\nscala> val tokenTagSlashStrings = for ((token, tag) <- tokenTagPairs) yield token + \"\/\" + tag\r\ntokenTagSlashStrings: List[java.lang.String] = List(the\/DT, program\/NN, halted\/VB)\r\n<\/pre>\n
Now we just need to turn that list of strings into a single string by concatenating all its elements with a space between each. The function mkString<\/strong> makes this easy.<\/p>\n
scala> tokenTagSlashStrings.mkString(\" \")\r\nres19: String = the\/DT program\/NN halted\/VB\r\n<\/pre>\n
Finally, here it all is in one step.<\/p>\n
scala> (for ((token, tag) <- tokens.zip(tags)) yield token + \"\/\" + tag).mkString(\" \")\r\nres23: String = the\/DT program\/NN halted\/VB\r\n<\/pre>\n
         Ripping a string into a useful data structure<\/h2>\n
It is common in computational linguistics to need convert string inputs into useful data structures. Consider the part-of-speech tagged sentence mentioned in the previous tutorial. Let\u2019s begin by assigning it to the variable sentRaw.<\/p>\n
val sentRaw = \"The\/DT index\/NN of\/IN the\/DT 100\/CD largest\/JJS Nasdaq\/NNP financial\/JJ stocks\/NNS rose\/VBD modestly\/RB as\/IN well\/RB .\/.\"\r\n<\/pre>\n
Now, let\u2019s turn it into a List of Tuples, where each Tuple has the word as its first element and the postag as its second. We begin with the single line that does this so that you can see what the desired result is, and then we\u2019ll examine each step in detail.<\/p>\n
scala> val tokenTagPairs = sentRaw.split(\" \").toList.map(x => x.split(\"\/\")).map(x => Tuple2(x(0), x(1)))\r\ntokenTagPairs: List[(java.lang.String, java.lang.String)] = List((The,DT), (index,NN), (of,IN), (the,DT), (100,CD), (largest,JJS), (Nasdaq,NNP), (financial,JJ), (stocks,NNS), (rose,VBD), (modestly,RB), (as,IN), (well,RB), (.,.))\r\n<\/pre>\n
Let\u2019s take each of these in turn. The first split<\/strong> cuts sentRaw<\/em> at each space character, and returns an Array of Strings, where each element is the material between the spaces.<\/p>\n
scala> sentRaw.split(\" \")\r\nres0: Array[java.lang.String] = Array(The\/DT, index\/NN, of\/IN, the\/DT, 100\/CD, largest\/JJS, Nasdaq\/NNP, financial\/JJ, stocks\/NNS, rose\/VBD, modestly\/RB, as\/IN, well\/RB, .\/.)\r\n<\/pre>\n
What\u2019s an Array? It\u2019s a kind of sequence, like List, but it has some different properties that we\u2019ll discuss later. For now, let\u2019s stick with Lists, which we can do by using the toList<\/strong> method. Additionally, let\u2019s assign it to a variable so that the remaining operations are easier to focus on.<\/p>\n
scala> val tokenTagSlashStrings = sentRaw.split(\" \").toList\r\ntokenTagSlashStrings: List[java.lang.String] = List(The\/DT, index\/NN, of\/IN, the\/DT, 100\/CD, largest\/JJS, Nasdaq\/NNP, financial\/JJ, stocks\/NNS, rose\/VBD, modestly\/RB, as\/IN, well\/RB, .\/.)\r\n<\/pre>\n
Now, we need to turn each of the elements in that list into pairs of token and tag. Let\u2019s first consider a single element, turning something like \u201cThe\/DT<\/em>\u201d into the pair (\u201cThe\u201d,\u201dDT\u201d)<\/em>. The next lines show how to do this one step at a time, using intermediate variables.<\/p>\n
scala> val first = \"The\/DT\"\r\nfirst: java.lang.String = The\/DT\r\n \r\nscala> val firstSplit = first.split(\"\/\")\r\nfirstSplit: Array[java.lang.String] = Array(The, DT)\r\n \r\nscala> val firstPair = Tuple2(firstSplit(0), firstSplit(1))\r\nfirstPair: (java.lang.String, java.lang.String) = (The,DT)\r\n<\/pre>\n
So, firstPair<\/em> is a tuple representing the information encoded in the string first. This involved two operations, splitting and then creating a tuple from the Array that resulted from the split. We can do this for all of the elements in tokenTagSlashStrings<\/em> using map. Let\u2019s first convert the Strings into Arrays.<\/p>\n
scala> val tokenTagArrays = tokenTagSlashStrings.map(x => x.split(\"\/\"))\r\nres0: List[Array[java.lang.String]] = List(Array(The, DT), Array(index, NN), Array(of, IN), Array(the, DT), Array(100, CD), Array(largest, JJS), Array(Nasdaq, NNP), Array(financial, JJ), Array(stocks, NNS), Array(rose, VBD), Array(modestly, RB), Array(as, IN), Array(well, RB), Array(., .))\r\n<\/pre>\n
And finally, we turn the Arrays into Tuple2s and get the result we obtained with the one-liner earlier.<\/p>\n
scala> val tokenTagPairs = tokenTagArrays.map(x => Tuple2(x(0), x(1)))\r\ntokenTagPairs: List[(java.lang.String, java.lang.String)] = List((The,DT), (index,NN), (of,IN), (the,DT), (100,CD), (largest,JJS), (Nasdaq,NNP), (financial,JJ), (stocks,NNS), (rose,VBD), (modestly,RB), (as,IN), (well,RB), (.,.))\r\n<\/pre>\n
Note<\/em>: if you are comfortable with using one-liners that chain a bunch of operations together, then by all means use them. However, there is no shame in using several lines involving a bunch of intermediate variables if that helps you break apart the task and get the result you need.<\/p>\n
One of the very useful things of having a List of pairs (Tuple2s) is that the unzip<\/strong> function gives us back two Lists, one with all of the first elements and another with all of the second elements.<\/p>\n
scala> val (tokens, tags) = tokenTagPairs.unzip\r\ntokens: List[java.lang.String] = List(The, index, of, the, 100, largest, Nasdaq, financial, stocks, rose, modestly, as, well, .)\r\ntags: List[java.lang.String] = List(DT, NN, IN, DT, CD, JJS, NNP, JJ, NNS, VBD, RB, IN, RB, .)\r\n<\/pre>\n
With this, we\u2019ve come full circle. Having started with a raw string (such as we are likely to read in from a text file), we now have Lists that allow us to do useful computations, such as converting those tags into another form.<\/p>\n
         Providing a function you have defined to map<\/h2>\n
Let\u2019s return to the postag simplification exercise we did in the previous tutorial. We\u2019ll modify it a bit: rather than shortening the Penn Treebank parts-of-speech, let\u2019s convert them to course parts-of-speech using the English words that most people are familiar with, like noun and verb. The following function turns Penn Treebank tags into these course tags, for more tags than we covered in the last tutorial (note: this is still incomplete, but serves to illustrate the point).<\/p>\n
def coursePos (tag: String) = tag match {\r\n  case \"NN\" | \"NNS\" | \"NNP\" | \"NNPS\"                       => \"Noun\"\r\n  case \"JJ\" | \"JJR\" | \"JJS\"                                => \"Adjective\"\r\n  case \"VB\" | \"VBD\" | \"VBG\" | \"VBN\" | \"VBP\" | \"VBZ\" | \"MD\" => \"Verb\"\r\n  case \"RB\" | \"RBR\" | \"RBS\" | \"WRB\" | \"EX\"                 => \"Adverb\"\r\n  case \"PRP\" | \"PRP$\" | \"WP\" | \"WP$\"                       => \"Pronoun\"\r\n  case \"DT\" | \"PDT\" | \"WDT\"                                => \"Article\"\r\n  case \"CC\"                                                => \"Conjunction\"\r\n  case \"IN\" | \"TO\"                                         => \"Preposition\"\r\n  case _                                                   => \"Other\"\r\n}\r\n<\/pre>\n
We can now map this function over the parts of speech in the collection obtained previously.<\/p>\n
scala> tags.map(coursePos)\r\nres1: List[java.lang.String] = List(Article, Noun, Preposition, Article, Other, Adjective, Noun, Adjective, Noun, Verb, Adverb, Preposition, Adverb, Other)\r\n<\/pre>\n
Voila! If we want to convert the tags in this manner and then output them as a string like what we started with, it\u2019s just a few steps. We\u2019ll start from the beginning and recap. Try running the following for yourself.<\/p>\n
val sentRaw = \"The\/DT index\/NN of\/IN the\/DT 100\/CD largest\/JJS Nasdaq\/NNP financial\/JJ stocks\/NNS rose\/VBD modestly\/RB as\/IN well\/RB .\/.\"\r\n \r\nval (tokens, tags) = sentRaw.split(\" \").toList.map(x => x.split(\"\/\")).map(x => Tuple2(x(0), x(1))).unzip\r\n \r\ntokens.zip(tags.map(coursePos)).map(x => x._1+\"\/\"+x._2).mkString(\" \")\r\n<\/pre>\n
A further point is that when you provide expressions like (x => x+1)<\/strong> to map<\/strong>, you are actually defining an anonymous function! Here is the same map operation with different levels of specification<\/p>\n
scala> val numbers = (1 to 5).toList\r\nnumbers: List[Int] = List(1, 2, 3, 4, 5)\r\n \r\nscala> numbers.map(1+)\r\nres11: List[Int] = List(2, 3, 4, 5, 6)\r\n \r\nscala> numbers.map(_+1)\r\nres12: List[Int] = List(2, 3, 4, 5, 6)\r\n \r\nscala> numbers.map(x=>x+1)\r\nres13: List[Int] = List(2, 3, 4, 5, 6)\r\n \r\nscala> numbers.map((x: Int) => x+1)\r\nres14: List[Int] = List(2, 3, 4, 5, 6)\r\n<\/pre>\n
So, it\u2019s all consistent: whether you pass in a named function or an anonymous function, map<\/strong> will apply it to each element in the list.<\/p>\n
Finally, note that you can use that final form to define a function.<\/p>\n
scala> def addOne = (x: Int) => x + 1\r\naddOne: (Int) => Int\r\n \r\nscala> addOne(1)\r\nres15: Int = 2\r\n<\/pre>\n
This is similar to defining functions as we had previously (e.g. def addOne (x: Int) = x+1<\/strong>), but it is more convenient in certain contexts, which we\u2019ll get to later. For now, the thing to realize is that whenever you map, you are either using a function that already existed or creating one on the fly.<\/p>\n
         Filtering and counting<\/h2>\n
The map<\/strong> method is a convenient way of performing computations on each element of a List, effectively transforming a List from one set of values to a new List with a set of values computed from each corresponding element. There are yet more methods that have other actions, such as removing elements from a List (filter<\/strong>), counting the number of elements satisfying a given predicate (count<\/strong>), and computing an aggregate single result from all elements in a List (reduce<\/strong> and fold<\/strong>). Let\u2019s consider a simple task: count how many tokens are not<\/em> a noun or adjective in a tagged sentence. As a starting point, let\u2019s take the list of mapped postags from before.<\/p>\n
scala> val courseTags = tags.map(coursePos)\r\ncourseTags: List[java.lang.String] = List(Article, Noun, Preposition, Article, Other, Adjective, Noun, Adjective, Noun, Verb, Adverb, Preposition, Adverb, Other)\r\n<\/pre>\n
One way of doing this is to filter out all of the nouns and adjectives to obtain a list without them and then get its length.<\/p>\n
scala> val noNouns = courseTags.filter(x => x != \"Noun\")noNouns: List[java.lang.String] = List(Article, Preposition, Article, Other, Adjective, Adjective, Verb, Adverb, Preposition, Adverb, Other)\r\n \r\nscala> val noNounsOrAdjectives = noNouns.filter(x => x != \"Adjective\")\r\nnoNounsOrAdjectives: List[java.lang.String] = List(Article, Preposition, Article, Other, Verb, Adverb, Preposition, Adverb, Other)\r\n \r\nscala> noNounsOrAdjectives.length\r\nres8: Int = 9\r\n<\/pre>\n
However, because filter<\/strong> just takes a Boolean value, we can of course use Boolean conjunction and disjunction to simplify things. And, we don\u2019t need to save intermediate variables. Here\u2019s the one liner.<\/p>\n
scala> courseTags.filter(x => x != \"Noun\" && x != \"Adjective\").length\r\nres9: Int = 9\r\n<\/pre>\n
If all we want is the number of elements, we can instead just use count<\/strong> with the same predicate.<\/p>\n
scala> courseTags.count(x => x != \"Noun\" && x != \"Adjective\")\r\nres10: Int = 9\r\n<\/pre>\n
As an exercise, try doing a one-liner that starts with sentRaw<\/em> and provides the value \u201cresX: Int = 9<\/strong>\u201d (where X<\/em> is whatever you get in your Scala REPL).<\/p>\n
In the next tutorial, we\u2019ll see how to use reduce<\/strong> and fold<\/strong> to compute aggregate results from a List.<\/p>\n
Reference: <\/i><\/strong>First steps in Scala for beginning programmers, Part 4<\/a> from our JCG partner<\/a> Jason Baldridge at the Bcomposes<\/a> blog.<\/p>\n
Related Articles :<\/i><\/strong><\/p>\n