{"id":641,"date":"2011-10-07T16:35:00","date_gmt":"2011-10-07T16:35:00","guid":{"rendered":"http:\/\/www.javacodegeeks.com\/2012\/10\/scala-tutorial-scala-io-source-accessing-files-flatmap-mutable-maps.html"},"modified":"2012-10-21T20:31:12","modified_gmt":"2012-10-21T20:31:12","slug":"scala-tutorial-scalaiosource-accessing","status":"publish","type":"post","link":"https:\/\/www.javacodegeeks.com\/2011\/10\/scala-tutorial-scalaiosource-accessing.html","title":{"rendered":"Scala Tutorial – scala.io.Source, accessing files, flatMap, mutable Maps"},"content":{"rendered":"
\n

Preface<\/h2>\n

This is part 8 of tutorials for first-time programmers getting into Scala. Other posts are on this blog, and you can get links to those and other resources on the links page of the Computational Linguistics course<\/a> I\u2019m creating these for. Additionally you can find this and other tutorial series on the JCG Java Tutorials<\/a> page.<\/p>\n

This tutorial is about accessing the file system in order to work with text files. The previous tutorial showed how to build a Map that contains the counts of each word type in a given text. However, it was assumed that the text was available in a String variable, and typically we are interested in knowing things about files that live on the file system, or on the internet. This tutorial shows how to read a file\u2019s contents into Scala for processing, both by building a single String for the file or by consuming it line-by-line in a streaming fashion. Along the way, immutable Maps are introduced as a way to enable word counting without reading an entire file into memory.<\/p>\n

Word count on the contents of a file<\/h2>\n

As an example, we\u2019ll use the complete Sherlock Holmes from project Gutenberg<\/a>. Download it, put it into a directory, and then start up the Scala REPL in that directory. To access files, we\u2019ll use the Source<\/strong> class, so to start you need to import it.<\/p>\n

scala> import scala.io.Source\r\nimport scala.io.Source\r\n<\/pre>\n
Source<\/strong> provides a number of ways to interact with files and make them accessible to you in your Scala program. The fromFile<\/strong> method is the one you\u2019ll probably need most.<\/p>\n
scala> Source.fromFile(\"pg1661.txt\")\r\nres3: scala.io.BufferedSource = non-empty iterator\r\n<\/pre>\n
This creates a BufferedSource<\/strong>, from which you can easily get all of file\u2019s contents as a String.<\/p>\n
scala> val holmes = Source.fromFile(\"pg1661.txt\").mkString\r\nholmes: String =\r\n\"Project Gutenberg's The Adventures of Sherlock Holmes, by Arthur Conan Doyle\r\n\r\nThis eBook is for the use of anyone anywhere at no cost and with\r\nalmost no restrictions whatsoever.  You may copy it, give it away or\r\nre-use it under the terms of the Project Gutenberg License included\r\nwith this eBook or online at www.gutenberg.net\r\n<...many more lines...>\r\n<\/pre>\n
With this, you can do the same things as shown it tutorial 7<\/a> to get the word counts (except that here we\u2019ll split on white space sequences rather than just a single space).<\/p>\n
scala> val counts = holmes.split(\"\\\\s+\").groupBy(x=>x).mapValues(x=>x.length)\r\ncounts: scala.collection.immutable.Map[java.lang.String,Int] = Map(wood-work, -> 1, \"Pray, -> 1, herself. -> 2, stern-post -> 1, \"Should -> 1, incident -> 8, serious -> 14, earth--\" -> 2, sinister -> 10, comply -> 7, breaks -> 1, forgotten -> 3, precious -> 10, 'It -> 3, compliment -> 2, suite, -> 1, \"DEAR -> 1, summarise. -> 1, \"Done -> 1, fine.' -> 1, lover -> 5, of. -> 2, lead. -> 1, plentiful -> 1, 'Lone -> 4, malignant -> 1, terrible -> 14, rate -> 1, mole -> 1, assert -> 1, lights -> 2, Stevenson, -> 1, submitted -> 4, tap. -> 1, beard, -> 1, band--a -> 1, force! -> 1, snow -> 7, Produced -> 2, ask, -> 1, purchasing -> 1, Hall, -> 1, wall. -> 5, remarked -> 32, laughing -> 4, member.\" -> 1, 30,000 -> 2, Redistributing -> 1, coat, -> 6, \"'One -> 2, 'band,' -> 1, relapsed -> 1, apol...\r\n\r\nscala> counts(\"Holmes\")\r\nres2: Int = 197\r\n\r\nscala> counts(\"Watson\")\r\nres3: Int = 4\r\n<\/pre>\n
Lest you think it strange that Watson<\/em> only shows up four times, keep in mind that we split on whitespace, and that means that in a sentence like the following, the token of interest is Watson,\u201d<\/em> rather than Watson<\/em>.<\/p>\n
\u201cYou could not possibly have come at a better time, my dear Watson,\u201d he said cordially.<\/em>
\n
\n<\/em><\/div>\n
Looking that and others up shows more tokens containing Watson<\/em> in the story.<\/p>\n
scala> counts(\"Watson,\\\"\")\r\nres4: Int = 19\r\n\r\nscala> counts(\"Watson,\")\r\nres5: Int = 40\r\n\r\nscala> counts(\"Watson.\")\r\nres6: Int = 10\r\n<\/pre>\n
Of course, the real problem is that tokenizing on whitespace is too crude. To do this properly generally takes a good hand-built tokenizer (which is able to keep tokens like e.g.<\/em> and Mr.<\/em> and Yahoo!<\/em> while splitting punctuation off most words) or a machine learned one that is trained on data hand-labeled for tokens. For an example of the latter, see the Apache OpenNLP toolkit tokenizers<\/a>, which includes pre-trained models for English.<\/p>\n
     Working line by line<\/h2>\n
Quite often, you need to work through a file line by line, rather than reading the entire thing in as a single string as we did above. For example, you might need to process each line differently, so just having it as a single String isn\u2019t particular convenient. Or, you might be working with a large file that cannot easily fit into memory (which is what happens when you read in the entire string). You can obtain the lines in the file as an Iterator[String]<\/strong>, in which each item is a single line from the file, using the getLines<\/strong> method.<\/p>\n
scala> Source.fromFile(\"pg1661.txt\").getLines\r\nres4: Iterator[String] = non-empty iterator\r\n<\/pre>\n
This iterator is ready for you to consume lines, but it doesn\u2019t read all of the file into memory right away \u2014 instead it buffers it such that each line will be available for you as you ask for it, essentially reading off disk as you demand more lines. You can think of this as streaming<\/em> the file to your Scala program, much like modern audio and video content is streamed to your computer: it is never actually stored, but is just transferred in parts to where it is needed, when it is needed.<\/p>\n
Of course, Iterators share much with sequence data structures like Lists: once we have an Iterator, we can use foreach<\/strong>, for<\/strong>, map<\/strong>, etc. on it. So to print out all of the lines in the file, we can do the following.<\/p>\n
scala> Source.fromFile(\"pg1661.txt\").getLines.foreach(println)\r\nProject Gutenberg's The Adventures of Sherlock Holmes, by Arthur Conan Doyle\r\n\r\nThis eBook is for the use of anyone anywhere at no cost and with\r\nalmost no restrictions whatsoever.  You may copy it, give it away or\r\nre-use it under the terms of the Project Gutenberg License included\r\nwith this eBook or online at www.gutenberg.net\r\n\r\nTitle: The Adventures of Sherlock Holmes\r\n\r\nAuthor: Arthur Conan Doyle\r\n<...many more lines...>\r\n<\/pre>\n
That creates a lot of output, but it shows you how you can easily create your own Scala implementation of the Unix cat<\/strong> program: just save the following line in a file called cat.scala<\/strong>:
 
<\/div> <\/div><\/p>\n
scala.io.Source.fromFile(args(0)).getLines.foreach(println)\r\n<\/pre>\n
And then call that with the name of the file to list its contents.<\/p>\n
$ scala cat.scala pg1661.txt\r\n<\/pre>\n
Back in the REPL, it is somewhat less-than-ideal to see the entire file. If you just want to see the start of the file, use the take<\/strong> method on the Iterator before the foreach<\/strong>.<\/p>\n
scala> Source.fromFile(\"pg1661.txt\").getLines.take(5).foreach(println)\r\nProject Gutenberg's The Adventures of Sherlock Holmes, by Arthur Conan Doyle\r\n\r\nThis eBook is for the use of anyone anywhere at no cost and with\r\nalmost no restrictions whatsoever.  You may copy it, give it away or\r\nre-use it under the terms of the Project Gutenberg License included\r\n<\/pre>\n
The take<\/strong> method is quite useful in general with any sequence, and provides the complement of the drop method, as shown in the following examples on a simple List[Int]<\/strong>.<\/p>\n
scala> val numbers = 1 to 10 toList\r\nnumbers: List[Int] = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)\r\n\r\nscala> numbers.take(3)\r\nres12: List[Int] = List(1, 2, 3)\r\n\r\nscala> numbers.drop(3)\r\nres13: List[Int] = List(4, 5, 6, 7, 8, 9, 10)\r\n\r\nscala> numbers.take(3) ::: numbers.drop(3)\r\nres14: List[Int] = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)\r\n<\/pre>\n
     Word counting line by line, first try<\/h2>\n
Now that we\u2019ve seen how to read a file and start working with it line-by-line, how do we count the number of occurrences of each word? Recall from tutorial 7 <\/a>and above that the starting point was to have a sequence (Array, List, etc) of Strings in which each element is a word token. To start moving toward that, we can simply use the toList<\/strong> method on the Iterator[String]<\/strong> obtained from getLines<\/strong>.<\/p>\n
scala> val holmes = Source.fromFile(\"pg1661.txt\").getLines.toList\r\nholmes: List[String] = List(The Project Gutenberg EBook of The Adventures of Sherlock Holmes, by Sir Arthur Conan Doyle, (#15 in our series by Sir Arthur Conan Doyle), \"\", Copyright laws are changing all over the world. Be sure to check the, copyright laws for your country before downloading or redistributing, this or any other Project Gutenberg eBook., \"\", This header should be the first thing seen when viewing this Project, Gutenberg file.  Please do not remove it.  Do not change or edit the, header without written permission., \"\", Please read the \"legal small print,\" and other information about the, eBook and Project Gutenberg at the bottom of this file.  Included is, important information about your specific rights and restrictions in, how the file may be used.  You can also find ou...\r\n<\/pre>\n
We now have the contents of the file as a List[String]<\/strong>, and may proceed to do useful things with it. For example, we could map<\/strong> each line (Strings) to be sequences of whitespace-separated Strings.<\/p>\n
scala> val listOfListOfWords = Source.fromFile(\"pg1661.txt\").getLines.toList.map(x => x.split(\" \").toList)\r\nlistOfListOfWords: List[List[java.lang.String]] = List(List(Project, Gutenberg's, The, Adventures, of, Sherlock, Holmes,, by, Arthur, Conan, Doyle), List(\"\"), List(This, eBook, is, for, the, use, of, anyone, anywhere, at, no, cost, and, with), List(almost, no, restrictions, whatsoever., \"\", You, may, copy, it,, give, it, away, or), List(re-use, it, under, the, terms, of, the, Project, Gutenberg, License, included), List(with, this, eBook, or, online, at, www.gutenberg.net), List(\"\"), List(\"\"), List(Title:, The, Adventures, of, Sherlock, Holmes), List(\"\"), List(Author:, Arthur, Conan, Doyle), List(\"\"), List(Posting, Date:, April, 18,, 2011, [EBook, #1661]), List(First, Posted:, November, 29,, 2002), List(\"\"), List(Language:, English), List(\"\"), List(\"\"), List(***, START, OF, THIS, PRO...\r\n<\/pre>\n
And, as we saw in tutorial 7<\/a>, when we have a List of Lists, we can use flatten<\/strong> to create one big List.<\/p>\n
scala> val listOfWords = listOfListOfWords.flatten\r\nlistOfWords: List[java.lang.String] = List(Project, Gutenberg's, The, Adventures, of, Sherlock, Holmes,, by, Arthur, Conan, Doyle, \"\", This, eBook, is, for, the, use, of, anyone, anywhere, at, no, cost, and, with, almost, no, restrictions, whatsoever., \"\", You, may, copy, it,, give, it, away, or, re-use, it, under, the, terms, of, the, Project, Gutenberg, License, included, with, this, eBook, or, online, at, www.gutenberg.net, \"\", \"\", Title:, The, Adventures, of, Sherlock, Holmes, \"\", Author:, Arthur, Conan, Doyle, \"\", Posting, Date:, April, 18,, 2011, [EBook, #1661], First, Posted:, November, 29,, 2002, \"\", Language:, English, \"\", \"\", ***, START, OF, THIS, PROJECT, GUTENBERG, EBOOK, THE, ADVENTURES, OF, SHERLOCK, HOLMES, ***, \"\", \"\", \"\", \"\", Produced, by, an, anonymous, Project, Gut...\r\n<\/pre>\n
But, now you might recognize that this is the map-then-flatten<\/em> pattern we saw previously, which means we can flatMap<\/strong> it instead.<\/p>\n
scala> val flatMappedWords = Source.fromFile(\"pg1661.txt\").getLines.toList.flatMap(x => x.split(\" \"))\r\nflatMappedWords: List[java.lang.String] = List(Project, Gutenberg's, The, Adventures, of, Sherlock, Holmes,, by, Arthur, Conan, Doyle, \"\", This, eBook, is, for, the, use, of, anyone, anywhere, at, no, cost, and, with, almost, no, restrictions, whatsoever., \"\", You, may, copy, it,, give, it, away, or, re-use, it, under, the, terms, of, the, Project, Gutenberg, License, included, with, this, eBook, or, online, at, www.gutenberg.net, \"\", \"\", Title:, The, Adventures, of, Sherlock, Holmes, \"\", Author:, Arthur, Conan, Doyle, \"\", Posting, Date:, April, 18,, 2011, [EBook, #1661], First, Posted:, November, 29,, 2002, \"\", Language:, English, \"\", \"\", ***, START, OF, THIS, PROJECT, GUTENBERG, EBOOK, THE, ADVENTURES, OF, SHERLOCK, HOLMES, ***, \"\", \"\", \"\", \"\", Produced, by, an, anonymous, Project,...\r\n<\/pre>\n
But you should be a bit bothered by all this: wasn\u2019t the idea here (in part) not to read all of the lines in at once? Indeed, with what we did above, as soon as we said toList<\/strong> on the Iterator, the whole file was read into memory. However, we can do without the toList<\/strong> step and just directly flatMap<\/strong> the Iterator and get a new Iterator over the tokens rather than the lines.<\/p>\n
scala> val flatMappedWords = Source.fromFile(\"pg1661.txt\").getLines.flatMap(x => x.split(\" \"))\r\nflatMappedWords: Iterator[java.lang.String] = non-empty iterator\r\n<\/pre>\n
Now, if we want to count the words, we can convert that to a List and do the groupBy<\/strong> the mapValues<\/strong> trick we\u2019ve seen already (output omitted).<\/p>\n
scala> val counts = Source.fromFile(\"pg1661.txt\").getLines.flatMap(x => x.split(\" \")).toList.groupBy(x=>x).mapValues(x=>x.length)\r\n<\/pre>\n
Oops \u2014 that worked, but we once again brought the whole file into memory because the List that was created from toList<\/strong> has all lines for the file. We\u2019ll see next how to use a mutable<\/em> Map<\/strong> to get around this.<\/p>\n
     Word counting by streaming with an Iterator and using mutable Maps<\/h2>\n
In all of the tutorials so far, I\u2019ve pretty much stuck to immutable data structures except when mutable ones show up due to context (like Arrays coming out of the toString<\/strong> method). It\u2019s good to try to make use of immutable data structures where possible, but there are times when mutable ones are more convenient and perhaps more appropriate.<\/p>\n
With the immutable Maps we saw in the previous tutorial, you could not change the assignment to a key, nor could you add a new key.<\/p>\n
lettersToNumbers: scala.collection.immutable.Map[java.lang.String,Int] = Map(A -> 1, B -> 2, C -> 3)\r\n\r\n1\r\nscala> lettersToNumbers(\"A\") = 4\r\n<console>:9: error: value update is not a member of scala.collection.immutable.Map[java.lang.String,Int]\r\nlettersToNumbers(\"A\") = 4\r\n\r\nscala> lettersToNumbers(\"D\") = 5\r\n<console>:9: error: value update is not a member of scala.collection.immutable.Map[java.lang.String,Int]\r\nlettersToNumbers(\"D\") = 5\r\n<\/pre>\n
There is another kind of Map, scala.collection.mutable.Map<\/strong>, that does allow this sort of behavior.<\/p>\n
scala> import scala.collection.mutable\r\nimport scala.collection.mutable\r\n\r\nscala> val mutableLettersToNumbers = mutable.Map(\"A\"->1, \"B\"->2, \"C\"->3)\r\nmutableLettersToNumbers: scala.collection.mutable.Map[java.lang.String,Int] = Map(C -> 3, B -> 2, A -> 1)\r\n\r\nscala> mutableLettersToNumbers(\"A\") = 4\r\n\r\nscala> mutableLettersToNumbers(\"D\") = 5\r\n\r\nscala> mutableLettersToNumbers\r\nres4: scala.collection.mutable.Map[java.lang.String,Int] = Map(C -> 3, D -> 5, B -> 2, A -> 4)\r\n<\/pre>\n
It also has a handy way to increase the count associated with a key, using the +=<\/strong> method.<\/p>\n
scala> mutableLettersToNumbers(\"D\") += 5\r\n\r\nscala> mutableLettersToNumbers\r\nres6: scala.collection.mutable.Map[java.lang.String,Int] = Map(C -> 3, D -> 10, B -> 2, A -> 4)\r\n<\/pre>\n
However, we can\u2019t use that method with a key that doesn\u2019t exist.<\/p>\n
scala> mutableLettersToNumbers(\"E\") += 1\r\njava.util.NoSuchElementException: key not found: E\r\n<...stacktrace...>\r\n<\/pre>\n
Fortunately, we can provide a default. Here\u2019s an example of starting a new Map with a default of 0.<\/p>\n
scala> val counts = mutable.Map[String,Int]().withDefault(x=>0)\r\ncounts: scala.collection.mutable.Map[String,Int] = Map()\r\n\r\nscala> counts(\"Z\") += 1\r\n\r\nscala> counts(\"Y\") += 1\r\n\r\nscala> counts(\"Z\") += 1\r\n\r\nscala> counts\r\nres11: scala.collection.mutable.Map[String,Int] = Map(Z -> 2, Y -> 1)\r\n<\/pre>\n
Note<\/em>: when you start with some values already in a Map, Scala can infer the types of the keys and the values, but when initializing an empty Map, it is necessary to explicitly declare the key and value types.<\/p>\n
With this in hand, here is how we can use flatMap<\/strong> plus a mutable Map to count words in a text without reading the entire text into memory.<\/p>\n
import scala.collection.mutable\r\nval counts = mutable.Map[String, Int]().withDefault(x=>0)\r\nfor (token <- scala.io.Source.fromFile(\"pg1661.txt\").getLines.flatMap(x =>x.split(\"\\\\s+\")))\r\ncounts(token) += 1\r\n<\/pre>\n
Having created the counts Map in this way, we can convert it to an immutable Map with the toMap<\/strong> method once we are done adding elements.<\/p>\n
scala> val fixedCounts = counts.toMap\r\nfixedCounts: scala.collection.immutable.Map[String,Int] = Map(wood-work, -> 1,\r\n<...output truncated...>\r\n<\/pre>\n
Now we can\u2019t modify the values on fixedCounts<\/em>, which has advantages in many contexts, e.g. we can\u2019t accidentally destroy values or add unwanted keys, and there are (positive) implications for parallel processing.<\/p>\n
scala> fixedCounts(\"Holmes\") = 0\r\n<console>:13: error: value update is not a member of scala.collection.immutable.Map[String,Int]\r\nfixedCounts(\"Holmes\") = 0\r\n^\r\n<\/pre>\n
     Reading a file from a URL<\/h2>\n
As it turns out scala.io.Source<\/strong> can do a lot more than read from a file. Another example is to read from a URL to access a file on the internet, using the fromURL<\/strong> method.<\/p>\n
val holmesUrl = \"\"\"http:\/\/www.gutenberg.org\/cache\/epub\/1661\/pg1661.txt\"\"\"\r\nfor (line <- Source.fromURL(holmesUrl).getLines)\r\nprintln(line)\r\n<\/pre>\n
If you are just going to analyze the same file again and again, this is probably not what you need \u2014 just download the file and use it locally. However, it can be quite useful in contexts where you are exploring links within pages (e.g. while processing Wikipedia or Twitter data) and need to read in content from URLs on the fly.<\/p>\n
     Use (up) the Source<\/h2>\n
A final note on the Iterators you get with Source.fromFile and Source.fromURL: you can only iterate through them once! This is part of what makes them more efficient \u2014 they aren\u2019t holding all thattext in memory. So, don\u2019t be surprised if you get the following behavior.<\/p>\n
scala> val holmesIterator = Source.fromFile(\"pg1661.txt\").getLines\r\n holmesIterator: Iterator[String] = non-empty iterator\r\n\r\nscala> holmesIterator.foreach(println)\r\n\r\nProject Gutenberg's The Adventures of Sherlock Holmes, by Arthur Conan Doyle\r\n\r\nThis eBook is for the use of anyone anywhere at no cost and with\r\n almost no restrictions whatsoever.  You may copy it, give it away or\r\n re-use it under the terms of the Project Gutenberg License included\r\n with this eBook or online at www.gutenberg.net\r\n\r\n<...many lines of output...>\r\n\r\nThis Web site includes information about Project Gutenberg-tm,\r\n including how to make donations to the Project Gutenberg Literary\r\n Archive Foundation, how to help produce our new eBooks, and how to\r\n subscribe to our email newsletter to hear about new eBooks.\r\n\r\nscala> holmesIterator.foreach(println)\r\n\r\n<...nothing output!...>\r\n<\/pre>\n
So, the Iterator is used up! If you want to go through the file again, you\u2019ll need to spin up a new Iterator just like you did the first time around. The neat thing about staying with the Iterators and not converting to Lists (and thus bringing everything into memory) is that each mapping operation we do on the Iterator applies only for the current item we are looking at, so we never need to read the whole file into memory.<\/p>\n
Of course, if you have a reasonably small file to work with, you should feel absolutely free to toList<\/strong> it and work with it that way if you prefer \u2014 it will often be more convenient since you can do the groupBy<\/strong> and mapValue<\/strong> pattern.<\/p>\n
Reference: <\/i><\/strong>First steps in Scala for beginning programmers, Part 8<\/a> from our JCG partner<\/a> Jason Baldridge at the Bcomposes<\/a> blog.<\/p>\n
Related Articles :<\/i><\/strong><\/p>\n