For Comprehensions

The last chapter focused on monads and the map and flatMap functions. In this chapter we’re going to focus on just flatMap behaviour. Specifically, we’ll look at how to chain flatMap function calls before finally yielding results. For comprehensions actually use flatMap under the hood, so we’ll look at the relationship in detail and explain how for comprehensions work.

Where We Left Off

Hopefully you’re now comfortable with the idea of flatMap. We looked at it for the collection classes and for Option. Recall that we used flatMap to map over customer names that may or may not exist in our database. By doing so, we could sum customer basket values.

  customers     .flatMap(name => database.find(name))     .map(customer => customer.total)     .sum 

Now let’s say that we’d like to generate a shipping label for a customer. We can look up a customer in our repository and if they have a street address and a postcode, we can generate a shipping label.

The caveats are:

1. A customer may or may not exist in the repository.
2. A given customer may or may not have an address object.
3. An address object must contain a street but may or may not contain a postcode.

So, to generate a label, we need to:

1. Find a customer (who may or may not exist) by name.
2. Get the customer’s address (which also may or may not exist).
3. Given the address, get the shipping information from it. (We can expect an Address object to contain a street address, but it may or may not have a postcode.)

Using Null Checks

If we were to implement this where the optionality was expressed by returning nulls, we’d be forced to do a bunch of null checks. We have four customers: Albert, Beatriz, Carol, and Sherlock. Albert has an address but no postcode, Beatriz hasn’t given us her address, and the last two have full address information.

  val customers = Customers()    val address1 = Some(Address("1a Bridge St", None))   val address2 = Some(new Address("2 Short Road", Some("AL1 2PY")))   val address3 = Some(new Address("221b Baker St", Some("NW1")))    customers.add(new Customer("Albert", address1))   customers.add(new Customer("Beatriz", None))   customers.add(new Customer("Carol", address2))   customers.add(new Customer("Sherlock", address3)) 

Given a list of customers, we can attempt to create shipping labels. As you can see, the list below includes people that don’t exist in the database.

  val all = Set("Albert", "Beatriz", "Carol", "Dave", "Erin", "Sherlock") 

Next, we create a function to return the list of shipping labels, collecting them in a mutable set. For every name in our list, we attempt to find the customer in the database (using customers.find). As this could return null, we have to check the returned value isn’t null before we can get their address.

Getting the address can return null, so we have to check for null again before getting their postcode. Once we’ve checked the postcode isn’t null, we can finally call a method (shippingLabel) to create a label and add it to the collection. Were we to run it, only Carol and Sherlock would get through all the null checks.

  def generateShippingLabels() = {     val labels = mutable.Set[String]()     all.foreach { name =>       val customer: Customer = customers.find(name)       if (customer != null) {         val address: Address = customer.address         if (address != null) {           val postcode: String = address.postcode           if (postcode != null) {             labels.add(                 shippingLabel(customer.name, address.street, postcode))           }         }       }     }     labels   }      def shippingLabel(name: String, street: String, postcode: String) = {     "Ship to:\n" + "========\n" + name + "\n" + street + "\n" + postcode   } 

Using flatMap with Option

If, instead of returning null for no customer, we were to use Option as the return type, we could reduce the code using flatMap.

 1   def generateShippingLabel(): Set[String] = {  2     all.flatMap {  3       name => customers.find(name).flatMap {  4         customer => customer.address.flatMap {  5           address => address.postcode.map {  6             postcode => {  7               shippingLabel(customer.name, address.street, postcode)  8             }  9           } 10         } 11       } 12     } 13   } 14  15   def shippingLabel(name: String, street: String, postcode: String) = { 16     "Ship to:\n" + "========\n" + name + "\n" + street + "\n" + postcode 17   } 

We start in the same way as before, by enumerating each of the names in our list, calling find on the database for each. We use flatMap to do this as we’re transforming from a single customer name (String) to a monad (Option).

You can think of the option as being like a list with one element in it (either a Some or a None), so we’re doing a “one-to-many”-like transformation. As we saw in the , this implies we’ll need to flatten the “many” back down into “one” later, hence the flatMap.

After the initial flatMap where we find a customer in the database, we flatMap the result. If no customer was found, it wouldn’t continue any further. So on line 4, we can be sure a customer actually exists and can go ahead and get their address. As address is optional, we can flatMap again, dropping out if a customer doesn’t have an address.

On line 5, we can request a customer’s postcode. Postcode is optional, so only if we have one do we transform it (and the address details) into a shipping label. The map call takes care of that for us; remember that map here only applies the function (shippingLabel) when we have a value (i.e., postcode is an instance of Some).

Notice that we didn’t need to create a mutable collection to store the shipping label. Any transformation function like map or flatMap will produce a new collection with the transformed results. So the final call to map on line 7 will put the shipping label into a newly created collection for us. One final comment: the resulting collection is of type String because the generateShippingLabel method returns a String.

How For Comprehensions Work

When you do a regular for loop, the compiler converts (or de-sugars) it into a method call to foreach.

  for (i <- 0 to 5) {     println(i)   }    // is de-sugared as    (0 to 5).foreach(println) 

A nested loop is de-sugared like this:

  for (i <- 0 to 5; j <- 0 to 5) {     println(i + " " + j)   }    // is de-sugared as    (0 to 5).foreach { i =>     (0 to 5).foreach { j => {         println(i + " " + j)       }     }   } 

If you do a for with a yield (a for comprehension) the compiler does something different:

  for (i <- 0 to 5) yield {     i + 2   } 

The yield is about returning a value. A for without a yield, although an expression, will return Unit. This is because the foreach method returns Unit. A for with a yield will return whatever is in the yield block. It’s converted into a call to map rather than foreach. So, the de-sugared form of the above would look like this:

  // de-sugared form of "for (i <- 0 to 5) yield i + 2"   (0 to 5).map(i => i + 2) 

It’s mapping a sequence of numbers (0 to 5) into another sequence of numbers (2 to 7).

It’s important to realise that whatever is in the yield block represents the function that’s passed into map. The map itself operates on whatever is in the for part (i.e., for (i <- 0 to 5)). It may be easier to recognise when we reformat the example above like this:

  for {     i <- 0 to 5           // map operates on this collection   } yield {     i + 2                 // the function to pass into map   } 

It gets more interesting when we have nesting between the parentheses and the yield.

  val x: Seq[(Int, Int)] = for {     i <- 0 to 5     j <- 0 to 5   } yield {     (i, j)   }   println(x) 

This will perform the nested loop like before but rather than translate to nested foreach calls, it translates to flatMap calls followed by a map. Again, the final map is used to transform the result using whatever is in the yield block.

  // de-sugared   val x: Seq[(Int, Int)] = (0 to 5).flatMap {     i => (0 to 5).map {       j => (i, j)     }   } 

It’s exactly the same as before; the yield block has provided the function to apply to the mapping function and what it maps over is determined by the for expression. In this example, we’re mapping two lists of 0 to 5 to a collection of tuples, representing their Cartesian product.

  Seq((0,0), (0,1), (0,2), (0,3), (0,4), (0,5),       (1,0), (1,1), (1,2), (1,3), (1,4), (1,5),       (2,0), (2,1), (2,2), (2,3), (2,4), (2,5),       (3,0), (3,1), (3,2), (3,3), (3,4), (3,5),       (4,0), (4,1), (4,2), (4,3), (4,4), (4,5),       (5,0), (5,1), (5,2), (5,3), (5,4), (5,5)) 

If we break this down and go through the steps, we can see how we arrived at the de-sugared form. We start with two sequences of numbers; a and b.

  val a = (0 to 5)   val b = (0 to 5) 

When we map the collections, we get a collection of collections. The final map returns a tuple, so the return type is a sequence of sequences of tuples.

  val x: Seq[Seq[(Int, Int)]] = a.map(i => b.map(j => (i, j))) 

To flatten these to a collection of tuples, we have to flatten the two collections, which is what flatMap does. So although we could do the following, it’s much more straightforward to call flatMap directly.

  val x: Seq[(Int, Int)] = a.map(i => b.map(j => (i, j))).flatten    // is equivalent to    val x: Seq[(Int, Int)] = a.flatMap(i => b.map(j => (i, j))) 

Finally, Using a For Comprehension for Shipping Labels

What does all that mean for our shipping label example? We can convert our chained flatMap calls to use a for comprehension and neaten the whole thing up. We started with a sequence of chained calls to flatMap.

  def generateShippingLabel_FlatMapClosingOverVariables(): Set[String] = {     all.flatMap {       name => customers.find(name).flatMap {         customer => customer.address.flatMap {           address => address.postcode.map {             postcode => shippingLabel(name, address.street, postcode)           }         }       }     }   } 

After converting to the for comprehension, each call to flatMap is placed in the for as a nested expression. The final one represents the map call. Its argument (the mapping function) is what’s in the yield block.

  def generateShippingLabel_ForComprehension(): Set[String] = {     for {       name <- all                                    // <- flatMap       customer <- customers.find(name)               // <- flatMap       address <- customer.address                    // <- flatMap       postcode <- address.postcode                   // <- map     } yield {       shippingLabel(name, address.street, postcode)  // <- map argument     }   } 

This is much more succinct. It’s easier to reason about the conditional semantics when it’s presented like this; if there’s no customer found, it won’t continue to the next line. If you want to extend the nesting, you can just add another line and not be bogged down by noise or indentation.

The syntax is declarative but mimics an imperative style. It doesn’t force a particular implementation on you. That’s to say, with imperative for loops, you don’t have a choice about how the loop is executed. If you wanted to do it in parallel, for example, you’d have to implement the concurrency yourself.

Using a declarative approach like this means that the underlying objects are responsible for how they execute, which gives you more flexibility. Remember, this just calls flatMap and classes are free to implement flatMap however they like.

For comprehensions work with any monad and you can use your own classes if you implement the monadic methods.

Summary

In this chapter we looked at chaining calls to flatMap in the context of printing shipping labels. We looked at how for comprehensions work and how they’re syntactic sugar over regular Scala method calls. Specifically, we looked at loops and nested loops, how they’re equivalent to calling foreach, how for loops with a yield block translate to mapping functions, and how nested loops with yield blocks translate to flatMap then map functions.

This last point is what allowed us to convert our lengthy shipping label example into a succinct for comprehension.