Lunatech's engineer blog

An Introduction to Finagle by example

Antoine Laffez — Mon, 28 Nov 2016 00:00:00 GMT

It is a challenging task to build a large-scale web application, there are fundamental characteristics to take into account: for example, efficiency, safety and robustness. Finagle is a asynchronous, Netty based JVM RPC system made by Twitter which makes it easy to build high available clients and servers in Java and Scala. And it can even simplify your application architecture. Here I want to show you how powerful Finagle is.

Quickstart

Let’s first have a quick look about how to create a Finagle micro web service and a Finagle http client to consume this api.Create a sbt project and import dependencies.

libraryDependencies ++= Seq(
"com.twitter" %% "finagle-http" % "6.38.0",
"org.scalatest" %% "scalatest" % "2.2.4" % "test"
)

First, let’s define a service. Here we define a service to receive a http request and get its url parameter as Integer then return a http response by plus 10.

import com.twitter.finagle.Serviceimport
import com.twitter.util.Futureimport
import com.twitter.finagle.http

// This is a plus 10 service
class PlusTenService extends Service[http.Request, http.Response] {

  override def apply(request: http.Request): Future[http.Response] = {
  	Future {
  		val input = request.getIntParam("num")
  		val output = input + 10
  		val response = http.Response(request.version, http.Status.Ok)
        response.setContentString(output.toString)
        response
    }
  }
}

Then initiate and start our server

import com.twitter.finagle.{http, Service, Http}
import com.twitter.util.Await

object QuickLookServer {
	def main(args: Array[String]): Unit = {
    val service: Service[http.Request, http.Response] = new PlusTenService    	  val server = Http.serve(":9090", service)
    Await.ready(server)
    }
}

Last let’s define a client to consume this server.

import com.twitter.finagle.{Service, Http}
import com.twitter.finagle.http
import com.twitter.util.Await

object QuickLookClient {
	def main(args: Array[String]): Unit = {
    	//define a client
    	val client: Service[http.Request, http.Response] = Http.newService("localhost:9090")
		//define a request
		val request = http.Request(http.Method.Get, "/?num=5")
		//apply request on the client
		val response = client(request)
		// print response
		response.foreach(rep => println(rep.getContentString()))
		Await.result(response)
    }
}

If you run the two application you will see the server running on localhost:9090 and client get response 15. Simple right? As you can see our service and client are both type of Service[http.Request, http.Response] . This data type really confuse me in the beginning. I will explan what’s the differences between them.

The core of Finagle

Service

Now let’s first have a look at the core of finagle Service[-Req, +Rep] . You can find the definition in com.twitter.finagle.Service . In Finagle 6.38.0 the definition of Service is an abstract class, in previous version it was a trait

abstract class Service[-Req, +Rep] extends (Req => Future[Rep])

A service is a function that takes request of type Req, and return a response of Future of Rep. This Services type are used to represent both clients and servers. To answer my previous question, the differences between service and client is that a Finagle client “imports” a Service from the network. However, a Finagle server “exports” a Service to the network.Note: the Future here is twitter future not scala future. There is no differences on conception.

Filter

Some times we want to add application-agnostic behaviour, we can use Filter to achieve this.

abstract class Filter[-ReqIn, +RepOut, +ReqOut, -RepIn]
extends ((ReqIn, Service[ReqOut, RepIn])
=> Future[RepOut])

If it is not clear please check image below.

In most common cases, ReqIn is equal to ReqOut, and RepIn is equal to RepOut. So we have this SimpleFilter class

abstract class SimpleFilter[Req, Rep] extends Filter[Req, Rep, Req, Rep]

A filter can attached to client and server side. Let’s try to implement a simple timeout filter.

import com.twitter.finagle.{SimpleFilter, Service}import com.twitter.util.{Duration, Timer, Future}

class TimeoutFilter[Req, Rep](timeout: Duration, timer: Timer)
extends SimpleFilter[Req, Rep] {

  def apply(request: Req, service: Service[Req, Rep]): Future[Rep] = {
  val res = service(request)
  res.within(timer, timeout)  }
  }

Here, a timeout filter is a class extends SimpleFilter trait. Below is how to use this filter on client side

client = Http.newService("localhost:9090")

val timeoutFilter = new TimeoutFilter[http.Request, http.Response](Duration.fromSeconds(1),
new JavaTimer(false))

val clientWithTimeoutFilter = timeoutFilter.andThen(quickLookClient)

A filter can be applied on server side too. Here is an example. First let’s define a filter.

class CountFilter[Req, Rep](countClient: Service[http.Request, http.Response])
extends SimpleFilter[Req, Rep]
{  override def apply(request: Req, service: Service[Req, Rep]): Future[Rep] =
{    val countRequest = http.Request(http.Method.Post, "/?count=5")    countClient(countRequest)    service(request)  }}

And then let’s use it on our plusTen service

al service: Service[http.Request, http.Response] = new PlusTenService

val countClient = Http.newService("localhost:9010")

val countFilter = new CountFilter[http.Request, http.Response](countClient)

val serviceWithCountFilter = countFilter.andThen(service)

You may notice the way to chain filter and service together is by using andThen method. Actually andThen method can not only chain filter with service but also chain multiple filters, like filter1 andThen filter2 andThen myservice

Client

This is the part that I like the most in finagle. Finagle http client is designed to maximize success and minimize latency. Each request will flow through various modules. These modules are logically separated into three stacks: Client stack, Endpoint stack, connection stack.

Client stack

manages name resolution and balances requests across multiple endpoints.

Endpoint stack

provides circuit breakers and connection pooling.

connection stack

provides connection life-cycle management and implements the wire protocol.

To use finagle http client is very simple. Define a client first and define a http request, then apply request on the client.

// create a http clientval client =
Http.client.newService("example.com:80")
// create a http requestval req =
Request("/foo", ("my-query-string", "bar")
)
// apply request on the clientval resp: Future[Response] = client(req)Note: client(req) is equal to client.apply(req)

What I want to emphasis here is the Load Balancer module. This module brings a lot of benefit for your application. It can simplify your application infstracture. Let’s compare it with traditional solution.

image

As you can see, the traditional solution highly rely on nginx as load balancer, once nginx dead your service is not reachable, in real production environment, you have master-slave nginx wiht keeplived installed on nginx machine for heartbeat detection. This looks really complex, what about if we can get rid of these nginx?Let’s have look at following code.

name: Name =
Name.bound(Address("localhost", 10010), Address("localhost", 10011), Address("localhost", 10012)
)
//define a clientval client: Service[http.Request, http.Response] = Http.newService(name, "client")

This means you supply three addresses and put it into finagle http client. Finagle client will dispatch the request to one of address based on certain load balance algorithmn. The default algorithmn is "Exponentially Weighted Moving Average (EWMA)". Now your infstracture architechture becomes like following

image

Pretty simple right. Your apis talk to each other directly.

Protocol-agnostic

Finagle is a protocol-agnostic RPC system. It means Finagle supports every protocol if people implement it. For example: finagle-thrift is using thrift protocol. finagle-mysql implements the mysql protocol.Now, let’s look at this scenario

image

We want to make a api count service to count how many times the web service has been called. In section Service and Filter. We send http request and put number as query parameter. It just feel strange that I just want to send a number to count server, to achieve that I have to send a http request. Because I don’t use any data from header, cookie and body. If the application is running on AWS, it those junk information cost money. So it’s ideal to just send a integer number to api count service. Let’s implement this by customize finagle protocol.First, we should tell finagle how to converts an scodec codec into a Netty encoder

import org.jboss.netty.buffer.{ChannelBuffer, ChannelBuffers}import org.jboss.netty.channel.{Channel, ChannelHandlerContext}import org.jboss.netty.handler.codec.oneone.{OneToOneDecoder, OneToOneEncoder}import scodec.Codecimport scodec.bits.BitVector

trait CodecConversions {  /**    * Converts an scodec codec into a Netty encoder.    */  protected def encoder[A: Codec] =
new OneToOneEncoder {
override def encode(ctx: ChannelHandlerContext, channel: Channel, msg: Object) =

ChannelBuffers.wrappedBuffer(        Codec.encodeValid(msg.asInstanceOf[A]).toByteBuffer      )
}

  /**    * Converts an scodec codec into a Netty decoder.
  */  protected def decoder[A: Codec] = new OneToOneDecoder {
  override def decode(ctx: ChannelHandlerContext, channel: Channel, msg: Object) =
  msg match {
  case cb: ChannelBuffer =>          Codec.decodeValidValue[A](BitVector(cb.toByteBuffer)).asInstanceOf[Object]        case other => other      }
  }
  }

And then channel pipeline and codec factories

trait Factories { this: CodecConversions =>  import com.twitter.finagle.{Codec => FinagleCodec, CodecFactory}  import org.jboss.netty.channel.{ChannelPipelineFactory, Channels}

  /**   * Creates a Netty channel pipeline factory given input and output types.   */  private[this] def pipeline[I: Codec, O: Codec] = new ChannelPipelineFactory {    def getPipeline = {      val pipeline = Channels.pipeline()      pipeline.addLast("encoder", encoder[I])      pipeline.addLast("decoder", decoder[O])
  pipeline    }
  }
  /**   * Creates a Finagle codec factory given input and output types.   */  protected def codecFactory[I: Codec, O: Codec] = new CodecFactory[I, O] {
  def server = Function.const {
  new FinagleCodec[I, O] { def pipelineFactory = pipeline[O, I] }
  }
    def client = Function.const {
    new FinagleCodec[I, O] { def pipelineFactory = pipeline[I, O] }
    }
    }
    }

And then the code that actually creates our Finagle server and client

import java.net.InetSocketAddress

import com.twitter.conversions.time._import com.twitter.finagle.Serviceimport com.twitter.finagle.builder.{ClientBuilder, ServerBuilder}import com.twitter.util.{Duration, Future}import scodec.Codec

object IntegerServerAndClient extends Factories with CodecConversions {

  /**    * Creates a Finagle server from a service that we have scodec codecs    * for both the input and output types.    */  def server[I, O](port: Int)(service: Service[I, O])(implicit ic: Codec[I], oc: Codec[O]) =    ServerBuilder()
  .name("server")
  .codec(codecFactory[I, O])
  .bindTo(new InetSocketAddress(port))      .build(service)

  /**    * Creates a Finagle client given input and output types with scodec codecs.    */  def client[I, O](host: String, timeout: Duration = 3.second)                  (implicit ic: Codec[I], oc: Codec[O]) =    ClientBuilder()
  .name("client")
  .codec(codecFactory[I, O])
  .hosts(host)
  .timeout(timeout)
  .build()
  }

Define our simple service

import com.twitter.finagle.Serviceimport com.twitter.util.Future

class IntegerService extends Service[Int, Int]{  var count = 0  override def apply(request: Int): Future[Int] = {    Future.value(count + request)  }
}

Run a server

import com.twitter.finagle.Serviceimport com.twitter.util.Awaitimport scodec.codecs.implicits.{ implicitIntCodec => _, _ }

object Server {  def main(args: Array[String]): Unit = {    implicit val intgerCodec =
scodec.codecs.uint8

    val service: Service[Int, Int] =
    new IntegerService
    val server = IntegerServerAndClient.server[Int, Int](9191)(service)    Await.ready(server)
    }
    }

Run a client

import com.twitter.finagle.Serviceimport com.twitter.util.Awaitimport scodec.codecs.implicits.{ implicitIntCodec => _, _ }

object Client {  def main(args: Array[String]): Unit = {

    implicit val intgerCodec = scodec.codecs.uint8

    //define a client
    val client: Service[Int, Int] = IntegerServerAndClient.client[Int, Int]("localhost:9191")    //define a request
    val request = 4
    //apply request on the client
    val response = client(request)
    //print response    response.foreach(rep => println(s"This is response $rep"))
    Await.result(response)
    }
    }

Conclusion

Finagle is a very flexible asychronous, protocol-agnostic RPC framework. It can help you to build high performance micro service with any protocol. It is worth to take a look at Finch the web framework based on Finagle. You can find more detail introduction from Twitter blog and more detailed example from Twitter scala school.

Functional IO with FS2 Streams

Antoine Laffez — Mon, 14 Nov 2016 00:00:00 GMT

Functional IO with FS2 Streams

One of the main principles of functional programming is to avoid side-effects. For the most part, working with immutable instances would be sufficient to satisfy that principle. But sometimes it’s needed to do some side effects e.g. when we want to read from file. One straight-forward way of doing it is to get a BufferedSource, and process it using an iterator as shown below.

def readFromFile(fileName: String) {
val src: BufferedSource = io.Source.fromFile(fileName)
val it: Iterator[String] = src.getLines
val _ = while(it.hasNext) {
val line = it.next()
println(line)
}
src.close()
}

Then it is not too difficult to recursively convert the iterator to a Scala Stream of lines, without processing the whole iterator, using lazy evaluation. So we would end up with a function with signature:

   def read(pathToFile: Path): Stream[String]

There are few problems that can arise from this function:

First, if an exception occurs and we still haven’t reached the end of the resulting Stream, or if we forget to close the BufferedSource, then the file would stay open, which is called resource leaking. Therefore, the above function is not resource-safe.- Second and more tricky, when we call the function read for a 2nd time, it may return a different result as someone may have modified the file between the two calls. We are forced to have some intrinsic knowledge about the file, therefore, the function read breaks referential transparency.

In this blog-post we are going to tackle those two problems and learn how to avoid imperative troublesome IO using a specialized Scala library - FS2 (previously known as Scalaz-Stream).

The Idea of Functional Streams

The idea for solving the problem of resource-safety is to define a wrapper data type - Process[F[],O], where the left generic type F would know how to close the resource (the file) when it’s needed. In that way we can guarantee resource-safety without having to do anything non-related to reading from file (such as handling an exception). We can think of Process[F[],O] as a type that represents a Stream builder, possibly producing/using some side effect. For example when reading from file we would work with Process[Task, String], where Task knows how to execute and handle an operation (by assigning it to a thread). But note that it doesn’t start the execution immediately. It only starts when we call some of its executor-methods. Another example is when writing to output stream, in which case we would use Process[Task, Unit]. Hence, we can say Process[F[_],O] is like a function F[O] ⇒ Stream[O], but more powerful, and we will see exactly how is it more powerful.

The idea for solving the second problem is to separate the logic of computation from actual computation. Note that the logic of how reading from file is done doesn’t change through the course of our program’s lifecycle. Only the result of executing that logic changes (since it involves a side-effect). So we are going to say that the left type - F[O] in Process knows how to evaluate effects. In that way we can represent our logic of computation, and delegate the responsibilities of executing that logic (and handling possible exceptions of the execution) to F. Most often we would substitute the generic type F with Task (which in addition allows concurrent composition of streams).

Right now it may seem too much is going on. But as with every other paradigm, once we get a hands-on experience, we are going to feel comfortable working with it. The important conceptual thing to remember is that the left type F is responsible for closing resources and assigning operations to threads. In what follows, we are going to see examples of functional streams using the library FS2, a library that implements the above ideas.

How to Use FS2

This is how we use FS2 for reading a file…

First we must add its dependencies:

"co.fs2" %% "fs2-core" % "0.9.1",
"co.fs2" %% "fs2-io" % "0.9.1",

Then, similar to Scala’s Future, we must provide a strategy for execution:

implicit val strategy = Strategy.fromFixedDaemonPool(4)

Then we define the pipeline of transformations:

val lines: Stream[Task, String] =
io.file.readAllAsync[Task](pathToFile, 4096)
.through(text.utf8Decode)
 .through(text.lines)
----
val lines: Stream[Task, String] =	io.file.readAllAsync[Task](pathToFile, 4096)` - we get a Stream[Task, Byte].through(text.utf8Decode)` - we get a Stream[Task, String], but newline is disregarded.through(text.lines)` - we get the final Stream[Task, String] representing lines.

Stream[F[], O] is the FS2 implementation of what we denoted by Process[F[],O].

Note that the above code doesn’t read anything. We postpone it as much as possible, usually until the end of the world (our main method). At that point we must do two things:

First, we compile the pipeline of transformations, combining all the intermediate Tasks into a single Task. We do that by calling lines.run, which gives us a` Task[Unit].- Second, we execute the pipeline by calling `lines.run.unsafeRun(), so we end up with a single result-value, in this case Unit.

In addition, Stream[F[], O] is also a Monad, so we can do almost everything that we can do with a Seq. That is one argument why the FS2 is much more powerful than a builder-function F[] ⇒ O.

Pulling

Another very useful and powerful functionality that FS2 supports is pulling. Sometimes we don’t want to map over all the elements of an FS2Stream , but halt the process of mapping and end up with a smaller FS2Stream. We can implement that with the method pull defined on FS2Stream:

.pull[Task, Path](using: (Handle[Task, Path]) => Pull[Task, Path, Nothing]))

Note that we wrote FS2Stream, so to avoid mixing it with the standard Scala Stream.

Now comes the tricky part - how to define the auxiliary function using. The simplest way to explain that function is to say Handle knows how to retrieve the next element in a FS2Stream, and Pull knows how to pick up elements that we want to select and output as a side-effect of the pulling.

For example, imagine we want to output elements produced in the process of iterating, but produce element only in some steps of the iteration. Then we can do the following:

def using(): (Handle[Task, A], accumulator: FS2Stream[Task,
B]) => Pull[Task, A, Nothing] = {  newHandle: Handle[Task, A]

=>    val nextPull: Pull[Task, A, Handle[Task, A]] =
for {      (nextElement: A, newHandle: Handle[Task, A])
<- newHandle.await1

updatedAcc = ...

//update the accumulator and pass back the updated one      _ <- someCheckingOfA match {
case ... =>          Pull.pure(())
//nothing to pick up         case ... =>          Pull.output1(something)
//pick up something
}
} yield (nextHandle, updatedAcc)

    nextPull.flatMap((nextHandle: Handle[Task, A]) =>      using()(nextHandle)    )
    }`

, and we would apply using() as follows:

someFS2Stream.pull[Task, A]((handle: Handle[Task, A])
=>                             using()(handle, emptyAccumulator)
)

Conclusion

FS2 is a masterpiece library that allows us to work with side effects in a resource-safe, consistent, and memory-efficient way. It does all that in a fully functional and composable way, and on top of all that, it supports concurrency. In another blog-post I am going to explain how we can do concurrent computations with FS2.

Useful Links

Functional IO with FS2 Streams

Nicolas Leroux — Mon, 14 Nov 2016 00:00:00 GMT

Functional IO with FS2 Streams

def readFromFile(fileName: String) {
	val src: BufferedSource = io.Source.fromFile(fileName)
	val it: Iterator[String] = src.getLines
	val _ = while(it.hasNext) {
		val line = it.next()
		println(line)
	}
	src.close()
}

   def read(pathToFile: Path): Stream[String]

There are few problems that can arise from this function:

First, if an exception occurs and we still haven’t reached the end of the resulting Stream, or if we forget to close the BufferedSource, then the file would stay open, which is called resource leaking. Therefore, the above function is not resource-safe.- Second and more tricky, when we call the function read for a 2nd time, it may return a different result as someone may have modified the file between the two calls. We are forced to have some intrinsic knowledge about the file, therefore, the function read breaks referential transparency.

In this blog-post we are going to tackle those two problems and learn how to avoid imperative troublesome IO using a specialized Scala library - FS2 (previously known as Scalaz-Stream).

The Idea of Functional Streams

How to Use FS2

This is how we use FS2 for reading a file…

First we must add its dependencies:

"co.fs2" %% "fs2-core" % "0.9.1",
"co.fs2" %% "fs2-io" % "0.9.1",

Then, similar to Scala’s Future, we must provide a strategy for execution:

implicit val strategy = Strategy.fromFixedDaemonPool(4)

Then we define the pipeline of transformations:

val lines: Stream[Task, String] =
	io.file.readAllAsync[Task](pathToFile, 4096)
						.through(text.utf8Decode)
 						.through(text.lines)

val lines: Stream[Task, String] = io.file.readAllAsync[Task](pathToFile, 4096) - we get a Stream[Task, Byte].through(text.utf8Decode) - we get a Stream[Task, String], but newline is disregarded.through(text.lines) - we get the final Stream[Task, String] representing lines.

Stream[F[], O] is the FS2 implementation of what we denoted by Process[F[],O].

Note that the above code doesn’t read anything. We postpone it as much as possible, usually until the end of the world (our main method). At that point we must do two things:

First, we compile the pipeline of transformations, combining all the intermediate Tasks into a single Task. We do that by calling lines.run, which gives us a Task[Unit].- Second, we execute the pipeline by calling lines.run.unsafeRun(), so we end up with a single result-value, in this case Unit.

Pulling

.pull[Task, Path](using: (Handle[Task, Path]) => Pull[Task, Path, Nothing]))

Note that we wrote FS2Stream, so to avoid mixing it with the standard Scala Stream.

For example, imagine we want to output elements produced in the process of iterating, but produce element only in some steps of the iteration. Then we can do the following:

def using(): (Handle[Task, A], accumulator: FS2Stream[Task, B]) => Pull[Task, A, Nothing] = {
	newHandle: Handle[Task, A] =>
    	val nextPull: Pull[Task, A, Handle[Task, A]] =
          for {
              (nextElement: A, newHandle: Handle[Task, A]) <- newHandle.await1   			  updatedAcc = ...
              //update the accumulator and pass back the updated one
              _ <- someCheckingOfA match {
                      case ... => Pull.pure(())
                      //nothing to pick up
                      case ... =>  Pull.output1(something)
                      //pick up something
              }
          } yield (nextHandle, updatedAcc)
        nextPull.flatMap((nextHandle: Handle[Task, A]) => using()(nextHandle))
}

, and we would apply using() as follows:

someFS2Stream.pull[Task, A]((handle: Handle[Task, A]) => using()(handle, emptyAccumulator))

Conclusion

Useful Links

Side Effects and How To Deal With Them The Cool Way, Part 2: Monads Introduction

Nicolas Leroux — Mon, 31 Oct 2016 00:00:00 GMT

In part 1 of this series we learnt how to decouple the code that deals with side effects from our pure functions by using functors (Any type constructor which implements the map function). Now lets see how we can extend our functors so that we can sequence computations that each fall under the indeterminism of side effects.

Function Composition And a Slight Issue

Remember that we mentioned that the Option[A] type in the standard library of scala is a functor, that actually handles the side effect of possible missing values. Now lets create two pure functions to compute integers:

val f: Int => Int =
x => x + 5

val g: Int => Boolean =
x => x > 10

val gof: Int => Boolean = g compose f

gof(6)
> true

The essence of functional programming is doing this, create programs from composing small functions into larger functions, which provide us with the type safety and the determinism of pure functions. but what happens when we have functions that require to handle a side effect and return the appropriate type constructor?

val f: Int => Int =
x => x + 5

val filter10: Int => Option[Int] =
x => if (x > 10) Some(x) else None

val gof: Int => Boolean = f compose filter10>
error: type mismatch;
found   : Int => Option[Int] required: ? => Int
f compose filter

On this case our map function from our functor can help us:

val sumIfMoreThan10: Int => Option[Int] =	x => filter10(x).map(f)

sumIfMoreThan10(15)>
Option[Int] = Some(20)

sumIfMoreThan10(9)>
Option[Int] = None

Great the map function is allowing a type of function composition, but now what happens when we want to compose more functions that return our functor?

val f: Int => Int =
x => x + 5

val filter10: Int => Option[Int] =
x => if (x > 10) Some(x) else None

val sumIfMoreThan10: Int => Option[Int] =
x => filter10(x).map(f)

val sumIfPositive: Int => Option[Int] =
x => if (x > 0) Some(f(x)) else None

val total: Int => Option[Int] =
x => sumIfPositive(6).map(sumIfMoreThan10)
> error: type mismatch;
found   : Int => Option[Int]
required: Int => Int
val total: Int => Option[Int] =
x => sumIfPositive(6).map(sumIfMoreThan10)

Oh no, we have a problem, we cannot compose side effects only using map, we need a new mechanism for combination. Lets do it then! lets introduce a function similar to map but called flatMap! (Because it will 'flatten' the structure between two functors.)

Monads! The Combinator We Were Looking For

trait Option[+A] {
def value: A

	def isDefined: Boolean

	def map[B](f: A=>B): Option[B] =
    flatMap(x => Some(f(x)))

	def flatMap[B](f: A=>Option[B]): Option[B] =
    if(isDefined) f(value) else None
    }

As you may have already guessed, the scala library already implements this function, but lets analyse it any way: flatMap is very similar to our last definition to map, but since the provided effectful function f is already giving us an Option then we do not need to wrap it. Also notice that we redefined map in terms of our new flatMap, clever right?

Now this new mechanism will allow us to combine effectful functions! (functions that return a functor instead of a pure value).

val f: Int => Int =
x => x + 5
val filter10: Int => Option[Int] =
x => if (x > 10) Some(x) else None
val sumIfMoreThan10: Int => Option[Int] =
x => filter10(x).map(f)
val sumIfPositive: Int => Option[Int] =
x => if (x > 0) Some(f(x)) else None
val total: Int => Option[Int] =
x => sumIfPositive(x).flatMap(sumIfMoreThan10)
total(6)
> Option[Int] = Some(16)
total(-1)
> Option[Int] = None

Amazing right! And you just have been introduced to the concept of a monad! Any type constructor that supports the flatMap function is known as a monad. flatMap and map (a monad) allow us to not just decouple the side effect handling code from our functions, but also gives us the mechanism to compose our functions (effectful or pure) to make bigger programs that handle side effects perfectly and are more maintainable because we can split the program into smaller peaces which are easy to combine.

More About Monads

On the last post we created a type class to generalise the concept of a functor, lets do the same for a monad. Notice that we implemented map in terms of flatMap, which means that every type constructor with flatMap can automatically have a map function, hence every Monad is also a Functor!

To further formalise the definition of a monad we need also a function called pure (which is actually the signature of an Applicative, but we can view Applicatives in another post [every Monad is an Applicative, and every Applicative is a Functor]). The pure function "lifts" a pure value into the context of a monad:

trait Monad[F[_]] extends Functor[F] {
def pure[A](a: A): F[A]
def map[A, B](M: F[A])(f: A => B): F[B] =
flatMap(M)(x => pure(f(x)))
def flatMap[A, B](M: F[A])(f: A => F[B]): F[B]
}

Left identity:

If we lift a pure value, and then flatMap with a monadic function (a function with the signature A ⇒ F[A] where A is a pure value and F[_] a Monad type constructor) then that must be equal to just passing the pure value through the monadic function:

pure(a).flatMap(f) === f(a)

Right identity:

If we take a monadic value m (a pure value that has been lifted to the context of a monad, has signature M[A]) and flatMap the pure function from it, that must be equal to the original m:

m.flatMap(pure) === m

Associativity:

Let m be a monadic value, and f and g monadic functions, then it must be that flat mapping f and then g be equal to composing f and g first (using flatMap) and then using the resulting monadic function to flatMap from m:

m.flatMap(f).flatMap(g) === m.flatMap(\x => f(x).flatMap g)

Play Framework - Beginner Tutorial : Make a post request and save the form data in Mongodb

Antoine Laffez — Mon, 03 Oct 2016 00:00:00 GMT

Before the tutorial

You should : - read - Play Documentation : ScalaActions - Play Documentation : ScalaForms - Play Documentation : ScalaJsonAutomated - have basic understanding of scala future transformation (map, flatMap) - have basic understanding of scala implicits - clone play2.5-skeleton-compileDI.

This example uses compile-time dependency injection. You can use run-time DI if you feel confident about it. - download and install mongodb (brew install mongo for mac or check the video) - download and install mongochef

Tutorial

After the tutorial

You should be able to : - create a compile-time DI play project - create a form in a play template - understand that a simple post in play requires two endpoints. - one to serve the form - one to handle the post request - create a model class - map a play Form to a case class - mapping - save a case class in Mongo DB

Play2-reactivemongo documentation

Be careful: we are using play2-reactivemongo plugin and not reactivemongo driver

Bonus

Do you know the difference between Action and Action.async ? Find out how to return different result statuses pages. check

Shapeless : Computing deltas

Antoine Laffez — Mon, 19 Sep 2016 00:00:00 GMT

A real life exemple

This is the code and some personal notes from the Shapeless? - Easy! talk where Valentin Kasas explains in a great way an advanced example of a real life use case (with slides).

Computing deltas

Imagine we want to be able to determine what are the difference between two objects of the same type
For example, we need to know what have changed in our DB since the last backup
We need to be able to compute such deltas over a wide variety of classes, that are unrelated
Of course, doing this by hand for each and every class is not an option

Our diff representation

sealed trait Diff[A]
final case class Identical[A](value: A) extends Diff[A]
final case class Different[A](left: A, right: A) extends Diff[A]
object Diff {
	def apply[A](left: A, right: A): Diff[A] =
    	if (left == right) Identical(left) else Different(left, right)
}

A first Delta implementation

We are creating a superclass here using DepFn . The return type differs according to the input type

trait SimpleDelta[R <: HList] extends DepFn2[R, R] {
  type Out <: HList //Result is bounded by HList
}

object SimpleDelta {
 /** type alias Aux adds as a type parameter the return Type.
  * It is convenient in order to constrain it
  * by passing it as a type parameter.
  */
	type Aux[I <: HList, O <: HList] = SimpleDelta[I] {
    	type Out = O
	}

	//Recursive Algorithm
	implicit def hnilDelta: Aux[HNil, HNil] = new SimpleDelta[HNil] {
		type Out = HNil
		def apply(l: HNil, r: HNil): Out = HNil
	}

	implicit def hconsDelta[H, T <: HList, DT <: HList](implicit tailDelta: Aux[T, DT]) : Aux[H::T, Diff[H] :: DT] = new SimpleDelta[H :: T] {
		type Out = Diff[H] :: DT
		def apply(l: H :: T, r: H :: T) : Out = Diff(l.head, r.head) :: tailDelta(l.tail, r.tail)
	}

	def apply[A, R <: HList](l: A, r: A)(implicit genA: Generic.Aux[A, R], delta: SimpleDelta[R]): delta.Out = delta(genA.to(l), genA.to(r))
}

A journey to ScalaCheck

Antoine Laffez — Mon, 12 Sep 2016 00:00:00 GMT

From the Spanish good weather to the Dutch every-possible-weather-in-one-day, from Waterfall to Agile, from just testing to property based testing. The path I took when joining Lunatech was an interesting one.

I want to share some of that and show how my journey to ScalaCheck started. I will prove that it’s not complicated to start with and it can uncover deeply hidden bugs in your code.

As developers, we need to be sure that we create code that performs exactly how it is meant to. This should be true in every possible scenario. However, how can we prove that our codebase actually does this for a wide range of data? Sometimes it is just not feasible to write innumerable amount of test cases for a specific function. We need to find a way to somehow prove our function works as expected in every possible case.

Property-based testing provides another way of thinking, that was new to me, about writing tests. Sometimes it is better to prove that a function satisfies a specific property, rather than to write a number of tests which try to confirm it is working fine. One way of proving is to generate an appropriate amount of data and apply these data to your test suite. These generated data should all have the same specific property, hence the name property-based testing.

As an example, imagine we want to test String concatenation. To do this we need to be sure that: For all given two strings, str1 and str2, the result of concatenating both strings must satisfy: str1.length + str2.length >= str1.length

Traditionally, we would write a test like:

test("Concatenate should generate a String of length s1+s2") {
concatenate("", "").length == 0
concatenate("Hello, ", "world.").length == 13 //Hello, world.    concatenate("Welcome to ","Lunatech.").length == 20
//Welcome to Lunatech.
}

But testing all possible combinations of 2 strings is impractical this way. In these cases, ScalaCheck is the recommended solution.

Let’s first understand the basic concepts in ScalaCheck: Properties and Generators.

Properties

In ScalaCheck you can specify what the input parameters are and what their properties are that must be satisfied by the input. It uses a very elegant and intuitive way for defining properties:

property(
"Concatenation length of two strings should be greater or equal to length of first string"
)
= forAll { (s1: String, s2: String) =>
   (s1 + s2).length >= s1.length
}

In this small piece of code, we declare a property ("Concatenation length of two strings …"), that holds forAll possible cases of concatenating 2 strings (s1 and s2). This seems reasonable but how can we prove what this property holds true. One way is by creating a lot of tests. And that is where Generators come in handy.

Generators

To generate this input data, ScalaCheck provides us with a wide range of generators available in objects Arbitraty and Gen.

The org.scalacheck.Arbitrary module defines implicit Arbitrary instances for common types, for convenient use in your properties and generators:

returns an arbitrary generator for the type T

The org.scalacheck.Gen uses Arbitrary and offers various generators:

alphaLowerChar, alphaUpperChar, alphaNumChar
identifier, alphaStr, numStr
negNum, posNum, chooseNum
listOf, listOfN, nonEmptyListOf
choose, oneOf, someOf
const

Some examples using arbitrary/generators:

id ← arbitrary[Int]

married ← arbitrary[Boolean]

age ← choose(0, 120)

currency ← const("euro")

description ← arbitrary[String]

However, most of the time we do not want to check such a general data type. For this, ScalaCheck also offers the possibility of defining custom generators where we can establish what the input data should look like.

Let’s use a simple example to understand the usage of custom generators. Imagine we are a Benelux bank that wants to verify that their Dutch customers who have a negative balance in at least one of their accounts, should be notified by email. For simplicity, we define customer and bank account as below:

case class Account(accountId: String, balance: Double, country: String)
case class Customer(customerId: String, name: String, nationality: String, accounts: Seq[Account])

So first we want to generate Account data. To do this, we make use of Arbitrary and Gen. Because we are only interested in Benelux accounts the country field will be one of "BE", "NL" or "LU"

 // Account generator - only Benelux accounts
val genAccount = for {
accountId <- Gen.identifier
balance <- arbitrary[Double]
country <- Gen.oneOf("NL", "BE", "LU")} yield Account(accountId, balance, country)

As a next step, we generate customer data. Because we are only interested in Dutch clients the nationality of the customers will be forced to be always "NL"

// Forcing customers to be Dutch will be as easy as:
val genDutchCustomer = for {
customerId <- Gen.identifier
name <- arbitrary[String].suchThat(_.nonEmpty)
nationality <- Gen.const("NL")
accounts <- nonEmptyListOf(genAccount)} yield Customer(customerId, name, nationality, accounts)

Finally, from the Dutch customers, we are interested on those having at least one account with negative balance

// Forcing customer to be Dutch and having negative balance:
val genDutchInRed = for {
customer <- genDutchCustomer.suchThat(_.accounts.exists(_.balance < 0))
} yield customer

Something which is worth mentioning at this point is the usage of .suchThat. It is recommended not to write very restrictive conditions in this filter, because ScalaCheck first generates all input data, and filters it later based on the condition provided. If the condition is too restrictive, it may end up with too many inputs discarded and the tests will not run.

To conclude with generators, let’s have a look to a sample of our Dutch customer with at least one account with negative balance:

scala> genDutchInRed.sample
res0: Option[Customer] = Some(
 Customer(uhcamdsjupssGeVftisrdb86mfbzflr,
 ïˆ¥ç§“é²›í—€ë€§ì–�å¤�î´‡å�½é”£ï°–ã¼½í˜‹á‘€æ§µÝ’ì…
 ¡ì†‰åš¿Ó”Ú¸å‚„íŽ½î³™ç‹‚ç±˜ï«©å¸•î¶™å’œáŠ¿æ¦�è˜žîŠ‡é�¥ï�¿ë‰Ÿá§¨áºŠë’¯á·”á´Žå‡ŸëŒ�ä¼“ä‹¨ç¹—
 ï¾•â‘£ä¼šæžºå³¸è£”â‡ºå¯œçŠ¼ê‡„è¼†ç‹Šç¯©ë—žâ™§ëžƒâ¶ªã«’êŽ™íˆ¥ì¦©,
   NL,
   List(

 Account(onScof2s4kBuphlrsal5ldWdh0oqbqbpgt03Snnrpryvlvzs89tnkh3fkreSsuoue0ntesrSlrpvDo7a4pe6bb
 qDly4cox,1.875359772688297E94),
 Account(yksznv4f48xezgep0daoyqtztcvruezwm,-3.9701238543851655E178),
 Account(uezzrfUxtbqPywvkXPbezZqtuX,4.8011482377734943E179),

 Account(htnlbxvtnDxiptwojhy4n36mzz2uovy5Xljoxgznkqomsk4rlhAxc9z6ebcwi6eMdnsass4cjhaerHfamcvzz0h6wtqn0pdgo6,6.04591158308268E-244),
  Account(s,-1.5255297073815315E-254),

  Account(vubpajf828dewljoarfp2uu0t9i3idnzhgDvjyediqyfax2fkfO6gAtgDqqNgaxkacswrcTzWpwkoopqt,-1.8
 68869258123239E-125),

 Account(guukirryuthlx4ejvhym6bVdiv8lleylBVfEkvslcvUskjlpzagtm2clfx4ashzdFQQWW,
 1.519776982857599E-66)
)
)
)

This shows us that maybe we should add some conditions to the accountId or the balance, because it is not normal to deal with such values in real life. This was for example one of the reasons to create scalaCheck-datetime

Writing tests

Now that we are familiar with properties and generators, it is time to write tests. We have good examples in the Scala community, because ScalaCheck is used by many Scala open source projects (like Akka or Play). In this case, we will continue with our concatenate example.

import org.scalacheck.Properties
import org.scalacheck.Prop.forAll
class StringProperties extends Properties("String Properties") {
property("Concatenation length equal or greater than zero") = forAll { (s1: String, s2:
String) =>
s1.length + s2.length >= 0
 }
 property("Concatenation length equal to length addition") = forAll { (s: String) =>
 val len = s.length
 (s + s).length == len + len
}
}

Our properties file can be as simple as that, or we can make it as complicated as we need. We can also integrate it with ScalaTest or Specs2

Running ScalaCheck tests

Using sbt, we run ScalaCheck tests in the same way we run ScalaTest tests: sbt test:compile test. If our code is correct and all the tests generated by ScalaCheck are successful, we can see the following as output:

+ String Properties.Concatenation length equal to length addition: OK, passed 100 tests.
+ String Properties.Concatenation length equal or greater than zero: OK, passed 100 tests.
ScalaCheck
Passed: Total 2, Failed 0, Errors 0, Passed 2

By default, ScalaCheck generates 100 tests per property, which must be satisfied for the test to pass.

In case a property is not satisfied by the generated test data, ScalaCheck yields an error. And not only shows the input data which makes the property to fail, but it also simplifies as much as possible to show you the minimum value which makes the test to fail. This helps us a lot when going back to the code and applying a solution to fix the wrong implementation.

How ScalaCheck helps with finding bugs

If you are not yet convinced we’ll give you another example of code that looks fine at first glance, but will not meet the requirements.

property("Absolute value should not be negative") = forAll { (input: Int) =>
input.abs >= 0
}

Looks reasonable, if we apply abs to a number, we will get a positive one (or zero). But… voilà! Here it is what ScalaCheck yields after running the test:

! String Properties.absolute value should not be negative: Falsified after 1 passed tests.
> ARG_0: -2147483648
ScalaCheck
Failed: Total 1, Failed 1, Errors 0, Passed 0

What ScalaCheck is showing is that the property fails for input = -2147483648 Then, we realize that Int numbers are not symmetric Int.MaxValue = 2147483647 Int.MinValue = -2147483648 So, when trying to apply abs to Int.MinValue, we get Int.MinValue.abs = -2147483648 which does not satisfy the condition of input.abs >= 0.

It is very likely that we write our code without thinking about these kind of corner cases, because we probably never expect an input with value -2147483648 But since -2147483648 is valid input our code will accept it and will crash if we do not add conditions to prevent it.

ScalaCheck focuses mainly on corner cases, where our functions are more sensible to fail. So for Int values, it will first test with MIN_VALUE, MAX_VALUE and 0; for String values will test with symbols and non-roman alphabet.

Useful links to get started

GitHub, projectScalaTest, integrationBook, and code examples

Summary

When you feel you are adding many tests based on input data, stop for a moment and think twice about the possibility of translating the functionality into a property that ScalaCheck can test for you.

If we can write properties for a given function, ScalaCheck provides an easy and very intuitive way of writing tests, which automatically generate large amounts of data for us, mainly focusing on corner and special cases.It is very helpful that ScalaCheck shrinks test cases to the minimal case.

ScalaCheck does NOT substitute ScalaTest or Specs2, but it complements them with property testing.Don’t forget that ScalaCheck is generating a finite number of tests, which means that there is always a chance that within this randomized set of tests, a bug might not be found (although it does exist in your code). However in case your input type is more constrained e.g. Byte, it can even generate all possible input data.

I started with ScalaCheck soon after I started with Scala and it changed the way I look at tests. Be always open to explore and try new options, because from all of them you will always learn something useful.

Shapeless: Introduction resources

Antoine Laffez — Mon, 05 Sep 2016 00:00:00 GMT

Where to start if you want to learn shapeless

Recently I decided to have a better look at shapeless, the generic programming library for Scala. In the previous couple of years I had bookmarked several links and have recently read all of them. In this blog post I present the most useful ones for someone who want to be introduced to the library.

The following blog posts are introductory to shapeless. Basically they are a smooth introduction to shapeless, something to warm up

Getting started with Shapeless by Julien Tournay (heterogenous lists, polymorphic function values, generic, tuples, lenses)
Bits of Shapeless part 1: HLists by Rui Batista (Heterogenous lists)

Also these slides on Type-Level Computations in Scala by Stefan Zeiger (not shapeless per se , but a lot of examples for type-level scala, like type-level booleans, type-level natural numbers, translation to types, recursion with types, type functions, type lambdas, heterogenous lists, hlist fold). I tried to find the video talk, but couldn’t find it online.

The Α shapeless primer blog post by Edoardo Vacchi is the best and most complete introductory blog post I could find. It really goes into a lot of explaining hlists, product types, Generic[T], FnToProduct[F] object, implicit value resolution, evidences and typeclasses, the Aux Pattern). There is also a video presentation by the author Edoardo Vacchi (with slides). In this blog post I find very interesting the comparison of Scala with Prolog and how implicit variables, implicit functions, type parameters in functions and implicit parameter lists of functions can be “interpreted” in a rule-based context.

Scala

implicit vals
implicit defs
type parameters in def
implicit parameters lists

Prolog

facts
rules
variable in a rule
bodies of a rule

In Shapeless? - Easy!, Valentin Kasas explains in a great way an advanced example of a real life use case (computing deltas) (with slides). From this presentation I took the following notes. There are also available two nice blog posts by Valentin : Shapeless : not a tutorial - part 1 (Heterogenous lists, polymorphic function values, high order poly functions) and part 2 (Generic, singleton types, records and LabelledGeneric)

In First-class polymorphic function values in shapeless (2 of 3) — Natural Transformations in Scala, Miles Sabin the creator of shapeless explains the concepts of polymorphic functions and natural transformations.

In the Shapeless for Mortals talk, Sam Halliday explains shapeless fundamentals like : type classes (not shapeless per se , but used extensively when using the library), singleton types, HList, LabelledGeneric, Coproduct, Hipster.Aux (SI-823), Higher Order Unification, Implicit Resolution: Recursion , Cycles and Priority, Tags (Notes from the talk)

The page Shapeless features provides a short overview of the main features of the library : polymorphic function values, heterogenous lists, HList-style operations on standard Scala tuples, facilities for abstracting over arity, heterogenous maps, singleton-typed literals, singleton-typed symbols, extensible records, coproducts and discriminated unions, generic representation of (sealed families of) case classes, boilerplate-free lenses for arbitrary case classes, automatic type class instance derivation, first class lazy values tie implicit recursive knots, collections with statically known sizes, type safe cast(Typeable/TypeCase), testing for non-compilation (illTyped)

After reading Shapeless features , the natural thing to follow is the Scala exercises in shapeless by 47 deg, which is almost identical to Shapeless features, but you will need to fill in the gaps so as to solve the exercises.

A lot of examples are available in Shapeless examples

bonus : Although not shapeless specific, there are two very interesting talks about Lenses

Lenses: Fields as Values (examples slides) where Seth Tissue explains the concept of Lenses and uses shapeless lenses.
Beyond Scala Lenses where Julien Truffaut (the creator of the Monocle library which is built with shapeless) explains the optics terms Iso, Prism, Lens and Optional.

Moving from Spain to the Netherlands

Antoine Laffez — Thu, 25 Aug 2016 00:00:00 GMT

Last week, I started talking about the main things I’ve faced when learning Scala, after being a Java developers for a few years. Today, I will talk about how it was to move from Spain to the Netherlands, in order to join Lunatech.

Food. Oh boy, the food. Spain is known for its Iberian ham, and you have a lot of different hams on every supermarket you visit. Usually, there is a big fridge, full of brands of ham, and also meat: pork, chicken, beef. And beef is also classified depending on the part of the animal it comes from, there are more than 10 possible kinds of beef and I don’t even know which one to choose when I go to buy something.

When I arrived to a dutch supermarket for the first time (Albert Heijn), I was surprised: there weren’t so many types of ham to choose, and most of the beef meat was ground meat, not slices. But, this supermarket shared one thing with the Spanish ones… the big fridge. This time, the fridge was full of cheese.

I’ve never seen so many different types of cheese in my life. On Spanish supermarkets, you usually find like 5 or 6 different cheese to choose about. But here, I’ve never seen less than 20 different cheeses on every supermarket I’ve visited so far. There is also cheese with spices! Never seen that before.Main differences between spanish and dutch gastronomy

Let’s talk about the weather. When I was still doing Skype interviews with the people from Lunatech before joining them, everybody made the same joke: "why do you want to come to the Netherlands? Because of the weather?" And they laughed afterwards. I didn’t understood why.

My parents were also confused: "why do you want to move to the Netherlands? Most of the people from Europe comes to Spain because of the sun and the beach, and now you are moving to a country that is well known for its rain and wind". I was sure of my decision, I always thought that Spain was too hot for me, too dry, and the weather from the Netherlands would fit me more, with more rain and more clouds. I was so wrong.

The first week I spent here, it rained every single day. Dutch weather welcomes foreigners by making them real men, under the rain. I wasn’t prepared for that. In Spain (not in the northern regions), when we say "rain" we are usually talking about some water falling over your shoulders and walking over wet floor. A standard dutch rain would be something to tell on the news if it happened on Spain. And I even remember the day some trains got delayed because of floods. In July, in summer. In Madrid (the city where I come from) we barely have 3 or 4 days of rain between May and September.

And the worst part about the dutch rain is that is also very irregular. Usually in Spain if you have rain, you have rain the whole day, and if you have sun, the same. But then, you wake up in the Netherlands, ready to go to the office, you see the sun in the sky and then you choose to go with a T-shirt. Few hours later, when you are coming home, the sky is black, you are absolutely wet on the street and you are wondering why you didn’t picked the jacket that morning.

I remember one conversation with one of my coworkers:

Him: “Oh look, the sun! Maybe you should go to the beach today”

Me: “But there are some clouds on the sky…"

Him: “This is the best you can get"

Main differences between spanish and dutch weather

But there is something that I really love about the dutch culture, and I wasn’t expecting to: bikes. Bikes here are very easy to get, and the streets are prepared to be crossed by bike. In Madrid it’s almost impossible to use a bike because the streets are narrow most of the time, and the cars are going through, but here, almost every street as a bike path, so you can safely go everywhere.

You can park your bike at the station (actually, everywhere, if you are brave enough), and a lot of the people use the bike to make small distances, like going to the supermarket. I mean, this is very important: the relief of being able to carry all your bags on the bike and park it in front of your house, is almost a blessing.

But not everything is wonderful about bikes: because you are in the Netherlands, and the dutch weather wants you to become a "real man". Sometimes you are having a great moment on your bike, and then suddenly the strongest wind you could ever imagine decides to go exactly on the opposite direction you are going, testing if your legs are strong enough, and if your will can handle the situation. And when is not coming from the front, is coming from your side, and then you will fall to the ground if you are not experienced enough.

But you can become an expert biker too. And dutch people love to show how professional they are on this matter. You think you are strong because you can carry all your supermarket bags and still going fast enough? Look at this mother carrying their two children on the bike and passing you on the left. You think you are cool because you can handle the bike with one hand while scratching your head with the other one? I’ve seen many times people texting on their phones, with both hands, while turning the bike. I’ve so much to learn.

Moving From Java To Scala

Antoine Laffez — Wed, 17 Aug 2016 00:00:00 GMT

Sometime ago I came from Madrid to Rotterdam to join Lunatech’s international workforce. Apart from introducing me to Dutch summers (rain), this also gave me an opportunity to learn a new programming language: Scala.

Since I finished university I’ve been working with Java and I really liked it, but Scala was something entirely new to me. In this post I’ll give my first view on Scala from a Java perspective.

Many people say that Scala is like Java but without semicolons, but I disagree. It’s true that Scala doesn’t need semicolons, and that may save developers some time, but it’s just a minor feature. One of the things that really caught my eye when I started, is related to style: unlike Java developers, Scala developers like to put everything on the same line. Let’s consider the next piece of Java code:

public Foo someMethod(Foo foo, Bar bar) {
  Foo firstResult = foo.doSomething();
  Bar secondResult = bar.doOtherThing();
  Foo finalResult = doLastThing(firstResult, secondResult);
  return finalResult;
}

We could have compressed that into something smaller, but usually Java developers love verbose stuff: to define variables with long names so everything looks clear and people can understand what’s happening on the code. On the other hand, a Scala developer would do something like this:

def someMethod(foo: Foo, bar : Bar) : Foo = doLastThing(foo.doSomething, bar.doOtherThing)

Basically, the main thing that me, as a Java developer, saw when reading Scala code is “try to put everything on the same line”, and that makes things a bit difficult to understand at the beginning. But in the end your Scala code ends being very small and powerful.

Another thing that stands out are Collections. When you are learning Collections in Java, most of the times you start with arrays (Foo[]) and later you probably move to ArrayList. Then, iterating over your Collection of choice is made with for/while loops most of the time, with some logic inside the curly brackets. But Scala’s Collection library is more powerful and easy to use than the one you have available in Java.

Recent versions of Java have implemented new libraries with similar tools as in Scala, but they have been developed a few years ago, and most of the times they require weird implementations of the Stream class. Scala, on its side, was created with this kind of tools in mind, so they are far more accessible and easy to understand.

In Scala, the map method (and the rest of their friends: flatMap, flatten, head, tail, filter, etc) makes iterating through Collections so easy and safe, that you even try to use it as much as you possibly can. You don’t have to worry about IndexOutOfBoundsException anymore. When you can transform this:

for (int i = 0; i < intList.size(); i++) {
	if (intList.get(i) % 2 == 0)
    	intList.set(i, intList.get(i) + 1)
}

Into this:

intList.map(x => if (x % 2 == 0) x + 1 else x)

Your fingers will be glad you that you’ve switched to Scala.

But the award of “most surprising thing for people that comes from Java” goes to… vals and immutable stuff!

The most common way of implementing immutability in Java is by using final on the variables/objects you want to remain constant through your code. But the way Scala uses immutability is somehow different: you try to make every single variable immutable. If you declare an Int or a String, you go immutable.

When you’re learning Java, you are told to declare as few variables as possible, and to reuse them as much as possible, in order to release memory and make the life of the garbage collector easier. But in Scala, you have two options: you can either use var (mutable variable) or val (immutable variable), but you are told to use val as much as possible, and to avoid var as much as you can. At the beginning this is confusing (you wonder "if I should use always val, why do you allow me to use var on the first place?") and many times you want to hit you head against a wall. But when you start to master Collections and your inner-developer is at peace with the Immutability Zen, it’s even rewarding and interesting to make code with everything immutable.

So I’m having fun learning Scala, and I will go with some more details next week about moving to the Netherlands.

Play Framework - Beginner Tutorial : How to handle a big json file in play ( more than 22 root variables)

Antoine Laffez — Mon, 15 Aug 2016 00:00:00 GMT

It is not a good practice to have such a big json with many root variables. Nevertheless, we might need to call a rest api that will give a json like this

{  "a1": ...,  "a2" : ...,  "a3" : ...,  "a4" : ...,  "a5" : ...,  "a6" : ...,  "a7" : ...,  "a8" : ...,  "a9" : ...,  "a10" : ...,  "a11" : ...,  "a12" : ...,  "a13" : ...,  "a14" : ...,  "a15" : ...,  "a16" : ...,  "a17" : ...,  "a18" : ...,  "a19" : ...,  "a20" : ...,  "a21": ...,  "a22" : ...,  "a23" : ...,  "a24" : ...,  ....}

First approach

Since scala 2.11 we can have case class with more than 22 fields, so the following compiles.

case class JsonExample (    a1:String, a2:String, a3:String, a4:String, a5:String, a6:String,    a7:String, a8:String, a9:String, a10:String, a11:String, a12:String,    a13:String, a14:String, a15:String, a16:String, a17:String, a18:String,    a19:String, a20:String, a21:String, a22:String, a23:String, a24:String)

But if we try to use Play JSON automated mapping

import play.api.libs.json._implicit val jsonExampleFormat = Json.format[JsonExample]

we get a compiler error

:16: error: No unapply or unapplySeq function found       implicit val jsonExampleFormat = Json.format[JsonExample]                                                   ^

Instead we can use Play-Json extensions(requires play-json >= 2.4)

import ai.x.play.json.Jsonximplicit val jsonExampleFormat = Jsonx.formatCaseClass[JsonExample]

jsonExampleFormat: play.api.libs.json.Format[JsonExample] = $anon$1@b7f7890

So now we have a Play Json format for a case class with more than 22 fields, and can use it as described in Play Json documentation

Second approach

I don’t consider having a case class with so many fields a good solution.

A better solution is to organize these fields in “logical” embedded case classes.

case class A (    a1:String, a2:String, a3:String, a4:String,    a5:String, a6:String, a7:String, a8:String)

case class B (    a9:String, a10:String, a11:String, a12:String,    a13:String, a14:String, a15:String, a16:String  )case class C (    a17:String, a18:String, a19:String, a20:String,    a21:String, a22:String, a23:String, a24:String  )case class JsonExample2 (    a : A,    b : B,    c : C  )

We use a “regular” play json format for the embedded case classes.

import play.api.libs.json._implicit val jsonAFormat : Format[A] = Json.format[A]implicit val jsonBFormat : Format[B]= Json.format[B]implicit val jsonCFormat : Format[C] = Json.format[C]

And the “trick” now is to use the JsPath Play Json Combinator for the embedded case classes without any path.

import play.api.libs.functional.syntax._implicit val jsonExample2Format: Format[JsonExample2]  = (  (JsPath).format[A] and  (JsPath).format[B] and  (JsPath).format[C])(JsonExample2.apply, unlift(JsonExample2.unapply))

That’s all folks!

Continuous Delivery on GitLab with Play, Scala, SBT and Heroku

Antoine Laffez — Mon, 08 Aug 2016 00:00:00 GMT

Last week we talked about Continuous Integration (CI) with Scala and SBT on GitLab. This week we are going to take a step further and implement Continuous Delivery (CD) on GitLab with Play and Heroku.

Let’s first recap last week’s post. To enable CI, we used the .gitlab-ci.yml file. Builds are run in Docker and to make sure we could execute our tests, we used a Java Docker image and made sure SBT was installed. The definition in our .gitlab-ci.yml file contained only a single stage: the test stage. This week we are going to extend our pipeline with a deploy stage.

Play Deployment with Heroku

The application we are going to deploy is a Play Framework application. For now we assume we already have such application with SBT as the build tool.

Heroku is a cloud application platform that has support for Play applications. This means that running your application on Heroku requires little setup and most important is getting the source of your application there. For that, we use the tool dpl from Travis CI.

Dpl is a simple command line deployment tool that supports Heroku. We can deploy our Play application to Heroku using dpl with the following command:

dpl --provider=heroku --app= --api-key=

First you have to create an application in Heroku. in the command above should be substituted with the name you gave the application. Additionally, should be substituted by your Heroku API key which you can find under (under Account > API Key).

Putting It All Together

We extend the definition of our pipeline in the .gitlab-ci.yml file by adding a deploy stage and in the deploy stage, we install dpl and instruct it to deploy the new version of the application to Heroku. To hide the API key it is passed via a secret variable. This can be added in GitLab in your project’s settings under Variables.

The .gitlab-ci.yml file now looks like this:

stages:
- test
- deploy
test:  stage: test
script:
# Test the project
- sbt clean test
deploy:
stage: deploy
script:
# Install dpl
- apt-get update
-yq
- apt-get install rubygems ruby-dev -y
- gem install dpl
# Deploy to Heroku
- dpl --provider=heroku --app=lunatech-demo-cd-play-scala --api-key=$HEROKU_API_KEY

After pushing this .gitlab-ci.yml file to our GitLab repository containing the Play project, our continuous delivery pipeline is enabled. On every new push, the tests of the project are run and if the tests pass, the new version of the application is deployed to Heroku using dpl.

Source Code

The source code of a demo project implementing the pipeline described in this post can be found here: https://gitlab.com/jasperdenkers/lunatech-demo-cd-play-scala-heroku.

Continuous Integration on GitLab with Scala and SBT

Nicolas Leroux — Mon, 01 Aug 2016 00:00:00 GMT

With Continuous Integration (CI) we aim to be able to test the changes we make to our projects automatically and externally. An usual setup involves e.g. GitHub and Jenkins and requires quite some configuration.

In this post we look at an alternative approach using GitLab which offers both the features of the aforementioned services: git hosting and continuous integration.

In this post we specifically focus on Scala projects with SBT as the build tool. Let’s assume we have an SBT project that contains tests and is hosted on GitLab. Now we have to tell GitLab that we want CI. We do this by adding a .gitlab-cy.yml file in the root directory of our project.

The `.gitlab-ci.yml` File

Builds are run in Docker and therefore we have to specify in the .gitlab-ci.yml file which Docker image we want to use. Additionally we define some commands before our tests are run. We use the Java 8 image and make sure SBT is installed and available in our builds:

image: java:8

before_script:
  # Enable the usage of sources over https
  - apt-get update -yqq
  - apt-get install apt-transport-https -yqq
  # Add keyserver for SBT
  - echo "deb http://dl.bintray.com/sbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list
  - apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 642AC823
  # Install SBT
  - apt-get update -yqq
  - apt-get install sbt -yqq
  # Log the sbt version
  - sbt sbt-version

Now let’s define the stages we want to use. We add the test stage with the following lines:

stages:
  - test

At last, we define a job in the test stage that executes the the tests in our project:

test:
  stage: test
  script:
    # Execute your project's tests
    - sbt clean test

Builds within the same stage run in parallel and the jobs in a next stage are started after all previous stages are successfully finished. We can define as much stages and builds as we want. Together, they are called a pipeline.

After we commit and push the .gitlab-ci.yml file to GitLab, CI is enabled and the pipeline is executed on every new push to the project.

Measuring Test Coverage

To display the test coverage of our test job we need to take two steps:

Add test coverage measuring in our SBT project.

Instruct GitLab how to extract the coverage percentage from the build trace.

We measure test coverage by using the SBT plugin sbt-scoverage. Add the following line to /project/plugins.sbt:

addSbtPlugin("org.scoverage" % "sbt-scoverage" % "1.3.5")

You can now measure test coverage with sbt clean coverage test coverageReport. We change this in .gitlab-ci.yml:

test:
  stage: test
  script:
    # Execute your project's tests and measure test coverage
    - sbt clean coverage test coverageReport

And in our project settings under CI/CD pipelines > Test coverage parsing we add the following regular expression: Coverage was \[\d+.\d+\%\]. This enables GitLab to extract the coverage percentage from a build trace and display it next the the build results.

Source Code

The source code of a demo SBT project implementing the steps described in this post can be found here:

https://gitlab.com/jasperdenkers/lunatech-demo-ci-scala-sbt

What’s Next?

Next week we are going a step further and we implement continuous delivery (CD) in GitLab with the Play Framework and Heroku.

References

http://docs.gitlab.com/ce/ci/

Recursion and Trampolines in Scala

Nicolas Leroux — Fri, 15 Jul 2016 00:00:00 GMT

Recursion is beautiful. As an example, let’s consider this perfectly acceptable example of defining the functions even and odd in Scala, whose semantics you can guess:

def even(i: Int): Boolean = i match {
  case 0 => true
  case _ => odd(i - 1)
}

def odd(i: Int): Boolean = i match {
  case 0 => false
  case _ => even(i - 1)
}

We’ve defined even in a very natural way: an value is even when it’s zero, or when it’s odd after subtracting one. Odd is defined in a similar way.

Let’s ignore the problem this code has with negative integers, and the fact that an O(n) algorithm for computing even and odd is not ideal, and focus on a single problem: this code breaks for large n:

scala> even(5000)
res3: Boolean = true
scala> even(50000)
java.lang.StackOverflowError
at Trampolines$Stack$.odd(trampolines.scala:14)
at Trampolines$Stack$.even(trampolines.scala:9)
at Trampolines$Stack$.odd(trampolines.scala:14)
at Trampolines$Stack$.even(trampolines.scala:9)
at Trampolines$Stack$.odd(trampolines.scala:14)
at Trampolines$Stack$.even(trampolines.scala:9)
...
at Trampolines$Stack$.even(trampolines.scala:9)
at Trampolines$Stack$.odd(trampolines.scala:14)

In the remainder of this post we look at various strategies for preventing recursive programs from running out of memory.

But what’s the problem?

The problem manifests itself in the stack trace: even calls odd, then odd calls even, which calls odd again. Each of these invocations causes the creation of a stack frame, eating away the space that’s reserved for them.

Keeping all the stack frames would not really be necessary: when even calls odd, that is the very last thing that the even function will ever do! When odd returns, even itself returns immediately with that same value.

When we compute even(500), after a bunch of recursive calls, the 501st stack frame will be created, for the invocation of even(0), which returns true, and the 500 method calls that are left on the stack will one after another all return that same value, until the whole stack is unwinded.

Some languages or runtimes are smart enough to figure out that if a calling function will simply return the value that a function it calls returns, it does not need to allocate a new stack frame for the callee, but can reuse the stack frame of the caller for that.

Unfortunately, the JVM does not support this.

And so Scala, as long as it sticks to JVM method calling semantics, also can not support this.

A bit of light at the end of the tunnel

Luckily, there is one special case of tail calls which Scala treats differently, and that is the case of tail recursion. If a method does a tail call to itself, it’s called tail recursion, and Scala will rewrite the recursion into a loop that does not consume stack space.

Many recursive algorithms can be rewritten in a tail recursive way. For our even and odd problem, this is a possible solution:

def even(i: Int): Boolean = i match {
case 0 => true
case 1 => false
case _ => even(i - 2)
}
def odd(i: Int): Boolean = !even(i)

We’ve rewritten even to be tail-recursive, and now Scala will optimize it for us:

scala> even(500000000)
res25: Boolean = true

it no longer blows up for large numbers.

The @tailrec annotation is useful in these cases. It does not change how the function will be compiled, but it does trigger an error if the function is not actually tail-recursive. So this will protect you from accidentally changing a function that you want to be tail-recursive function into something that’s no longer tail recursive.

You can try this out by putting the @tailrec annotation on the first and the latest version of even that we’ve defined.

Trampolines

Another way of solving the problem of consuming too much stack space, is by moving the computation from the stack to the heap.

Instead of letting the even function recursively call odd, we could also let it return some recipe to it’s caller, how to continue with the computation. Let’s look at the following code:

sealed trait EvenOdd

case class Done(result: Boolean) extends EvenOdd
case class Even(value: Int) extends EvenOdd
case class Odd(value: Int) extends EvenOdd

def even(i: Int): EvenOdd = i match {
  case 0 => Done(true)
  case _ => Odd(i - 1)
}

def odd(i: Int): EvenOdd = i match {
  case 0 => Done(false)
  case _ => Even(i - 1)
}

Here we defined a little language, EvenOdd, that can express three things:

The computation is Done, and we have the result value
We need to compute whether a value is Even
We need to compute whether a value is Odd
Our even and odd functions no longer return a boolean, but return an expression in this little language.

Finally, we create an interpreter, that will recursively evaluate expressions in this language:

// Using Scala's self recursive tail call optimization
@tailrec
def run(evenOdd: EvenOdd): Boolean = evenOdd match {
  case Done(result) => result
  case Even(value) => run(even(value))
  case Odd(value) => run(odd(value))
}

Now we have expressed even and odd in their natural, mutually recursive way, and we still have a stack safe interpreter:

scala> run(even(5000000))
res1: Boolean = true

The disadvantage of this is that this is significantly slower. Unfortunately, we can’t seem to have our cake and eat it too :(

This strategy is sometimes called trampolining, because instead of creating a big stack, we go up to even, then down to run, then up to odd, then down to run, then up to even, down to run, etcetera. The size of our stack keeps growing and shrinking by one frame for every step in the computation. This looks a lot like going up and down on a trampoline :)

Generalizing

There is no need to specialize our little language to computing even and odd. We can also make a little language that can express recursion in a general way:

sealed trait Computation[A]

class Continue[A](n: => Computation[A]) extends Computation[A] {
  lazy val next = n
}

case class Done[A](result: A) extends Computation[A]
  def even(i: Int): Computation[Boolean] = i match {
    case 0 => Done(true)
    case _ => new Continue(odd(i - 1))
  }

  def odd(i: Int): Computation[Boolean] = i match {
    case 0 => Done(false)
    case _ => new Continue(even(i - 1))
  }

  @tailrec
  def run[A](computation: Computation[A]): A = computation match {
    case Done(a) => a
    case c: Continue[A] => run(c.next)
  }

Recursion and Trampolines in Scala

Here our even and odd functions don’t return domain specific values, but a general value that indicates whether the computation is done, or whether more steps are needed. The latter includes the next step as a by-name parameter, that the tail recursive runner function can call.

Note that our run function is no longer tied to computing even and odd, it can compute anything.

TailRec in the standard library

Something similar in spirit, but with a better implementation is also available in the Scala standard library:

import scala.util.control.TailCalls.{ TailRec, done, tailcall }

def even(i: Int): TailRec[Boolean] = i match {
  case 0 => done(true)
  case _ => tailcall(odd(i - 1))
}

def odd(i: Int): TailRec[Boolean] = i match {
  case 0 => done(false)
  case _ => tailcall(even(i - 1))
}

even(3000).result

Comparing performance

I compared the performance of these solutions with JMH, and these are the results:

[info] Benchmark Mode Cnt Score Error Units
[info] Trampolines.GeneralTrampolineRunner.bench thrpt 30 44916.024 ± 388.202 ops/s
[info] Trampolines.ScalaTrampolineRunner.bench thrpt 30 52106.426 ± 408.242 ops/s
[info] Trampolines.SpecializedTrampolineRunner.bench thrpt 30 94002.234 ± 1584.913 ops/s
[info] Trampolines.StackRunner.bench thrpt 30 358382.321 ± 6622.659 ops/s

As expected, the version that runs on the stack is the fastest. But remember that this is the version that breaks for a large number of recursions.

The specialized trampolining version, with the EvenOdd domain specific language and a runner optimized for this particular problem, takes about a 4 times speed hit compared to the stack version.

The general trampoline version that we defined here is about 2 times slower than the specialized version, and about 8 times slower than the stack version.

The TailRec version from the Scala standard library is about 20% faster than our general trampoline, making it about 7 times slower than the stack version.

Source code

The source code of the benchmarks (and all the code), is available on https://github.com/eamelink/scala-trampolines

Clever Cloud’s CEO to speak at Lunatech

Antoine Laffez — Thu, 26 May 2016 00:00:00 GMT

Friday 27 May 2016 Clever Cloud’s CEO, Quentin Adam, will be visiting Lunatech where he will give a presentation about using Clever Cloud with Scala.

Clever Cloud is a “Europe-based PaaS company” that helps “developers deploy and run their apps with bulletproof infrastructure, automatic scaling, fair pricing” and aims “to make an easy-to-use service, without any vendor lock-in”.

Due to his experience at Clever Cloud — where he can work a wide range of technologies and tools — he has a lot of knowledge and is able to speak about many different subjects. He regularly speaks at various tech conferences.

We are opening up the presentation to everyone. Feel free to join us at 16:00.

Scala training camp at Lunatech

Antoine Laffez — Thu, 19 May 2016 00:00:00 GMT

Every friday our teams work together to improve their skills on Scala and other technologies. They explore new grounds, new ideas and new techniques. They share knowledge between teams. Join us if you want to make your passion your work.

On the road again: Scala days Berlin 2016

Antoine Laffez — Mon, 16 May 2016 00:00:00 GMT

Lunatech is proud to present the real star of this event: the Lunatech carry-on suitcase. With its class and convenience, it will accompany all Lunatech employees to the 2016 Scala days in Berlin. All suitcases are personalised with their owner’s name to make it difficult to sell them to the highest bidder and to prevent people ending up with their colleague’s suitcase.

Platinum sponsor of Scala Days Berlin

Antoine Laffez — Thu, 14 Apr 2016 00:00:00 GMT

Lunatech is proud to be a platinum sponsor for Scala Days Berlin 2016. This year Lunatech is taking the whole team to Scala Days Berlin.

If you’re there and are looking for a job with the kind of company that sponsors and takes all its employees to Scala Days, have a chat with us, there’ll be a lot of us walking around. If you can’t wait, get in touch with us right now.

Fast Track to Scala Training by Lunatech

Antoine Laffez — Mon, 18 Jan 2016 00:00:00 GMT

On Feb 4 and 5th Francisco Canedo, Senior Developer at Lunatech and a Typesafe certified trainer, will be teaching Typesafe’s Fast Track to Scala Course at our office.

This course is designed to give experienced developers the know-how to confidently start programming in Scala. The course ensures you will have a solid understanding of the fundamentals of the language, the tooling and the development process as well as a good appreciation of the more advanced features. If you already have Scala programming experience, then this course could be a useful refresher, yet no previous knowledge of Scala is assumed.

The course starts each day at 9:00 with breakfast, lunch is also provided, and ends between 17:00 and 18:00 depending on the needs of the students and costs €600 per person.

If you are interested in joining the course, please contact us by sending a mail to training@lunatech.com or tweet us @LunatechLabs

Our first award!

Antoine Laffez — Mon, 04 Jan 2016 00:00:00 GMT

Lunatech has been awarded by the “The National Business Success Award Institute” as branch winner 2015 in the category “ICT”.

The jury cited the company’s service quality, technology excellence and innovation. Representatives of the award winner also make appearances on the RTL television show "De Success Factor" .

Functional Rotterdam - 5th Edition

Antoine Laffez — Mon, 04 Jan 2016 00:00:00 GMT

Every first Tuesday of the month, at Lunatech we are organize a meetup focussed on Functional Programming. Tomorrow is the 5th edition of the event. You can join the fun by going to our meetup page

This is an informal event where we share our experience of using functional programing in our projects and learn from others who work in different languages. In the past we had presenations about Scala, Spark, Elm, Frege apart from a coding dojo at the last event.

Do join us by RSVPing on the meetup page

Lunatech management realises MBO

Antoine Laffez — Tue, 22 Dec 2015 00:00:00 GMT

Ray Kemp and Nicolas Leroux, jointly CEO, have successfully realized a management buy-out (“MBO”) of Lunatech Labs BV (“Lunatech”). Also, some staff members acquired a small stake in the company in this transaction. Lunatech is a Dutch IT service provider specialised in system integration, customised software implementation and web applications.

Main Corporate Finance has structured the MBO and arranged the debt financing. Bank financing has been provided by Rabobank. The transaction was partly financed with a mezzanine loan provided by Main Mezzanine Capital.Lunatech stands out in linking and structuring applications between back-end and front-end. Lunatech provides support for complex IT system integration and migration projects and they support IT departments of organisations. The company offers high quality solutions including consultancy and supplying project teams of Scala professionals.

Lunatech was founded in 1993 and located in Rotterdam. Lunatech provides its services to proven customers in the Netherlands and France, as for example: ING, SDU, Alcatel, Disney and UPS.

The strength of Lunatech is also recognised by the investors, says Lars van ‘t Hoenderdaal:

“Lunatech is a very nice company with a strong management team and international growth potential. We experienced great interest among our funders to participate in this mezzanine proposition.”

After reaching agreement on the transaction structure and financing structure, the transaction has been completed within 4 weeks. Rabobank has also shown their strength by acting quickly.

Ray Kemp, CEO Lunatech, is very satisfied with the realised MBO:

“Due to funding from the bank and mezzanine we were able to realise the MBO without a private equity investor. In this way we can realise our growth independently.”

Main Mezzanine Capital is part of Main Capital Partners, an investment and financing company, as well as a corporate finance advisor, with a focus on growing and profitable companies in the TMT sector in the Netherlands. Main Mezzanine Capital is funded by its own shareholders, wealthy individuals, family offices and (former) entrepreneurs, who like to invest in companies with an interesting risk-return profile.

Syntactically correct and type-safe JPA queries in Play 2.0

Antoine Laffez — Wed, 15 Jul 2015 00:00:00 GMT

For all three approaches we’ll use the same use case: find a User object by its ‘username’ property.

Java Persistence API query language

An easy way to read data from our database is by creating a ‘dynamic query’ with use of the Java Persistance API (JPA) query language, a string-based query language. To find a User by username we would write something like:

try {
    return JPA.em()
        .createQuery("from User where username = :username", User.class)
        .setParameter("username", "foo").getSingleResult();
} catch (NoResultException nre) {
    return null;
} catch (NonUniqueResultException nure) {
    return null;
}

The problem with this example is syntax checking. The query language is not checked at compile time, so if we write from Uzer instead of from User no compile-time errors will occur. However, because the query is incorrect, we will get a runtime error. To prevent these kind of runtime errors we need a way to create syntactically checked queries. This is where the JPA Criteria API comes in.

JPA Criteria API

The Criteria API is an alternative to the query language. It allows us to define object-based queries instead of the string-based approach of the query language. The advantage is that these object-based queries are more syntactically checked.

If we want to find the User with help of the Criteria API it becomes this piece of code:

try {
    CriteriaBuilder cb = JPA.em().getCriteriaBuilder();
    CriteriaQuery query = cb.createQuery(User.class);
    Root user = query.from(User.class);
    query.where(cb.equal(user.get("username"), "foo"));
    return JPA.em().createQuery(query).getSingleResult();
} catch (NoResultException nre) {
    return null;
} catch (NonUniqueResultException nure) {
    return null;
}

Here the compiler will give us an error if we write Uzer.class instead of User.class. So more syntax checking. But as you probably already noticed, user.get("username") is still not checked for correct syntax. Our code will successfully compile if we replace username with uzrname. This is because the compiler doesn’t know if uzrname is a property of the User class. So we need more syntax checking to prevent these kind of errors. With the help of Java Persistence Metamodel API we can create fully syntactically checked queries.

JPA 2.0 Metamodels

A metamodel class is a class that represents the datastructure of the corresponding model class. For example, this model class:

@Entity
public class User {
    public Long id;
    public String username;
}

Would have the following metamodel class:

@StaticMetamodel(User.class)
public abstract class User_ {
    public static volatile SingularAttribute username;
}

Now we are able to create a type-safe query:

try {
    CriteriaBuilder cb = JPA.em().getCriteriaBuilder();
    CriteriaQuery<User> query = cb.createQuery(User.class);
    Root<User> user = query.from(User.class);
    query.where(cb.equal(user.get(User_.username), "foo"));
    return JPA.em().createQuery(query).getSingleResult();
} catch (NoResultException nre) {
    return null;
} catch (NonUniqueResultException nure) {
    return null;
}

These metamodels are pretty useful because our queries are syntactically correct and type-safe! Syntactically correct because all the syntax is checked. We will get an error is we replace username by uzername. So if we, for example, rename a variable, the compiler will tell us which queries we forgot to update. We won’t get a runtime error.

The query is also type-safe. Type safety means that the compiler will validate types while compiling. If a wrong type is assigned to a variable an exception is thrown at compile time. As you probably know Play 2.0 is [focused on type safety](http://www.playframework.org/documentation/2.0/Philosophy).

Because of the metamodels, the compiler knows the type of the variables. So you can’t for example do:

try {
    CriteriaBuilder cb = JPA.em().getCriteriaBuilder();
    CriteriaQuery<User> query = cb.createQuery(User.class);
    Root<User> user = query.from(User.class);
    query.like(cb.like(user.get(User_.id), "1"));
    return JPA.em().createQuery(query).getSingleResult();
} catch (NoResultException nre) {
    return null;
} catch (NonUniqueResultException nure) {
    return null;
}

This query won’t compile because the like method needs a Expression as first parameter and id is a Expression.

Because it takes some time to write these metamodels by hand, it would be nicer to generate them. The next paragraph will explain how to generate metamodels in Play 2.0.

JPA 2.0 Metamodel genaration in Play 2.0

A way to generate our metamodels is by using the org.hibernate.jpamodelgen.JPAMetaModelEntityProcessor. The processor will automatically run if the hibernate-jpamodelgen.jar is added to the classpath and when you are using JDK 6. So we add a project dependecy to our Build.scala:

val appDependencies = Seq(
    "org.hibernate" % "hibernate-jpamodelgen" % "1.2.0.Final"
)

And specify a folder where the generated source files are placed. This is done by passing an argument to the Java compiler. Passing an argument is also done in Build.scala:

val main = PlayProject(appName, appVersion, appDependencies, mainLang = JAVA).settings(
    javacOptions ++= Seq("-s", "metamodel")
)

Note that the provided folder must exist, the Java compiler won’t generate it for you. So it is probably not a good idea to put the metamodels in our target folder, because its contents are deleted when the the play clean command is run.

Eclipse users can add the metamodel folder to their ‘source folders’ (Project → Properties → Java Build Path → Source → Add Folder) for autocompletion etc.

Conclusion

Using the Criteria API with metamodels gives us the opportunity to write syntactically correct and type-safe queries. It is also quite easy to generate these metamodels. So with a bit of effort we can get nice object-based, syntactically correct and type-safe queries that cause fewer runtime errors.

Sources

[JSR-317 - Java Persistence 2.0](http://jcp.org/aboutJava/communityprocess/final/jsr317/)

[Hibernate JPA 2 Metamodel Generator](http://docs.jboss.org/hibernate/jpamodelgen/1.0/reference/en-US/html_single)

Lunatech's engineer blog

An Introduction to Finagle by example

Quickstart

The core of Finagle

Service

Filter

Client

Protocol-agnostic

Conclusion

Functional IO with FS2 Streams

Functional IO with FS2 Streams

The Idea of Functional Streams

How to Use FS2

Pulling

Conclusion

Useful Links

Functional IO with FS2 Streams

Functional IO with FS2 Streams

The Idea of Functional Streams

How to Use FS2

Pulling

Conclusion

Useful Links

Side Effects and How To Deal With Them The Cool Way, Part 2: Monads Introduction

Function Composition And a Slight Issue

Monads! The Combinator We Were Looking For

More About Monads

Left identity:

Right identity:

Associativity:

Play Framework - Beginner Tutorial : Make a post request and save the form data in Mongodb

Before the tutorial

Tutorial

After the tutorial

Bonus

Shapeless : Computing deltas

Computing deltas

A journey to ScalaCheck

Properties

Generators

Writing tests

Running ScalaCheck tests

How ScalaCheck helps with finding bugs

Useful links to get started

Summary

Shapeless: Introduction resources

Where to start if you want to learn shapeless

Moving from Spain to the Netherlands

Moving From Java To Scala

Play Framework - Beginner Tutorial : How to handle a big json file in play ( more than 22 root variables)

First approach

Second approach

Continuous Delivery on GitLab with Play, Scala, SBT and Heroku

Play Deployment with Heroku

Putting It All Together

Source Code

Continuous Integration on GitLab with Scala and SBT

The .gitlab-ci.yml File

Measuring Test Coverage

Source Code

What’s Next?

References

Recursion and Trampolines in Scala

But what’s the problem?

A bit of light at the end of the tunnel

Trampolines

Generalizing

Recursion and Trampolines in Scala

Comparing performance

Source code

Clever Cloud’s CEO to speak at Lunatech

Scala training camp at Lunatech

On the road again: Scala days Berlin 2016

Platinum sponsor of Scala Days Berlin

Fast Track to Scala Training by Lunatech

Our first award!

Functional Rotterdam - 5th Edition

Lunatech management realises MBO

Syntactically correct and type-safe JPA queries in Play 2.0

Java Persistence API query language

The `.gitlab-ci.yml` File