Introduction to Maker, part 2

Part 2, building our first project with Maker:

In the second part of this series of blogs we'll turn from fetching and bootstrapping Maker itself (from sources) as covered in part 1 to actually using Maker to build a simple project. This will introduce us to the process of defining projects in Maker and in particular the key data structures and APIs involved.

The first part of the series is here if you're new to this blog:

http://louisbotterill.blogspot.co.uk/2012/05/introduction-to-maker-part-1.html

Ok, so starting with the basics. A Maker based build project definition is essentially a Scala file defining the project structures and dependencies. Perhaps the easiest way to get started with a project definition file is to treat it as a Scala script and let the maker script (bin/maker.sh) load the project into the Scala REPL for you.

Maker can pre-compile project definitions for speed, say where a more complex build is perhaps slow in terms of loading/interpretation in the REPL. For now we'll stick to the simple case, a simple project defined in a single Scala file, loaded by the maker script. The example project used in this blog will be a simple Hello world web application. Compiling project definitions for speed will be covered later in this series of blogs.


Defining the application sources:

Let's start by making a directory to hold the whole project. We'll call it testwebapp.

$mkdir testwebapp

While we're at it, let's make the directory structures for the source code.

$mkdir -p src/main/scala
$mkdir -p src/main/webapp

The location of the source directories and output directories is configurable in Maker. The default is a standard Maven style layout. Layouts are easy to customise and other layouts can be defined, more on this later.


Ivy files:

Maker uses Ivy for dependency management, so lets add an Ivy settings file to define the settings for Ivy (resolvers and properties, etc). Using your favourite editor, create the file ivysettings.xml in the root of the testwebapp directory.

$vi ivysettings.xml

<ivysettings>
  <property name="test" value="webapp" />
  <property name="jetty_version" value="7.6.3.v20120416" />

  <settings>
    <settings name="default" transitive="false"/>
  </settings>
  <settings defaultResolver="default"/>
  
  <resolvers>
    <ibiblio name="central" m2compatible="true"/>
    <chain name="default" returnFirst="true">
      <resolver ref="central"/>
    </chain>
  </resolvers>
</ivysettings>

The above defined the version of Jetty we want to use as a property, turns off transitive dependencies and defines the name of the resolver we want to use (Maven central). This blog isn't meant to be a tutorial on Ivy, but it's fairly simple to get to grips with and you can find documentation here as required. Many concepts are similar to those in Maven, so if you've ever user Maven then you won't feel too out of place.

[A quick aside on transitive dependencies; Whilst transitive dependencies can be used (since Maker uses Ivy and Ivy it supports this feature) a suggestion would be to switch this feature off and define dependencies explicitly - that way there can be no nasty surprises as your projects grow in size and complexity]

Let's now define the ivy.xml file. This defines the dependencies specific to the project module. Generally in a single build there would be one ivysettings.xml and then one ivy.xml per project module (that has external dependencies).

$vi ivy.xml

<ivy-module version="1.0" xmlns:e="http://ant.apache.org/ivy/extra">
  <info organisation="${group_id}" module="utils" revision="${maker.module.version}" />
  
  <configurations>
    <conf name="default" transitive="false"/>
    <conf name="compile" transitive="false"/>
    <conf name="test" transitive="false"/>
  </configurations>

  <publications>
    <artifact name="test-webapp" type="pom"/>
    <artifact name="test-webapp" type="jar" ext="jar" conf="default" />
  </publications>

  <dependencies defaultconfmapping="*->default,sources">
    <dependency org="log4j" name="log4j" rev="1.2.16" />
    <dependency org="org.slf4j" name="slf4j-api" rev="1.6.1"/>
    <dependency org="org.slf4j" name="slf4j-log4j12" rev="1.6.1" />

    <dependency org="org.eclipse.jetty" name="jetty-server" rev="${jetty_version}" />
    <dependency org="org.eclipse.jetty" name="jetty-webapp" rev="${jetty_version}" />
    <dependency org="org.eclipse.jetty" name="jetty-util" rev="${jetty_version}" />
    <dependency org="org.eclipse.jetty" name="jetty-servlet" rev="${jetty_version}" />
    <dependency org="org.eclipse.jetty" name="jetty-security" rev="${jetty_version}" />
    <dependency org="org.eclipse.jetty" name="jetty-http" rev="${jetty_version}" />
    <dependency org="org.eclipse.jetty" name="jetty-io" rev="${jetty_version}" />
    <dependency org="org.eclipse.jetty" name="jetty-xml" rev="${jetty_version}" />
  
    <dependency org="javax.servlet" name="servlet-api" rev="2.5" />
  </dependencies>
</ivy-module>

ivysettings.xml and ivy.xml are the default file names Maker will look for when using Ivy to resolve dependencies. Again the name and location of these defaults can be changed.

Ok, so now we have put the files necessary for Ivy in place and the source directory structure, lets have a look at defining a minimal Maker project file.

Creating a Maker project definition:

Create a file called say maker.scala, in the root of the project directory.

$vi maker.scala

import maker.project.Project

import maker.utils.FileUtils._

import maker.Props

lazy val properties : Props = file("Maker.conf")

val webApp = Project(

  file("."),

  "test-webapp", 

  props = properties,

  webAppDir = Some(file(".", "src/main/webapp"))

)

So this file defines one Maker project, called "test-webapp", as an instance called webApp. Maker projects take a number of arguments but most can be defaulted. At a minimum just the project root directory must be specified. Since we're making this project a single self contained project, the root directory is just the current project directory. The presence of the webAppDir means this application will be treated as a web application, not a regular jar module. In case you're wondering, file(...) is just one of a number of helper function defined in the object maker.utils.FileUtils that make working with Java files a bit less tedious.

A Maker Project is a regular Scala case class defining a build module. All tasks, such as clean, compile and test are invoked on a project instance.

The key parameters to a Maker Project definition are:

• root : File - a File representing the root of this project
• name : String - a name for the project, can be omitted in which case it defaults to the root file name
• layout : ProjectLayout - a case class defining the file locations all all inputs (source files etc) and outputs (classes etc)
• props : Props - a key-value property file defining any properties for maker, this may be omitted as Maker does default most properties
• webAppDir :Option[File] - this is an optional file, when provided it represents the directory containing the webapp content (WEB-INF etc) and means the project acts like a web application (produces a war packaged artifact, and can be run using the embedded Jetty runner, for example).

Documentation on the Project class can be viewed here: http://cage433.github.com/maker/maker/target/docs/#maker.project.Project

So that's it for defining the build of a simple web application. Maker will default everything else required for a typical build.

Lets add some content to make this simple Hello world project functional.

Adding some web content:

First we'll define some static resources, a simple test web page:

$vi src/main/webapp/hello.html 

<html>
<body>
Hello world - from html
</body>
</html>

and a simple servlet for good measure:

package test

import javax.servlet.http.{HttpServlet, HttpServletRequest, HttpServletResponse}

class HelloServlet extends HttpServlet {

  println("Initialised HelloServlet")

  override def doGet(req : HttpServletRequest, resp : HttpServletResponse) =
    resp.getWriter().print("<HTML>" +
      "<HEAD><TITLE>Hello, Scala!</TITLE></HEAD>" +
      "<BODY>Hello, Scala! This is a servlet.</BODY>" +
      "</HTML>")
}

and now the files necessary for the web application:

$ cat src/main/webapp/WEB-INF/web.xml 

<?xml version="1.0" encoding="ISO-8859-1"?>

<web-app xmlns="http://java.sun.com/xml/ns/j2ee"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_5.xsd"
         Version="2.5">

  <display-name>Test webapp</display-name>

  <servlet>
    <servlet-name>hello</servlet-name>
    <servlet-class>
      test.HelloServlet
    </servlet-class>
    <load-on-startup>1</load-on-startup>
  </servlet>

  <servlet-mapping>
    <servlet-name>hello</servlet-name>
    <url-pattern>/servlets/hello</url-pattern>
  </servlet-mapping>

</web-app> 

Which wires up our test servlet. So that's all the plumbing to put together a trivial test hello-world web application.

Booting the project in Maker:

Let's now boot Maker with the project definition. This is done by running the bin/maker.sh script that you should have from part one of this blog. Unlike in the last part where we used the -b switch to get Maker to build and bootstrap itself from sources, we're now using that build Maker to load a user defined project, with the -p option.

$path/to/maker/bin/maker/.sh -p $path/to/testwebapp/maker/scala

for example, if Maker was a sibling to our test project directory, we might type the following from the root of the test project directory:

$ ../maker/bin/maker.sh -p maker.scala 

remembering that the maker.sh script is the entry point for maker and the -p option specifies a Maker project definition file to load - you should be greeted with the following REPL prompt:

Loading maker.scala...

import org.apache.log4j.Level._

import maker.project.Project

import maker.Props

import maker.RichProperties._

import maker.utils.FileUtils._

import maker.utils.Log

makerProps: maker.Props =

webApp: maker.project.Project = Project org.acme:test-webapp

Welcome to Scala version 2.9.1.final (Java HotSpot(TM) 64-Bit Server VM, Java 1.6.0_33).

Type in expressions to have them evaluated.

Type :help for more information.

scala>

Updating and compiling:


From here we can issue Maker commands as required. If you type webApp (the name of our project instance) and the dot (.) and then tab you should get a list of available methods. There are many, but some will jump out as immediately familiar, such as clean, test and so on. For a new project, the first thing to do is invoke Ivy to update any project dependencies:

scala> webApp.update

which should then output some details about the dependencies as they are resolved (via Ivy) and downloaded, ending in some output like:

...

:: retrieving :: ${group_id}#utils [sync]

        confs: [default]

        24 artifacts copied, 0 already retrieved (3001kB/90ms)

24 Jun 2012 22:58:57:682 INFO  root  - Completed ProjectAndTask(Project org.acme:test-webapp,UpdateTask), took  409(ms)

...

result was Success

There are two things to note here, by default;

1 Maker only invokes Ivy when instructed (this can be overridden to be fully automatic before compilation)

2 Maker puts the dependencies in a sub-directory of the project, by default called lib_managed (like SBT) - again this is configurable as you'll see shortly

We can check this. Every Maker project has a layout, by default Maker uses the Maven style layout. Within the layout is a definition of where the managed libraries go (managed libraries being the name of those 'managed' by Ivy, as opposed to user managed libraries that have been provided by hand) 

Within a project's layout the field called managedLibDir is a regular java.io.File, so we could use it perhaps to list the files in that directory:

scala> webApp.layout.managedLibDir.listFiles.foreach(println)

./lib_managed/jetty-http-7.6.3.v20120416-sources.jar

./lib_managed/jetty-http-7.6.3.v20120416.jar

./lib_managed/jetty-io-7.6.3.v20120416-sources.jar

./lib_managed/jetty-io-7.6.3.v20120416.jar

./lib_managed/jetty-security-7.6.3.v20120416-sources.jar

./lib_managed/jetty-security-7.6.3.v20120416.jar

./lib_managed/jetty-server-7.6.3.v20120416-sources.jar

./lib_managed/jetty-server-7.6.3.v20120416.jar

./lib_managed/jetty-servlet-7.6.3.v20120416-so

...

These are the dependency files that Maker has fetched for our project (using Ivy) and put in the lib_managed directory of our project. Of course these libraries will be cached in the ~/.ivy2 Ivy cache directory first, so once populated the libraries will only be fetched from the Internet when new dependencies are added. All unmanaged and managed libraries are added to the classpath for compilation and to packaged artifacts (e.g. .wars) as would be expected.


Ok, so far so good. Now we have a loaded project with dependencies resolved, we can compile it:



scala> webApp.compile
14:21:21 INFO  - Starting task test-webapp, Source compile, press ctrl-] to terminate
14:21:23 INFO  - Completed test-webapp, Source compile, took  1(s) 563(ms)


res1: maker.task.BuildResult = 
test-webapp, Source compile result was Success
     Project: Project org.acme:test-webapp
     Task   : Source compile


scala> 

So the project has compiled ok, in around 1.5s in my machine.

If we had added some tests then we could run webApp.test to compile and run the tests - but as we've not added any for this simple example we'll skip this step.

The presence of the webAppDir on the project definition triggers certain web application specific features that are not available for standard Java projects. For example, webApp.package will package up a standard format WebARchive (.war) file for us rather than a JavaARchive (.jar).


Running the web-app within Maker:

For web applications, Maker can run the application using an embedded Jetty container using the runJetty task. By default this runs your project Jetty on port 8080, but a new port number can optionally be supplied to the task if required.

So let's give that a go now.

scala> webApp.runJetty
14:26:29 INFO  - Starting task test-webapp, RunJettyTask, press ctrl-] to terminate
14:26:29 INFO  - 
Packaging project dirs:
14:26:29 INFO  - /Users/louis/dev/projects/opensource/github/testwebapp/./src/main/resources
14:26:29 INFO  - /Users/louis/dev/projects/opensource/github/testwebapp/./target/classes
14:26:29 INFO  - Packaging web app, web app dir = /Users/louis/dev/projects/opensource/github/testwebapp/./src/main/webapp
14:26:29 INFO  - Making war image.../Users/louis/dev/projects/opensource/github/testwebapp/./target/package/webapp
14:26:29 INFO  - Packaging artifact /Users/louis/dev/projects/opensource/github/testwebapp/./target/package/test-webapp.war
14:26:30 INFO  - running webapp of project test-webapp
14:26:31 INFO  - Starting HTTP on port: 8080
14:26:31 INFO  - jetty-7.6.3.v20120416
14:26:31 INFO  - Extract jar:file:/Users/louis/dev/projects/opensource/github/testwebapp/target/package/test-webapp.war!/ to /private/var/folders/wv/s7b3m27j39sbct3d97rhs8km0000gn/T/jetty-0.0.0.0-8080-test-webapp.war-_test-webapp-any-/webapp
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/private/var/folders/wv/s7b3m27j39sbct3d97rhs8km0000gn/T/jetty-0.0.0.0-8080-test-webapp.war-_test-webapp-any-/webapp/WEB-INF/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/louis/dev/projects/opensource/github/maker/.maker/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
14:26:33 INFO  - started o.e.j.w.WebAppContext{/test-webapp,file:/private/var/folders/wv/s7b3m27j39sbct3d97rhs8km0000gn/T/jetty-0.0.0.0-8080-test-webapp.war-_test-webapp-any-/webapp/},/Users/louis/dev/projects/opensource/github/testwebapp/./target/package/test-webapp.war
Initialised HelloServlet
14:26:33 INFO  - Started SelectChannelConnector@0.0.0.0:8080
14:26:33 INFO  - Press ctrl-] to end...

Now Jetty is running and Maker is waiting for that process to finish. Since Web Applications don't really terminate from inside the user code, the way to quit this long-running process is to type ctrl-] when we're done. Don't do that just yet, we'll check it works first.

From your favourite browser we should be able to view the web content here.

http://localhost:8080/test-webapp/

and the static web page here:

http://localhost:8080/test-webapp/hello.html

and the servlet using:

http://localhost:8080/test-webapp/servlets/hello

Ok, so it works. Press ctrl-] in the REPL to quit the Jetty runner and return to the REPL prompt.

So far so good. We've seen how to define a simple project (a web application in this example) and define the necessary minimal Maker project definition necessary to build, test and run it.

Before we wrap up this second bog in the series, it's worth just highlighting the project layout as it's quite likely you might want to customise it for your real-world projects at some stage.


Customising project layouts:

Within Project, the layout field is a ProjectLayout (as mentioned earlier). Maker currently provides two default layouts; 'maven' and 'maker', these are regular values (val fields) in the companion ProjectLayout class.  The default maven layout follows the standard maven layout in terms of sources and output classes. Similarly the maker layout is similar but 'flattened' out and is the layout maker uses for its own project.

Any existing layout can be modified, or you're free to create your own instance of ProjectLayout in any way you want to meet your project needs. Being a simple case class we can use the free copy function to make modifications to the immutable class.

Here's a quick example. Say we want a Maven based layout, but with a couple of changes. We want to add another source directory and rename the managed lib directory - this is one way to do that:


...


import maker.utils.FileUtils._


def myLayout(root : File) = { 
  val layout = ProjectLayout.maven(root)
  layout.copy(sourceDirs = new File(root, "my_extra_src") :: layout.sourceDirs, managedLibDir = file(root, "managed_libs))
}


val webApp = Project(

  file("."),

  "test-webapp",
  layout = myLayout(file("."))

  props = properties,

  webAppDir = Some(file(".", "src/main/webapp"))

)


...

Also layout has some utility functions, so to just add a source directory we might more concisely do this:

...



val baseWebApp = Project(

  file("."),

  "test-webapp",

  props = properties,

  webAppDir = Some(file(".", "src/main/webapp"))
)
val webApp = baseWebApp.copy(
  layout = baseWebApp.layout.withSourceDirs(List("src1", "src2").map(file(baseWebApp.root, _))

)


...

Any attribute of the layout can be customised in this way, or you might want to define new layouts from scratch if they are significantly different from those provided.


Documentation on the layout class: http://cage433.github.com/maker/maker/target/docs/#maker.project.ProjectLayout

Now that we've covered project definitions and using maker to compile and run, a few quick words on properties. 


Maker provides defaults for most parameters so that it works out-of-the-box with minimal configuration, but most aspects can be configured either using properties or via optional arguments to the tasks.


In our simple maker project definition you'll notice that we created a Maker.conf based Props class and passed it to the project.




Maker properties:


Various things can be configured using the property file. For example it was previously mentioned that the default Ivy update on compile could be changed. So in this example we could achieve that by adding the property to the Maker.conf file.

$vi Maker.conf


UpdateOnCompile=true

Now when we run any task that compiles (or depends on compilation) the Ivy update task will run first.




Wrapping up:


That about concludes this second blog in the series of byte-sized introductions to Maker. We've still not covered several key aspects, but these should get covered in up and coming blogs of this series. Some of the topics still to be covered are:

• Accessing Maker documentation
• Continually running tasks - e.g. being able to run tests continually while you develop code to make them pass
• Testing - how unit tests can be run and inspected. Re-running failing tests
• Packaging - examples of how packaging works and what can be configured to give different packaging options
• Documentation - producing scala-doc documentation from Maker projects
• Publishing - how to publish Maven style artifacts from Maker to repositories like Nexus.
• Analyzing failures - identifying compilation or test problems, inspecting build results in detail
• Dependency analysis - viewing dependency graphs and querying dependencies
• Compiling project files - for speed of Maker start up

And, finally - I'd like to cover getting started with Maker as a packaged tool, distributed from binaries - but we're still working on this part. So for now the sequence described in blog part one is a valid way to 'bootstrap' maker straight from sources. Hopefully we'll have a good solution for using Maker from a binary distribution soon and a forthcoming blog will then cover how to install and use it in this way.


Documentation: Scala-docs from Github can be found here: http://cage433.github.com/maker/maker/target/docs/




Now you've seen how easy it is to start using Maker to compile your Scala and Java projects, what are you waiting for - get using it and please give us your valuable feedback!

Any problems, issue tracking is on GitHub: https://github.com/cage433/maker/issues?sort=created&state=open

<<< introduction to maker part 1