Saturday 30 June 2012

Introduction to Maker, part 2

Part 2, building our first project with Maker:

In the second part of this series of blogs we'll turn from fetching and bootstrapping Maker itself (from sources) as covered in part 1 to actually using Maker to build a simple project. This will introduce us to the process of defining projects in Maker and in particular the key data structures and APIs involved.

The first part of the series is here if you're new to this blog:

http://louisbotterill.blogspot.co.uk/2012/05/introduction-to-maker-part-1.html

Ok, so starting with the basics. A Maker based build project definition is essentially a Scala file defining the project structures and dependencies. Perhaps the easiest way to get started with a project definition file is to treat it as a Scala script and let the maker script (bin/maker.sh) load the project into the Scala REPL for you.

Maker can pre-compile project definitions for speed, say where a more complex build is perhaps slow in terms of loading/interpretation in the REPL. For now we'll stick to the simple case, a simple project defined in a single Scala file, loaded by the maker script. The example project used in this blog will be a simple Hello world web application. Compiling project definitions for speed will be covered later in this series of blogs.


Defining the application sources:

Let's start by making a directory to hold the whole project. We'll call it testwebapp.

$mkdir testwebapp

While we're at it, let's make the directory structures for the source code.

$mkdir -p src/main/scala
$mkdir -p src/main/webapp

The location of the source directories and output directories is configurable in Maker. The default is a standard Maven style layout. Layouts are easy to customise and other layouts can be defined, more on this later.


Ivy files:

Maker uses Ivy for dependency management, so lets add an Ivy settings file to define the settings for Ivy (resolvers and properties, etc). Using your favourite editor, create the file ivysettings.xml in the root of the testwebapp directory.

$vi ivysettings.xml

<ivysettings>
  <property name="test" value="webapp" />
  <property name="jetty_version" value="7.6.3.v20120416" />

  <settings>
    <settings name="default" transitive="false"/>
  </settings>
  <settings defaultResolver="default"/>
  
  <resolvers>
    <ibiblio name="central" m2compatible="true"/>
    <chain name="default" returnFirst="true">
      <resolver ref="central"/>
    </chain>
  </resolvers>
</ivysettings>

The above defined the version of Jetty we want to use as a property, turns off transitive dependencies and defines the name of the resolver we want to use (Maven central). This blog isn't meant to be a tutorial on Ivy, but it's fairly simple to get to grips with and you can find documentation here as required. Many concepts are similar to those in Maven, so if you've ever user Maven then you won't feel too out of place.

[A quick aside on transitive dependencies; Whilst transitive dependencies can be used (since Maker uses Ivy and Ivy it supports this feature) a suggestion would be to switch this feature off and define dependencies explicitly - that way there can be no nasty surprises as your projects grow in size and complexity]

Let's now define the ivy.xml file. This defines the dependencies specific to the project module. Generally in a single build there would be one ivysettings.xml and then one ivy.xml per project module (that has external dependencies).

$vi ivy.xml


<ivy-module version="1.0" xmlns:e="http://ant.apache.org/ivy/extra">
  <info organisation="${group_id}" module="utils" revision="${maker.module.version}" />
  
  <configurations>
    <conf name="default" transitive="false"/>
    <conf name="compile" transitive="false"/>
    <conf name="test" transitive="false"/>
  </configurations>

  <publications>
    <artifact name="test-webapp" type="pom"/>
    <artifact name="test-webapp" type="jar" ext="jar" conf="default" />
  </publications>

  <dependencies defaultconfmapping="*->default,sources">
    <dependency org="log4j" name="log4j" rev="1.2.16" />
    <dependency org="org.slf4j" name="slf4j-api" rev="1.6.1"/>
    <dependency org="org.slf4j" name="slf4j-log4j12" rev="1.6.1" />

    <dependency org="org.eclipse.jetty" name="jetty-server" rev="${jetty_version}" />
    <dependency org="org.eclipse.jetty" name="jetty-webapp" rev="${jetty_version}" />
    <dependency org="org.eclipse.jetty" name="jetty-util" rev="${jetty_version}" />
    <dependency org="org.eclipse.jetty" name="jetty-servlet" rev="${jetty_version}" />
    <dependency org="org.eclipse.jetty" name="jetty-security" rev="${jetty_version}" />
    <dependency org="org.eclipse.jetty" name="jetty-http" rev="${jetty_version}" />
    <dependency org="org.eclipse.jetty" name="jetty-io" rev="${jetty_version}" />
    <dependency org="org.eclipse.jetty" name="jetty-xml" rev="${jetty_version}" />
  
    <dependency org="javax.servlet" name="servlet-api" rev="2.5" />
  </dependencies>
</ivy-module>

ivysettings.xml and ivy.xml are the default file names Maker will look for when using Ivy to resolve dependencies. Again the name and location of these defaults can be changed.

Ok, so now we have put the files necessary for Ivy in place and the source directory structure, lets have a look at defining a minimal Maker project file.


Creating a Maker project definition:

Create a file called say maker.scala, in the root of the project directory.

$vi maker.scala

import maker.project.Project
import maker.utils.FileUtils._
import maker.Props

lazy val properties : Props = file("Maker.conf")

val webApp = Project(
  file("."),
  "test-webapp", 
  props = properties,
  webAppDir = Some(file(".", "src/main/webapp"))
)

So this file defines one Maker project, called "test-webapp", as an instance called webApp. Maker projects take a number of arguments but most can be defaulted. At a minimum just the project root directory must be specified. Since we're making this project a single self contained project, the root directory is just the current project directory. The presence of the webAppDir means this application will be treated as a web application, not a regular jar module. In case you're wondering, file(...) is just one of a number of helper function defined in the object maker.utils.FileUtils that make working with Java files a bit less tedious.

A Maker Project is a regular Scala case class defining a build module. All tasks, such as clean, compile and test are invoked on a project instance.

The key parameters to a Maker Project definition are:
  • root : File - a File representing the root of this project
  • name : String - a name for the project, can be omitted in which case it defaults to the root file name
  • layout : ProjectLayout - a case class defining the file locations all all inputs (source files etc) and outputs (classes etc)
  • props : Props - a key-value property file defining any properties for maker, this may be omitted as Maker does default most properties
  • webAppDir :Option[File] - this is an optional file, when provided it represents the directory containing the webapp content (WEB-INF etc) and means the project acts like a web application (produces a war packaged artifact, and can be run using the embedded Jetty runner, for example).

Documentation on the Project class can be viewed here: http://cage433.github.com/maker/maker/target/docs/#maker.project.Project

So that's it for defining the build of a simple web application. Maker will default everything else required for a typical build.

Lets add some content to make this simple Hello world project functional.


Adding some web content:

First we'll define some static resources, a simple test web page:

$vi src/main/webapp/hello.html 
<html>
<body>
Hello world - from html
</body>
</html>

and a simple servlet for good measure:
package test

import javax.servlet.http.{HttpServlet, HttpServletRequest, HttpServletResponse}

class HelloServlet extends HttpServlet {

  println("Initialised HelloServlet")

  override def doGet(req : HttpServletRequest, resp : HttpServletResponse) =
    resp.getWriter().print("<HTML>" +
      "<HEAD><TITLE>Hello, Scala!</TITLE></HEAD>" +
      "<BODY>Hello, Scala! This is a servlet.</BODY>" +
      "</HTML>")
}

and now the files necessary for the web application:

$ cat src/main/webapp/WEB-INF/web.xml 


<?xml version="1.0" encoding="ISO-8859-1"?>

<web-app xmlns="http://java.sun.com/xml/ns/j2ee"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_5.xsd"
         Version="2.5">

  <display-name>Test webapp</display-name>

  <servlet>
    <servlet-name>hello</servlet-name>
    <servlet-class>
      test.HelloServlet
    </servlet-class>
    <load-on-startup>1</load-on-startup>
  </servlet>

  <servlet-mapping>
    <servlet-name>hello</servlet-name>
    <url-pattern>/servlets/hello</url-pattern>
  </servlet-mapping>

</web-app> 

Which wires up our test servlet. So that's all the plumbing to put together a trivial test hello-world web application.




Booting the project in Maker:

Let's now boot Maker with the project definition. This is done by running the bin/maker.sh script that you should have from part one of this blog. Unlike in the last part where we used the -b switch to get Maker to build and bootstrap itself from sources, we're now using that build Maker to load a user defined project, with the -p option.

$path/to/maker/bin/maker/.sh -p $path/to/testwebapp/maker/scala

for example, if Maker was a sibling to our test project directory, we might type the following from the root of the test project directory:

$ ../maker/bin/maker.sh -p maker.scala 


remembering that the maker.sh script is the entry point for maker and the -p option specifies a Maker project definition file to load - you should be greeted with the following REPL prompt:

Loading maker.scala...
import org.apache.log4j.Level._
import maker.project.Project
import maker.Props
import maker.RichProperties._
import maker.utils.FileUtils._
import maker.utils.Log
makerProps: maker.Props =
webApp: maker.project.Project = Project org.acme:test-webapp

Welcome to Scala version 2.9.1.final (Java HotSpot(TM) 64-Bit Server VM, Java 1.6.0_33).
Type in expressions to have them evaluated.
Type :help for more information.

scala>


Updating and compiling:


From here we can issue Maker commands as required. If you type webApp (the name of our project instance) and the dot (.) and then tab you should get a list of available methods. There are many, but some will jump out as immediately familiar, such as clean, test and so on. For a new project, the first thing to do is invoke Ivy to update any project dependencies:

scala> webApp.update

which should then output some details about the dependencies as they are resolved (via Ivy) and downloaded, ending in some output like:

...

:: retrieving :: ${group_id}#utils [sync]
        confs: [default]
        24 artifacts copied, 0 already retrieved (3001kB/90ms)
24 Jun 2012 22:58:57:682 INFO  root  - Completed ProjectAndTask(Project org.acme:test-webapp,UpdateTask), took  409(ms)
...
result was Success



There are two things to note here, by default;

1 Maker only invokes Ivy when instructed (this can be overridden to be fully automatic before compilation)
2 Maker puts the dependencies in a sub-directory of the project, by default called lib_managed (like SBT) - again this is configurable as you'll see shortly

We can check this. Every Maker project has a layout, by default Maker uses the Maven style layout. Within the layout is a definition of where the managed libraries go (managed libraries being the name of those 'managed' by Ivy, as opposed to user managed libraries that have been provided by hand) 

Within a project's layout the field called managedLibDir is a regular java.io.File, so we could use it perhaps to list the files in that directory:

scala> webApp.layout.managedLibDir.listFiles.foreach(println)

./lib_managed/jetty-http-7.6.3.v20120416-sources.jar
./lib_managed/jetty-http-7.6.3.v20120416.jar
./lib_managed/jetty-io-7.6.3.v20120416-sources.jar
./lib_managed/jetty-io-7.6.3.v20120416.jar
./lib_managed/jetty-security-7.6.3.v20120416-sources.jar
./lib_managed/jetty-security-7.6.3.v20120416.jar
./lib_managed/jetty-server-7.6.3.v20120416-sources.jar
./lib_managed/jetty-server-7.6.3.v20120416.jar
./lib_managed/jetty-servlet-7.6.3.v20120416-so
...


These are the dependency files that Maker has fetched for our project (using Ivy) and put in the lib_managed directory of our project. Of course these libraries will be cached in the ~/.ivy2 Ivy cache directory first, so once populated the libraries will only be fetched from the Internet when new dependencies are added. All unmanaged and managed libraries are added to the classpath for compilation and to packaged artifacts (e.g. .wars) as would be expected.


Ok, so far so good. Now we have a loaded project with dependencies resolved, we can compile it:



scala> webApp.compile
14:21:21 INFO  - Starting task test-webapp, Source compile, press ctrl-] to terminate
14:21:23 INFO  - Completed test-webapp, Source compile, took  1(s) 563(ms)


res1: maker.task.BuildResult = 
test-webapp, Source compile result was Success
     Project: Project org.acme:test-webapp
     Task   : Source compile


scala> 

So the project has compiled ok, in around 1.5s in my machine.

If we had added some tests then we could run webApp.test to compile and run the tests - but as we've not added any for this simple example we'll skip this step.

The presence of the webAppDir on the project definition triggers certain web application specific features that are not available for standard Java projects. For example, webApp.package will package up a standard format WebARchive (.war) file for us rather than a JavaARchive (.jar).


Running the web-app within Maker:

For web applications, Maker can run the application using an embedded Jetty container using the runJetty task. By default this runs your project Jetty on port 8080, but a new port number can optionally be supplied to the task if required.

So let's give that a go now.

scala> webApp.runJetty
14:26:29 INFO  - Starting task test-webapp, RunJettyTask, press ctrl-] to terminate
14:26:29 INFO  - 
Packaging project dirs:
14:26:29 INFO  - /Users/louis/dev/projects/opensource/github/testwebapp/./src/main/resources
14:26:29 INFO  - /Users/louis/dev/projects/opensource/github/testwebapp/./target/classes
14:26:29 INFO  - Packaging web app, web app dir = /Users/louis/dev/projects/opensource/github/testwebapp/./src/main/webapp
14:26:29 INFO  - Making war image.../Users/louis/dev/projects/opensource/github/testwebapp/./target/package/webapp
14:26:29 INFO  - Packaging artifact /Users/louis/dev/projects/opensource/github/testwebapp/./target/package/test-webapp.war
14:26:30 INFO  - running webapp of project test-webapp
14:26:31 INFO  - Starting HTTP on port: 8080
14:26:31 INFO  - jetty-7.6.3.v20120416
14:26:31 INFO  - Extract jar:file:/Users/louis/dev/projects/opensource/github/testwebapp/target/package/test-webapp.war!/ to /private/var/folders/wv/s7b3m27j39sbct3d97rhs8km0000gn/T/jetty-0.0.0.0-8080-test-webapp.war-_test-webapp-any-/webapp
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/private/var/folders/wv/s7b3m27j39sbct3d97rhs8km0000gn/T/jetty-0.0.0.0-8080-test-webapp.war-_test-webapp-any-/webapp/WEB-INF/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/louis/dev/projects/opensource/github/maker/.maker/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
14:26:33 INFO  - started o.e.j.w.WebAppContext{/test-webapp,file:/private/var/folders/wv/s7b3m27j39sbct3d97rhs8km0000gn/T/jetty-0.0.0.0-8080-test-webapp.war-_test-webapp-any-/webapp/},/Users/louis/dev/projects/opensource/github/testwebapp/./target/package/test-webapp.war
Initialised HelloServlet
14:26:33 INFO  - Started SelectChannelConnector@0.0.0.0:8080
14:26:33 INFO  - Press ctrl-] to end...



Now Jetty is running and Maker is waiting for that process to finish. Since Web Applications don't really terminate from inside the user code, the way to quit this long-running process is to type ctrl-] when we're done. Don't do that just yet, we'll check it works first.

From your favourite browser we should be able to view the web content here.

http://localhost:8080/test-webapp/

and the static web page here:

http://localhost:8080/test-webapp/hello.html

and the servlet using:

http://localhost:8080/test-webapp/servlets/hello

Ok, so it works. Press ctrl-] in the REPL to quit the Jetty runner and return to the REPL prompt.

So far so good. We've seen how to define a simple project (a web application in this example) and define the necessary minimal Maker project definition necessary to build, test and run it.

Before we wrap up this second bog in the series, it's worth just highlighting the project layout as it's quite likely you might want to customise it for your real-world projects at some stage.


Customising project layouts:

Within Project, the layout field is a ProjectLayout (as mentioned earlier). Maker currently provides two default layouts; 'maven' and 'maker', these are regular values (val fields) in the companion ProjectLayout class.  The default maven layout follows the standard maven layout in terms of sources and output classes. Similarly the maker layout is similar but 'flattened' out and is the layout maker uses for its own project.

Any existing layout can be modified, or you're free to create your own instance of ProjectLayout in any way you want to meet your project needs. Being a simple case class we can use the free copy function to make modifications to the immutable class.

Here's a quick example. Say we want a Maven based layout, but with a couple of changes. We want to add another source directory and rename the managed lib directory - this is one way to do that:


...


import maker.utils.FileUtils._


def myLayout(root : File) = { 
  val layout = ProjectLayout.maven(root)
  layout.copy(sourceDirs = new File(root, "my_extra_src") :: layout.sourceDirs, managedLibDir = file(root, "managed_libs))
}


val webApp = Project(
  file("."),
  "test-webapp",
  layout = myLayout(file("."))
  props = properties,
  webAppDir = Some(file(".", "src/main/webapp"))
)


...



Also layout has some utility functions, so to just add a source directory we might more concisely do this:

...



val baseWebApp = Project(
  file("."),
  "test-webapp",
  props = properties,
  webAppDir = Some(file(".", "src/main/webapp"))
)
val webApp = baseWebApp.copy(
  layout = baseWebApp.layout.withSourceDirs(List("src1", "src2").map(file(baseWebApp.root, _))
)


...

Any attribute of the layout can be customised in this way, or you might want to define new layouts from scratch if they are significantly different from those provided.


Documentation on the layout class: http://cage433.github.com/maker/maker/target/docs/#maker.project.ProjectLayout

Now that we've covered project definitions and using maker to compile and run, a few quick words on properties. 


Maker provides defaults for most parameters so that it works out-of-the-box with minimal configuration, but most aspects can be configured either using properties or via optional arguments to the tasks.


In our simple maker project definition you'll notice that we created a Maker.conf based Props class and passed it to the project.




Maker properties:


Various things can be configured using the property file. For example it was previously mentioned that the default Ivy update on compile could be changed. So in this example we could achieve that by adding the property to the Maker.conf file.

$vi Maker.conf


UpdateOnCompile=true

Now when we run any task that compiles (or depends on compilation) the Ivy update task will run first.




Wrapping up:


That about concludes this second blog in the series of byte-sized introductions to Maker. We've still not covered several key aspects, but these should get covered in up and coming blogs of this series. Some of the topics still to be covered are:
  • Accessing Maker documentation
  • Continually running tasks - e.g. being able to run tests continually while you develop code to make them pass
  • Testing - how unit tests can be run and inspected. Re-running failing tests
  • Packaging - examples of how packaging works and what can be configured to give different packaging options
  • Documentation - producing scala-doc documentation from Maker projects
  • Publishing - how to publish Maven style artifacts from Maker to repositories like Nexus.
  • Analyzing failures - identifying compilation or test problems, inspecting build results in detail
  • Dependency analysis - viewing dependency graphs and querying dependencies
  • Compiling project files - for speed of Maker start up
And, finally - I'd like to cover getting started with Maker as a packaged tool, distributed from binaries - but we're still working on this part. So for now the sequence described in blog part one is a valid way to 'bootstrap' maker straight from sources. Hopefully we'll have a good solution for using Maker from a binary distribution soon and a forthcoming blog will then cover how to install and use it in this way.


Documentation: Scala-docs from Github can be found here: http://cage433.github.com/maker/maker/target/docs/




Now you've seen how easy it is to start using Maker to compile your Scala and Java projects, what are you waiting for - get using it and please give us your valuable feedback!

Any problems, issue tracking is on GitHub: https://github.com/cage433/maker/issues?sort=created&state=open

<<< introduction to maker part 1


Saturday 12 May 2012

Introduction to Maker, part 1


Introduction to Maker, part 1:

Maker - A simple build system for Scala (and Java) projects

Maker is a fast, yet simple, command line build system for development of Scala (and Java) projects.

Main features, at a glance:
  • Fast, incremental, parallel compilation. Maker will utilize as many cores as you have available to parallelise non-sequential module dependencies
  • Simple, flexible project configuration and APIs
  • Standard interactive REPL based session
  • Integrated dependency management (delegated to and handled by Ivy)
  • Test Runner integration
  • Can transparently run 'offline'
  • Supports web-applications, with embedded Jetty runner
  • Integrates with Maven and Nexus for standard artifact publishing

So why Maker?

If you work with Scala, SBT is the natural build system everyone tends to gravitate towards. However the S in SBT once stood for Simple. Since the changes from .1x onwards this is perhaps less so. While SBT is powerful, its APIs and dependency management can be opaque and difficult to understand and get right (particularly on large projects).

Maker aims to put the 'Simple' back into a build tool for Scala, whilst still retaining the performance and feature set we've come to expect from a modern build tool.

Whilst still very much a work in progress, we are dog-food-ing this build system on our own project and we're almost at the point where we'll switch out SBT 0.1x for this replacement. Maker should be able to build projects of all sizes, but as with all new projects there may be some feature gaps - these should be closed down over time.


Maker resource quick-links:

GitHub Wiki: https://github.com/cage433/maker/wiki

Jenkins continuous integration: http://www.chillipower.com:8081


By means of an example of bootstrapping maker and using it to build itself, this blog is aimed at the beginner and will hopefully show the basics of getting started. So here goes.

Getting started; getting the code and bootstrapping the build:

Maker is in GitHub, using a Git client, clone the repo:

$ git clone git@github.com:cage433/maker.git maker

$ cd maker


Maker is launched through a single script, this is bin/maker.sh, we can get a list of options using --help:

maker$ bin/maker.sh --help

usage
maker.sh

options
-h, --help
-p, --project-file
-c, --cmd
run command directly then quit
-j, --use-jrebel (requires JREBEL_HOME to be set)
-m, --mem-heap-space
default is one quarter of available RAM
-y, --do-ivy-update
update will always be done if /.maker/lib doesn't exist
-b, --boostrap
builds maker.jar from scratch
-d, --download-project-scala-lib
downloads scala compiler and library to /.maker/scala-lib
download is automatic if this directory does not exist
-x, --allow-remote-debugging
runs a remote JVM
-i, --developer-mode
For maker developers.
Sets the maker classpath to maker/classes:utils/classes etc rather than
maker.jar. Allows work on maker and another project to be done simultaneously.
-nr, --no-repl
skip repl launch (just performs bootstrapping/building and returns)
-ntty, --no-tty-restore
skip save and restore tty (for integration with automation such as TeamCity reporting)
--args, --additional-args
additional variable length argument list to pass to JVM process directly. Must come at the end of the arguments
--mem-permgen-space
default is 1/10th of heap space
--ivy-proxy-host
--ivy-proxy-port
--ivy-non-proxy-hosts
--ivy-jar
defaults to /usr/share/java/ivy.jar


Now we can bootstrap maker by building a maker jar lib, this is done through the main maker script

maker$ bin/maker.sh -y -b

-y tells maker to fetch dependency libraries required by maker
-b tells maker to bootstrap itself using a brute force build from sources

(note, if you're connecting to the internet via a proxy you might also need to specify –ivy-proxy-host and –ivy-proxy-port options).

This should take a few seconds and then you'll find yourself in the probably familiar Scala repl.

At this stage maker has fetched dependencies, built the maker jar and booted it on the classpath. If you want to just use it to make another project then you could quit and start using it immediately (which I'll show in a subsequent blog of this series)

But while we're here, lets take a quick look around…

Maker has the standard concept of projects which can be viewed as modules of software compilation. These projects can be joined up in a dependency tree – just as you would expect from a build tool. In maker's own project definition, the parent module is called mkr and we can check that in the repl:

scala> mkr

res3: maker.project.Project = Project com.google.code.maker:maker

Here we see that the project is defined as an instance of the class maker.project.Project

Projects are where all the action happens. Now that we've bootstrapped maker, we can ask Maker to actually build itself! 

Note on Ivy: By default, updating dependencies from Ivy automatically is disabled (it can be enabled by default using Maker properties, more on that later). So to update Maker's dependencies (makers own build is separated from the bootstrap classpath) we can use the update command on the project.


scala> mkr.update

:: loading settings :: file = maker-ivysettings.xml
:: loading settings :: file = maker/maker-ivy-dynamic.ivy

...

:: resolving dependencies :: com.google.code.maker#utils;${maker.module.version}
confs: [default]
found log4j#log4j;1.2.16 in central
found commons-io#commons-io;2.1 in central
found com.typesafe.akka#akka-actor;2.0 in akka
found com.typesafe.akka#akka-remote;2.0 in akka
found com.typesafe.akka#akka-kernel;2.0 in akka
found org.scala-tools.testing#scalacheck_2.9.1;1.9 in central
found org.scalatest#scalatest_2.9.1;1.7.1 in central
found org.scalaz#scalaz-core_2.9.1;6.0.4 in central
found org.slf4j#slf4j-api;1.6.1 in central
found org.slf4j#slf4j-log4j12;1.6.1 in central
found org.apache.ant#ant;1.8.2 in central
found io.netty#netty;3.4.2.Final in central
found com.google.protobuf#protobuf-java;2.4.1 in central
found net.debasishg#sjson_2.9.1;0.15 in central
found voldemort.store.compress#h2-lzf;1.0 in akka
found org.eclipse.jetty#jetty-server;7.6.3.v20120416 in central
found org.eclipse.jetty#jetty-webapp;7.6.3.v20120416 in central
found org.eclipse.jetty#jetty-util;7.6.3.v20120416 in central
found org.eclipse.jetty#jetty-servlet;7.6.3.v20120416 in central
found org.eclipse.jetty#jetty-security;7.6.3.v20120416 in central
found org.eclipse.jetty#jetty-http;7.6.3.v20120416 in central
found org.eclipse.jetty#jetty-io;7.6.3.v20120416 in central
found org.eclipse.jetty#jetty-xml;7.6.3.v20120416 in central
found org.eclipse.jetty#jetty-continuation;7.6.3.v20120416 in central
found org.eclipse.jetty#jetty-jsp;7.6.3.v20120416 in central
found org.mortbay.jetty#jsp-2.1-glassfish;2.1.v20100127 in central
found javax.servlet#servlet-api;2.5 in central
found org.apache.tomcat#jsp-api;6.0.20 in central
found org.mockito#mockito-all;1.8.2 in central
:: resolution report :: resolve 1022ms :: artifacts dl 74ms
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 29 | 0 | 0 | 0 || 54 | 0 |
---------------------------------------------------------------------
:: retrieving :: com.google.code.maker#utils [sync]
confs: [default]
0 artifacts copied, 54 already retrieved (0kB/43ms)
12 May 2012 09:35:28 INFO root - Task[utils:UpdateTask] completed in 1256ms
12 May 2012 09:35:28 INFO root - Completed Task[maker:UpdateTask], took 1(s) 720(ms)

res4: maker.task.BuildResult[AnyRef] = OK


What you should see (assuming you have connectivity to the internet) is a set of resolved dependencies for Maker and most importantly the BuildResult = OK at the end.

Whilst this project has not yet compiled before, if it had, we might first clean it using:

scala> mkr.clean
12 May 2012 09:37:43 INFO root - Task[maker:CleanTask] completed in 131ms
12 May 2012 09:37:43 INFO root - Completed Task[maker:CleanTask], took 214(ms)

res5: maker.task.BuildResult[AnyRef] = OK


Ok, let's now compile Maker using itself, as a test:

scala> mkr.compile
12 May 2012 09:37:46 INFO root - Compiling Project com.google.code.maker:utils
warning: there were 2 unchecked warnings; re-run with -unchecked for details
12 May 2012 09:38:04 INFO root - Task[utils:CompileSourceTask] completed in 18638ms
12 May 2012 09:38:04 INFO root - Compiling Project com.google.code.maker:plugin
12 May 2012 09:38:10 INFO root - Task[plugin:CompileSourceTask] completed in 5319ms
12 May 2012 09:38:10 INFO root - Compiling Project com.google.code.maker:maker
warning: there were 6 unchecked warnings; re-run with -unchecked for details
12 May 2012 09:38:29 INFO root - Task[maker:CompileSourceTask] completed in 18852ms
12 May 2012 09:38:29 INFO root - Completed Task[maker:CompileSourceTask], took 42(s) 813(ms)

res6: maker.task.BuildResult[AnyRef] = OK


All build commands in Maker are functions that return results, for example, from the compile command we get a result that we can use and inspect. We can even get this result into a named variable and use it in other expressions.

Running compile again will also demonstrate the incremental compilation, note the second compile completes almost instantly and there is no further compilation reported because we are all up to date.

scala> val r = mkr.compile
12 May 2012 09:39:52 INFO root - Completed Task[maker:CompileSourceTask], took 113(ms)

r: maker.task.BuildResult[AnyRef] = OK

scala> r.stats.foreach(println)
Task[maker:CompileSourceTask], took 57ms, status OK
Task[plugin:CompileSourceTask], took 25ms, status OK
Task[utils:CompileSourceTask], took 26ms, status OK


Projects have all the normal tasks you'd expect available on them as functions, the main tasks being:
  • clean – clean up the compilation state (remove classes etc)
  • compile - compile the main source code
  • testCompile – compile the test source code
  • test – run all tests
  • testClass – run a specific test class (or test suite name) by name
  • pack – package up the modules into jars (or wars for webapps)
  • update – update dependency libraries
  • publishLocal – publish artifacts to the local file system
  • publish – publish artifacts to a remote repo
  • runMain – run a main class
  • runJetty – run the module as a web-app (uses embedded Jetty within Maker)

Most of these tasks also have a corresponding [taskname]Only counterpart which runs the task only on the module and not on any descendent modules. By default, all tasks first run on dependency modules as appropriate, as would be expected. Similarly, tasks may depend on each other. For example, project.test will ensure all child modules are compiled and their tests are compiled before starting the tests.

It's also possible to run compilation, all tests, individual tests, and main methods continuously, say for development purposes while you work on code to make a particular test pass.


We can inspect project module dependencies, for example:

scala> mkr.children

res3: List[maker.project.Project] = List(Project com.google.code.maker:plugin)

where the returned value is a list of child projects that this project directly depends on.


We can also see all dependencies including indirect dependencies using:

scala> mkr.allDeps

res4: List[maker.project.Project] = List(Project com.google.code.maker:maker, Project com.google.code.maker:plugin, Project com.google.code.maker:utils)

Here we see that mkr depends on plugin and utils (these are all the project modules in maker).


We can inspect library dependencies, for example:

scala> mkr.classpathDirectoriesAndJars.foreach(println)
maker/classes
maker/java-classes
maker/test-classes
/Users/louis/dev/tools/scala/scala/lib/scala-compiler.jar
/Users/louis/dev/tools/scala/scala/lib/scala-library.jar
maker/resources
plugin/classes
plugin/java-classes
plugin/test-classes
plugin/resources
utils/classes
utils/java-classes
utils/test-classes
libs/ivy-2.2.0-sources.jar
libs/ivy-2.2.0.jar


Let's run Maker's tests, to check all is good:

scala> val r = mkr.test

12 May 2012 09:49:58 INFO root - Compiling Project com.google.code.maker:utils
12 May 2012 09:49:59 INFO root - Task[utils:CompileTestsTask] completed in 1760ms
12 May 2012 09:49:59 INFO root - Testing Project com.google.code.maker:utils
Run starting. Expected test count is: 9
MemoizeTests:
FileUtilsTests:
LogTests:
ProcessIDTests:


Output from the test runs should be displayed in the standard test output format, at the end overall success or failure is reported.

We can also query for a particular fragment of a library name or find out how a module depends on a particular library:

scala> mkr.findLibs("scalatest")

utils/lib_managed/scalatest_2.9.1-1.7.1-sources.jar
utils/lib_managed/scalatest_2.9.1-1.7.1.jar

This tells us that Maker depends on a library named scalatest, via the utils module's managed library dependencies.


We can find the dependency path from one project module to another using:

scala> mkr.dependsOnPaths(utils)

res15: List[List[maker.project.Project]] = List(List(Project com.google.code.maker:maker, Project com.google.code.maker:plugin, Project com.google.code.maker:utils))

This tells us that the project module mkr depends on the module utils via the module path; mkr → plugin → utils

Maker doesn't have a complex module structure, but in a complex tree of module dependencies these functions can be useful when trying to understand project structures and dependencies, perhaps for refactoring.

One last thing, maker can run commands from the command line without launching the repl. As we're about done for this setting, let's drop out of the repl and try it ( :q or ctrl-c quits the repl session in case you didn't know).

maker$ bin/maker.sh -c "mkr.clean"

No project file defined, searching for a maker Scala file
Omitting ivy update as /Users/louis/dev/projects/opensource/github/maker/.maker/lib exists
Omitting bootstrap as /Users/louis/dev/projects/opensource/github/maker/maker.jar exists
setting cmd as -e mkr.clean

Maker v.1

12 May 2012 10:35:40 INFO root - Completed Task[maker:CleanTask], took 151(ms)

returning exit status: 0


There are many more features in maker that will be explored in subsequent blogs. At this stage we're now at least able to obtain and bootstrap maker, and explore maker's basic features to build and test itself.

Maker can do more, including drawing graphical graphs of build and test results as well as module dependencies – more on this in a future blog.

In the next blog I'll walk you through making a new project from scratch. We'll see how to construct a new project definition and use Maker's features to build, test, and run a simple hello-world web application from the repl...

>>> introduction to maker part 2

Wednesday 3 February 2010

GlassFish v3 and Java EE 6 Sun-Oracle roadshow - key notes

GlassFish Roadshow 2010 - London 03-02-2010

The following is some notes I took down during the above event.

In brief summary of GlassFish v3 and Java EE 6, here are some of the key takeaways:

  • GlassFish v3 continues to be developed and supported, as the Java EE 5 & 6 RI app server
  • GlassFish v3 currently has no clustering, but offers OSGi modularization and extensibility
  • Supports and takes advantage of new Java EE 6 specifications
  • Big push on modularity and flexibility in both GlassFish and Java EE 6
  • Java EE 6 supports annotation based EJBs, RESTful web services
  • Java EE 6 greatly simplified configuration, optional web-inf.xml etc
  • Java EE 6 simplied simple class EJBs and improved JPA specification
  • Ongoing road-map for GlassFish, details TBA later this year


Java EE 6 (Roberto Chinnici):

Released Dec 10 2009

Key new features: New API, Web profiles, Pluggabiliy, dependancy injection

New technologies:
  • Jax-RS 1.1
  • Bean validation 1.0
  • DI 1.0
  • CDI 1.0
  • Managed beans 1.0
Closed down gap between EJB and POJO with unification and annotations

CDI works with POJO and EJB classes

Unification of platform types, more uniform programming model

  • EJB 3.1
  • JPA 2.0
  • Servlet 3.0
  • JSF 2.0
  • Connectors 1.6
  • Interceptors 1.1
  • JAX-WS 2.2
  • JSR-109 1.3
  • JSP 2.2
  • JSR-250 1.1
  • JACC 1.4
  • JASPIC 1.1

Key goal in Java EE 6 - flexibility, pruning, extensibility and web profiles


Profiles:

Bundles of technologies targeting specific classes of applications
Decoupled from each other and the platform
Guarantees compatibility of required technologies, individually and in combination - must satisfy all joint requirements


Web Profile:

Modern Java web application stack
First profile to be definite
Mid sized, fully functional (expected 80% web-app coverage), add additional components via extensibility (such as web services API, 3rd party frameworks)


Web profile contents:

Servlet, JSP/EL, JSTL, JSF, Bean Validation, EJB Lite, JPA, JTA, Di, CDI, Managed beans, Interceptors, JSR-250


Pruning:

Goal - to address bloat concerns

2 step process:
Declare components as "Proposed optional"
The make fully optional for the next release

Proposed optional technologies:

JAX-RPC (JAX-WS)
EJB 2.x Entity beans (JPA entity classes)
JAXR (little use)
JSR-88 (deployment, tools API - not used)

* Don't use - use replacements / alternatives


Pluggability / extensibility:

Focus on web tier
Level playing field for 3-rd party libraries and frameworks
Two major extensibility points: Servlets and CDI
Simplify packaging of web-apps
Zero-configuration!


Modular Web applications:

Libraries can contain web-fragment.xml descriptor
web.xml is now optional
Can server resources out of jars with: /META-INF/resources

e.g.
/WEB-INF/lib/catalog.jar
/META-INF/resources/catalog/books.html

e.g. Dojo jar in resources


Web fragments in servlet 3.0:

META-INF.web-fragment.xml

same structure as web.xml, can override in web.xml


Servlet container pluggability:

ServletContainerInitializer interface implements by extensions

@HandlesTypes to declare interest in on or more annotation types

ServletContext now contains method to dynamically register servlets and filters
i.e. ServletContext API has been extended

Registered using META-INF/services


onStartup method gets called with a Set of classes available

** Can only add services at startup, not dynamically once running


Asynchronous HTTP processing:

New programming model for async request processing
e.g. Comet, char, push apps
Opt-in model
Threads are managed by the container

@WebServlet(asyncSupported=true)
public class MyServlet extends HTTPServlet {

}

Goal - Decouple requests from threads

* Changes to the way filters work - thread not attached to the socket, response not written. Filters modified to make safe - option than can be turned on to identify asyncSupported=true
- do this on servlet and any filter involved in the chain

Ideal when waiting for some external resource etc.
Low level API - not very elegant
Idea is that frameworks will use this - see for example Atmosphere framework which is based on annotations - under the hood this async API is used.


JSF 2.0:

Facelet as a standard view declaration language
Composite components
Ajax (declarative and programmatic) e.g. f:ajax tag
Partial state saving (track deltas and sends changes in response, previously all would have been sent even if not changed!)
System events e.g. f:event tag
Resources
Validation e.g. new f:validateBean tag (better integration, validation API. can handle multiple errors, not first one at a time!)


EJB 3.1:

@Singleton beans
@Startup beans (invoked at app startup, works well with Singleton pattern)
@Asynchronous invocations (biggest change, allows non-blocking sync EJB invocations as first class call. Method must be either void or return a Future object)
No interface view (bean impl does not need an interface anymore, now 1 class = 1 EJB!)
Define EJBs directly inside a web app, inside a war file
New API - EJBContainer API works on Java SE, can bootstrap an EJB container in a Java SE application (ideal for testing/development, could be useful in client applications)


Simplified packaging:

EJB class directly into the war file
Previously must built an EJB jar to be included


EJB 3.1 lite:

A subset of EJB 3.1
All types of session beans (stageful, stateless, singleton) - other bean types not supported (timer, entity etc)
Declarative transactions and security
Interceptors
ejb-jar.xml descriptor allows (is optional, probably not useful)

* Class loading, new visibility rules in Java EE spec

Slight differences in class loading rules - if in doubt check the specs


New JNDI Namespaces:

Until now - only java:comp
Now added:
java:module - a module (war, ejb, jar)
java:app - an application
java:global the whole server / cluster

e.g. @Resource(lookup="java:app/CustomerDB") DataSource db;

EJB components have global names now:

e.g. java:global/app1/module2/SomeBeanIcom.acme.Foo

Helps solve problem of remote EJB communication in same app server


Dependency injection (DI):

Combination of DI 1.0 / CDI 1.0
New @Inject annotation
@Inject @LoggedIn User user; (@LoggedIn = "Which one", User = "What")
Beans auto-discovered at start-up
Extensible
Injection metamodel (BeanManager API)
@Resource still available for container resources

* Identified by type and qualifiers (no longer just a string name alone)

* No bean declaration as per Spring declaration, bean discovery at startup
* Injection errors all reported at startup, rather than on use at runtime

Beans can be associated with session - i.e. loggedIn
Beans can be more ephemeral, i.e. for request
Beans can be more long lived, i.e. shopping cart conversation flows
Can add class for beans on the fly, via APIs at runtime

Example of DI annotation:

@Inject
CheckoutHandler(
@LoggedIn User user,
@Reliable @PayBy(CREDIT_CARD)
PaymentProcessor processor,
@Default Cart cart)

* Note that constructor injection is possible
* Note different scopes, PaymentProcessor is probably conversation scope, LoggedIn is session, PaymentProcessor is probably application scope singleton

* Instance and state management handled "for free" by framework/APIs

JAX-RS 1.1:
Already widely adoped
Really a high level HTTP API
Annotation-based programming model
Programmatic API when needed

* think of it as the new HTTP level API (higher level than HTTPServlets, remove low level detail and tedium)

Jax-RS resource class, example:

Identified by the @Path annotation

@Path("widgets./{id}")
@Produces("application/widgets+xml")
public class WidgetResource {
pubic WidgetResource(@PathParam("id") String id { … }

@GET
Widget getWidget() { … }
}

Provides higher level HTTP handling, more declarative, better match with conceptual needs of developer


Bean Validation 1.0:

Integrated with JSF, JPA
Constraints represented by annotations

e.g.

@NotNull
@Size(max=40)
String address;

Fully extensible
@Email
String recipient;

Validation API's for validation directly, create a new validation object etc
Validation of trees of objects is possible (including loops)


JPA 2.0:

Supported for collections of basic types and embeddable objects
JPQL enhancements e.g. CASE WHEN, NULLIF
Pessimistic locking added (annotations added)
Criteria API for dynamic query construction

Criteria API: Uses the canonical metamodel classes

CriteriaBuilder, create criteria
CriterialQuery, typed criteria

Strongly types checking, type parasitised equals checking, compiler errors generated if query does not have the right types etc, so robust and safe query (rather than say String SQL construction directly).

Connectors 1.6 added too (not covered in any detail)


Summary overview:

Improved, more powerful, more flexible, more extensible, easier to use

http://java.sun.com/javaee


--

GlassFish V3 (Alexis Moussine-Pouchkine):

Java EE 6 Reference Implementation (RI)

Geographic download map:

http://maps.glassfish.org/server

Healthy increase in downloads and usage over time

GlassFish V1 first shipped 2006, reusing much from Tomcat

V2.1.1 Nov 2009, V3 (Java EE 6) Dec 10th 2009

GlassFish V3 Open Source CDDL, GPL (with 'classpath exception') licensing

Java EE 5 & 6, enterprise quality - full support is available.

Sub projects:
  • Jersey (JAX-RS)
  • Metro (JAX-WS)
  • Grizzly (NIO)
  • Atmosphere (Comet)
  • OpenMQ (JMS)
  • and scripting jRoR Grails and now Django (python)

Main difference from Tomcat - Grizzly core (rewritten)

Netbeans 6.8 tooling
Support available in Eclipse too

GlassFish development continues
Support contracts through to 2017+ unlimited
Remains the Java EE reference implementation
Now also sold with WebLogic and standalone

Roadmap -> expected soon for remaining year

Don't have to deliver as much standard runtime / frameworks jars as part of the application jar

Netbeans in-place edit of classes, incremental compilation, deploy on save, GF v3 preserves session across redeployments(!)

Session retention:
Deployment option to maintain statefull sessions across re-deployments!


GlassFish v3 key goals:

Modular and dynamic
Modular: Apache Felix (OSGi)
Extensible: HK2 (100k kernel)
Still very fast!

Centralized configuration, modules configured through centralised control


Key Features:

No ejbjar.xml needed, no web.xml, annotation driven, EJB's as single classes

Declarative annotations:

@Stateless annotation for class stateless bean
@Schedule annotation for timers

Eclipse - GlassFish tool bundle for eclipse (contains everything!)

Ultra fast auto-deploy of all Java EE and static artefacts

Maven support: mvn gf:run gf:start gf-deploy

Containers can be added / removed dynamically


New API for EJB testing (EJBContainer):

Example:

EJBContainer c = EJBContainer.createEJBContainer();
Context ice = c.getContext();
SimpleEjb ejb (SImpleEjb)ic.lookup("java:global/sample/SimpleEjb");
ejb.sayHello();


GlassFish "Embedded" - allows all features of GlassFish to be automated

org.glassfish.api.embedded.Server server;
Server.Build builder = new Server.Builder();
server = builder.build();
ContainerBuilder b = server.createConfig(ContainerBuilder.Type.web);
server.addContainer(b);

File archive = new File("hello.war");
server.getDeployer().deply(archive);

i.e. Ship app server inside an application!


OSGi:

GlassFish runs on top of OSGi (Felix by default)
Also runs unmodified on Knopflerfish and Equinox
GlassFish ships with 200+ bundles
Can run without OSGi (static mode based on HK2)
Can use OSGi management tools (CLI or Web)

Any OSGi bundles will run in GlassFish v3
Drop it in glassfish/modules

Servlets can get hold of OSGi bundles (using @Resource DI)


Update centre:

Graphical tool (GlassFish does not need to be started), available in web admin console
CLI version available
Was in v2.x but not from admin console


RESTful admin API:

JAX-RS/Jersey + Grizzly to provide REST interfaces to
Configure runtime (via GET, POST, DELETE)
Invoke commands (restart, stop, deploy, etc)
Monitoring (GET only)
Log rotation etc

e.g. Available from:
localhost:4848/management/domain
localhost:4848/monitoring/domain


Further advantages:

Dynamic language support via modules:
Rails, Grails, Django, Scala/Lift


Comet:
Cometd/Bayeux
Atmosphere

Full support for:
mod_jk
WebDAV, CGI, SSI

OpenMQ 4.4

Web Services Metro 1.4
.NET 3.5 interoperability


v3 Clustering:
Lower priority after Java EE 6 and modularity, so not yet...
Clustering is not built in as per v2
More similar to v1, single instance
Have to take on own clustering (load balancing, deployment)
See roadmap for details...


Doing More with GlassFish (Steve Elliott):

GlassFish v3 - Management and monitoring

Management:

User Friendly, pluggable and extensible for administration
Feature rich Admin console (GUI)
Easy to use Command Line Interface (CLI)
RESTful management and monitoring API
Fully documented AMX API (app server management API)
All management features built on AMX API

OSGi, load on demand = fast initial start up

v3 AdminConsole:
Frame-set removed, now Ajax based pages
Pluggable console (admin panes, trees etc loaded on demand)


Monitoring:

Lightweight prode architecture
Ad hoc monitoring in Production
Client-scripting (JavaScript)
DTrace integration on Solaris (similar to OS probes, uniform tracing experience with MySQL etc)
Extensibility / Pluggability

No overhead when there is no monitoring
Allows Monitoring to be turned on in a production environment with minimal impact
Generate and listen to only interested
Turn on monitoring when needed
BTrace integration
Portable and dynamic


Instrumentation:

Modules expose probes
POJO with annotations
XML

Modules register probe listeners

In code, POJO annotations can be used

@ProbeProvider(providername="glassfish", modulename="web")

ProbeProvider XML configuration also possible

ProbeListeners
JMX exposed

@ManagedAttribute(id="jspcount")


OSGi:

big move to OSGi technology
Big move to more modular development approach

Demands and enforces stronger modularity

OSGi is largely under the covers
Visible to GlassFish developers, but not to GlassFish users


Service based architecture:

Core modules loaded on app startup
Rest loaded on demand

Module Management:
add, remove, update installed modules

OSGi as a container


Web services - Metro : JAX-WS / Jersey : JAX-RS

Metro - SOAP-based web services stack
Built into GlassFish
Works with any servlet 2.5 compliant web container
WebLogic, WebSphere, JBoss, Tomcat
Also standalone
Advances interoperability with .NET 3.x/4.0

Project Tango - focused on interoperability with .NET

JAXB based XML Data Binding (XSD, XPATH)

SOAP Messaging MTOM etc

Bi-directional interoperability with .NET (Java or .NET as client or server)


Standards:

JCP: JAX-WS 2.2 & JAXB 2.2
W3C SPAP 1.1/1.2 WSDL 1.1, WS-Addressing
… etc


JAX-RS:

JAX-RS 1.1 is final and part of EE6

Not a web profile
but included with GlassFish v3 web profile
JCP 311
Spec - JSR 311

http://www.oracle.com/java
Contains links to GlassFish etc


.