Introduction to Maker, part 1

Introduction to Maker, part 1:

Maker - A simple build system for Scala (and Java) projects

Maker is a fast, yet simple, command line build system for development of Scala (and Java) projects.

Main features, at a glance:

• Fast, incremental, parallel compilation. Maker will utilize as many cores as you have available to parallelise non-sequential module dependencies
• Simple, flexible project configuration and APIs
• Standard interactive REPL based session
• Integrated dependency management (delegated to and handled by Ivy)
• Test Runner integration
• Can transparently run 'offline'
• Supports web-applications, with embedded Jetty runner
• Integrates with Maven and Nexus for standard artifact publishing

So why Maker?

If you work with Scala, SBT is the natural build system everyone tends to gravitate towards. However the S in SBT once stood for Simple. Since the changes from .1x onwards this is perhaps less so. While SBT is powerful, its APIs and dependency management can be opaque and difficult to understand and get right (particularly on large projects).

Maker aims to put the 'Simple' back into a build tool for Scala, whilst still retaining the performance and feature set we've come to expect from a modern build tool.

Whilst still very much a work in progress, we are dog-food-ing this build system on our own project and we're almost at the point where we'll switch out SBT 0.1x for this replacement. Maker should be able to build projects of all sizes, but as with all new projects there may be some feature gaps - these should be closed down over time.

Maker resource quick-links:

GitHub: https://github.com/cage433/maker home and readme: https://github.com/cage433/maker#readme

GitHub, Issues and feature tracking: https://github.com/cage433/maker/issues?direction=desc&sort=created&state=open

GitHub Wiki: https://github.com/cage433/maker/wiki

Jenkins continuous integration: http://www.chillipower.com:8081

Google code: http://code.google.com/p/maker/

By means of an example of bootstrapping maker and using it to build itself, this blog is aimed at the beginner and will hopefully show the basics of getting started. So here goes.

Getting started; getting the code and bootstrapping the build:

Maker is in GitHub, using a Git client, clone the repo:

$ git clone git@github.com:cage433/maker.git maker

$ cd maker

Maker is launched through a single script, this is bin/maker.sh, we can get a list of options using --help:

maker$ bin/maker.sh --help

usage

maker.sh *

options

-h, --help

-p, --project-file

-c, --cmd

run command directly then quit

-j, --use-jrebel (requires JREBEL_HOME to be set)

-m, --mem-heap-space

default is one quarter of available RAM

-y, --do-ivy-update

update will always be done if /.maker/lib doesn't exist

-b, --boostrap

builds maker.jar from scratch

-d, --download-project-scala-lib

downloads scala compiler and library to /.maker/scala-lib

download is automatic if this directory does not exist

-x, --allow-remote-debugging

runs a remote JVM

-i, --developer-mode

For maker developers.

Sets the maker classpath to maker/classes:utils/classes etc rather than

maker.jar. Allows work on maker and another project to be done simultaneously.

-nr, --no-repl

skip repl launch (just performs bootstrapping/building and returns)

-ntty, --no-tty-restore

skip save and restore tty (for integration with automation such as TeamCity reporting)

--args, --additional-args

additional variable length argument list to pass to JVM process directly. Must come at the end of the arguments

--mem-permgen-space

default is 1/10th of heap space

--ivy-proxy-host

--ivy-proxy-port

--ivy-non-proxy-hosts

--ivy-jar

defaults to /usr/share/java/ivy.jar

Now we can bootstrap maker by building a maker jar lib, this is done through the main maker script

maker$ bin/maker.sh -y -b

-y tells maker to fetch dependency libraries required by maker

-b tells maker to bootstrap itself using a brute force build from sources

(note, if you're connecting to the internet via a proxy you might also need to specify –ivy-proxy-host and –ivy-proxy-port options).

This should take a few seconds and then you'll find yourself in the probably familiar Scala repl.

At this stage maker has fetched dependencies, built the maker jar and booted it on the classpath. If you want to just use it to make another project then you could quit and start using it immediately (which I'll show in a subsequent blog of this series)

But while we're here, lets take a quick look around…

Maker has the standard concept of projects which can be viewed as modules of software compilation. These projects can be joined up in a dependency tree – just as you would expect from a build tool. In maker's own project definition, the parent module is called mkr and we can check that in the repl:

scala> mkr

res3: maker.project.Project = Project com.google.code.maker:maker

Here we see that the project is defined as an instance of the class maker.project.Project

Projects are where all the action happens. Now that we've bootstrapped maker, we can ask Maker to actually build itself! 

Note on Ivy: By default, updating dependencies from Ivy automatically is disabled (it can be enabled by default using Maker properties, more on that later). So to update Maker's dependencies (makers own build is separated from the bootstrap classpath) we can use the update command on the project.

scala> mkr.update

:: loading settings :: file = maker-ivysettings.xml

:: loading settings :: file = maker/maker-ivy-dynamic.ivy

...

:: resolving dependencies :: com.google.code.maker#utils;${maker.module.version}

confs: [default]

found log4j#log4j;1.2.16 in central

found commons-io#commons-io;2.1 in central

found com.typesafe.akka#akka-actor;2.0 in akka

found com.typesafe.akka#akka-remote;2.0 in akka

found com.typesafe.akka#akka-kernel;2.0 in akka

found org.scala-tools.testing#scalacheck_2.9.1;1.9 in central

found org.scalatest#scalatest_2.9.1;1.7.1 in central

found org.scalaz#scalaz-core_2.9.1;6.0.4 in central

found org.slf4j#slf4j-api;1.6.1 in central

found org.slf4j#slf4j-log4j12;1.6.1 in central

found org.apache.ant#ant;1.8.2 in central

found io.netty#netty;3.4.2.Final in central

found com.google.protobuf#protobuf-java;2.4.1 in central

found net.debasishg#sjson_2.9.1;0.15 in central

found voldemort.store.compress#h2-lzf;1.0 in akka

found org.eclipse.jetty#jetty-server;7.6.3.v20120416 in central

found org.eclipse.jetty#jetty-webapp;7.6.3.v20120416 in central

found org.eclipse.jetty#jetty-util;7.6.3.v20120416 in central

found org.eclipse.jetty#jetty-servlet;7.6.3.v20120416 in central

found org.eclipse.jetty#jetty-security;7.6.3.v20120416 in central

found org.eclipse.jetty#jetty-http;7.6.3.v20120416 in central

found org.eclipse.jetty#jetty-io;7.6.3.v20120416 in central

found org.eclipse.jetty#jetty-xml;7.6.3.v20120416 in central

found org.eclipse.jetty#jetty-continuation;7.6.3.v20120416 in central

found org.eclipse.jetty#jetty-jsp;7.6.3.v20120416 in central

found org.mortbay.jetty#jsp-2.1-glassfish;2.1.v20100127 in central

found javax.servlet#servlet-api;2.5 in central

found org.apache.tomcat#jsp-api;6.0.20 in central

found org.mockito#mockito-all;1.8.2 in central

:: resolution report :: resolve 1022ms :: artifacts dl 74ms

---------------------------------------------------------------------

| | modules || artifacts |

| conf | number| search|dwnlded|evicted|| number|dwnlded|

---------------------------------------------------------------------

| default | 29 | 0 | 0 | 0 || 54 | 0 |

---------------------------------------------------------------------

:: retrieving :: com.google.code.maker#utils [sync]

confs: [default]

0 artifacts copied, 54 already retrieved (0kB/43ms)

12 May 2012 09:35:28 INFO root - Task[utils:UpdateTask] completed in 1256ms

12 May 2012 09:35:28 INFO root - Completed Task[maker:UpdateTask], took 1(s) 720(ms)

res4: maker.task.BuildResult[AnyRef] = OK

What you should see (assuming you have connectivity to the internet) is a set of resolved dependencies for Maker and most importantly the BuildResult = OK at the end.

Whilst this project has not yet compiled before, if it had, we might first clean it using:

scala> mkr.clean

12 May 2012 09:37:43 INFO root - Task[maker:CleanTask] completed in 131ms

12 May 2012 09:37:43 INFO root - Completed Task[maker:CleanTask], took 214(ms)

res5: maker.task.BuildResult[AnyRef] = OK

Ok, let's now compile Maker using itself, as a test:

scala> mkr.compile

12 May 2012 09:37:46 INFO root - Compiling Project com.google.code.maker:utils

warning: there were 2 unchecked warnings; re-run with -unchecked for details

12 May 2012 09:38:04 INFO root - Task[utils:CompileSourceTask] completed in 18638ms

12 May 2012 09:38:04 INFO root - Compiling Project com.google.code.maker:plugin

12 May 2012 09:38:10 INFO root - Task[plugin:CompileSourceTask] completed in 5319ms

12 May 2012 09:38:10 INFO root - Compiling Project com.google.code.maker:maker

warning: there were 6 unchecked warnings; re-run with -unchecked for details

12 May 2012 09:38:29 INFO root - Task[maker:CompileSourceTask] completed in 18852ms

12 May 2012 09:38:29 INFO root - Completed Task[maker:CompileSourceTask], took 42(s) 813(ms)

res6: maker.task.BuildResult[AnyRef] = OK

All build commands in Maker are functions that return results, for example, from the compile command we get a result that we can use and inspect. We can even get this result into a named variable and use it in other expressions.

Running compile again will also demonstrate the incremental compilation, note the second compile completes almost instantly and there is no further compilation reported because we are all up to date.

scala> val r = mkr.compile

12 May 2012 09:39:52 INFO root - Completed Task[maker:CompileSourceTask], took 113(ms)

r: maker.task.BuildResult[AnyRef] = OK

scala> r.stats.foreach(println)

Task[maker:CompileSourceTask], took 57ms, status OK

Task[plugin:CompileSourceTask], took 25ms, status OK

Task[utils:CompileSourceTask], took 26ms, status OK

Projects have all the normal tasks you'd expect available on them as functions, the main tasks being:

• clean – clean up the compilation state (remove classes etc)
• compile - compile the main source code
• testCompile – compile the test source code
• test – run all tests
• testClass – run a specific test class (or test suite name) by name
• pack – package up the modules into jars (or wars for webapps)
• update – update dependency libraries
• publishLocal – publish artifacts to the local file system
• publish – publish artifacts to a remote repo
• runMain – run a main class
• runJetty – run the module as a web-app (uses embedded Jetty within Maker)

Most of these tasks also have a corresponding [taskname]Only counterpart which runs the task only on the module and not on any descendent modules. By default, all tasks first run on dependency modules as appropriate, as would be expected. Similarly, tasks may depend on each other. For example, project.test will ensure all child modules are compiled and their tests are compiled before starting the tests.

It's also possible to run compilation, all tests, individual tests, and main methods continuously, say for development purposes while you work on code to make a particular test pass.

We can inspect project module dependencies, for example:

scala> mkr.children

res3: List[maker.project.Project] = List(Project com.google.code.maker:plugin)

where the returned value is a list of child projects that this project directly depends on.

We can also see all dependencies including indirect dependencies using:

scala> mkr.allDeps

res4: List[maker.project.Project] = List(Project com.google.code.maker:maker, Project com.google.code.maker:plugin, Project com.google.code.maker:utils)

Here we see that mkr depends on plugin and utils (these are all the project modules in maker).

We can inspect library dependencies, for example:

scala> mkr.classpathDirectoriesAndJars.foreach(println)

maker/classes

maker/java-classes

maker/test-classes

/Users/louis/dev/tools/scala/scala/lib/scala-compiler.jar

/Users/louis/dev/tools/scala/scala/lib/scala-library.jar

maker/resources

plugin/classes

plugin/java-classes

plugin/test-classes

plugin/resources

utils/classes

utils/java-classes

utils/test-classes

libs/ivy-2.2.0-sources.jar

libs/ivy-2.2.0.jar

…

Let's run Maker's tests, to check all is good:

scala> val r = mkr.test

12 May 2012 09:49:58 INFO root - Compiling Project com.google.code.maker:utils

12 May 2012 09:49:59 INFO root - Task[utils:CompileTestsTask] completed in 1760ms

12 May 2012 09:49:59 INFO root - Testing Project com.google.code.maker:utils

Run starting. Expected test count is: 9

MemoizeTests:

FileUtilsTests:

LogTests:

ProcessIDTests:

…

Output from the test runs should be displayed in the standard test output format, at the end overall success or failure is reported.

We can also query for a particular fragment of a library name or find out how a module depends on a particular library:

scala> mkr.findLibs("scalatest")

utils/lib_managed/scalatest_2.9.1-1.7.1-sources.jar

utils/lib_managed/scalatest_2.9.1-1.7.1.jar

This tells us that Maker depends on a library named scalatest, via the utils module's managed library dependencies.

We can find the dependency path from one project module to another using:

scala> mkr.dependsOnPaths(utils)

res15: List[List[maker.project.Project]] = List(List(Project com.google.code.maker:maker, Project com.google.code.maker:plugin, Project com.google.code.maker:utils))

This tells us that the project module mkr depends on the module utils via the module path; mkr → plugin → utils

Maker doesn't have a complex module structure, but in a complex tree of module dependencies these functions can be useful when trying to understand project structures and dependencies, perhaps for refactoring.

One last thing, maker can run commands from the command line without launching the repl. As we're about done for this setting, let's drop out of the repl and try it ( :q or ctrl-c quits the repl session in case you didn't know).

maker$ bin/maker.sh -c "mkr.clean"

No project file defined, searching for a maker Scala file

Omitting ivy update as /Users/louis/dev/projects/opensource/github/maker/.maker/lib exists

Omitting bootstrap as /Users/louis/dev/projects/opensource/github/maker/maker.jar exists

setting cmd as -e mkr.clean

Maker v.1

12 May 2012 10:35:40 INFO root - Completed Task[maker:CleanTask], took 151(ms)

returning exit status: 0

There are many more features in maker that will be explored in subsequent blogs. At this stage we're now at least able to obtain and bootstrap maker, and explore maker's basic features to build and test itself.

Maker can do more, including drawing graphical graphs of build and test results as well as module dependencies – more on this in a future blog.

In the next blog I'll walk you through making a new project from scratch. We'll see how to construct a new project definition and use Maker's features to build, test, and run a simple hello-world web application from the repl...

>>> introduction to maker part 2