Searching Jars with Java Closures

(Ed's note: see this post for a better example of closures. This one is intended for direct contrast with a Groovy script: the resulting Java is awkward.)

The last post has a "Hello World", of sorts, for Neal Gafter's closure proposal.

In it, I mentioned that I have converted a Groovy jar searcher into pure Java, using Neal Gafter's prototype.

Here is the code. It is probably best to understand the previous post before looking into this. I may also be instructive to understand the Groovy code first.

The code is documented but here is a view from space:

• The program takes a directory and a target string as arguments. It searches all jars in the directory for the target string, and reports the number of matches found.
• A use-case is that you are looking for MyMissingClass in a bunch of jar files.
• The program uses 2 closures as private members to the class.
• One closure is myEntryChecker, which accepts a JarEntry and uses the target variable as a 'free lexical binding'. It returns a boolean if the name of the JarEntry contains the target string.
• The other closure is myFileChecker, which accepts a File and uses the above closure as a 'free lexical binding'. Note that this one declares an IOException.
• Note that there is a cool proposal for loop abstractions that is not yet implemented. This will make run() much more elegant. In fact, the invocation of closures seems to be a major motivation for the proposal.
  

(Btw, if you find the example less-than-elegant, blame me, not closures. The intent is not to introduce closures, per se, but instead to illustrate the new Java syntax by leveraging a simple-yet-useful program implemented in Groovy.)

import java.io.*;
import java.util.jar.*;
import java.util.Enumeration;

public class JarSearcher {

private String target;
private String fileName;

// This closure checks a JarEntry's name for a target string
// @param jarEntry
// @return boolean true if found
// @free target the string we're looking for
// ('free' is my way of denoting a free variable)

private { JarEntry => boolean } myEntryChecker = {
JarEntry jarEntry =>
// expression! no return statement or semi-colon
// Thanks to Christian Ullenboom for the simpler version
jarEntry.getName().contains(target)
};

// This closure iterates over the entries in a jar file and
// checks each one for a target string (via another closure)
// @param file
// @return int # of occurences
// @throws IOException
// @free myEntryChecker
// ('free' is my way of denoting a free variable)

private { File => int throws IOException } myFileChecker = {
File file =>
int count = 0;

fileName = file.getName();

if( fileName.indexOf(".jar") != -1 ) {

  JarFile jarFile = new JarFile(file);

  // old-style Enumeration
  // don't blame me or closures!
  for( Enumeration e = jarFile.entries() ;
       e.hasMoreElements() ; ) {
    JarEntry entry = (JarEntry) e.nextElement();
    if( myEntryChecker.invoke(entry) ) {
      count++;
    }
  }    
}

// expression! no return statement
count
};

// This method lists the files in a directory and
// uses a closure to check each file for a target
// string (if it is a jar file)
// @param args[]
// @throws IOException (because the closure declares it)

public void run(String[] args) throws IOException {
String searchDirStr = args[0];
target = args[1];

File searchDir = new File(searchDirStr);

// NOTE: Neal has a proposal that would make this
// iteration much easier
for(File file : searchDir.listFiles() ) {
  int count = myFileChecker.invoke(file);

  if( count != 0 ) {
     System.out.println(
         "found " + count + " match(es) in " + fileName );
  }
}
}

public static void main(String[] args) {
try {
  JarSearcher jarSearcher = new JarSearcher();
  jarSearcher.run(args);
} catch(Exception ex) {
  // TODO
}
}
}

Java Closures: A Quick Look

Neal Gafter is back! And he has brought treats for Halloween by way of a prototype on the closure proposal.

Essentially, it's a binary version of a modified JDK 6, released under the Java Research License. If Java is our home, Neal is building a whole new section, with a deck and swimming pool.

As a first exercise, I have converted my modest Groovy jar searcher into pure closure-Java. While doing the research, I realized:

• Syntax is like construction. You can watch both all day but it's when you get your hands dirty and pick up a tool, that things begin to click.
• Much like generics, closures will bring a lot of new terms and concepts. I don't mean the notion of a closure, per se, but rather the implementation in Java.
• Already there is a lot of discussion: most of it from hardcore syntax surfers who revel in the corner cases and possibilities. (And more power to 'em)

The upshot is: I wonder if there is an opening in the blogtomonous collective for a gentler introduction. Even easier (and quicker) than the Groovy example mentioned above.

Here's a quick look....

The Setup

• The download is available via Neal's original post Be sure to read the JRL
• If you're on Windows, you might want a tool like 7-Zip to open the ztar file
• Define CLOSURES to be the main directory
• Ensure that JDK 6 is in the path. The shell files in CLOSURES/bin are key, as they modify boot classpath to use the new stuff. Though I love Bash on Windows, I used straight Windows BAT files that were morphed from javac and java in ~/bin.
  

The Gist

I'll assume that you're familiar with closures from Groovy, Ruby, Scheme, etc. With respect to the Java syntax, the most important things to understand are the Closure Literal and the Function Type.

In essence, the Closure Literal is the closure body; a Function Type is an abstract data type, a generalization of the venerable T for Generics. The difference is that the Function Type specifies many types for a function: (1) the types of the parameters, (2) the return type, and (3) possible exceptions.

I know: ZZZzzzz. To the ExampleCave, Batman!

// SNIP: public static void main...

String bulletin = " Hello Closure! ";

// Below, think: Type varName = closure;
//
// where Type = Function Type = { String, String => String }
//       varName = myEasyClosure
//       closure = Closure Literal (i.e. _the_ closure)

{ String, String => String } myEasyClosure =
{ String a, String b => a + bulletin + b };

// The closure is built into a class with an invoke method.
// To use it, simply call invoke with appropriate parameters.

String s = "this ";
String t = " is cool";

String result = myEasyClosure.invoke( s, t );

// output:
this  Hello Closure!  is cool 

As annotated above:

• We have a closure called myEasyClosure
• Just as a generic type gives us, well, the type of a generic class, the Function Type states the types of the entire closure. Gafter could ask to us write IT_TAKES_2_STRINGS_AND_RETURNS_A_STRING but mercifully we write { String, String => String }. It is vital to understand that syntax.
• The Closure Literal echoes the syntax of the Function Type: { params => expression }. Note that it doesn't return anything.
  
• The whole point of closures are free lexical bindings: e.g. the free bulletin variable. Though this example is silly, that's where things get cool.
  

Keen readers will note that Function Types also specify exceptions. Let's decide that our closure should check for null on the parameters, and throw an IllegalArgumentException if that contract is violated.

Here's how to express it:

// remember:
// FunctionType varName = closure;

{ String, String => String throws IllegalArgumentException }
myEasyClosure =
{
String a, String b =>

if( a == null || b == null ) {
 throw new IllegalArgumentException
               ("null parameters not allowed");
}

a + bulletin + b // no return or semi-colon !
};

Two points here:

• The Function Type looks complicated, but we simply added a throws IllegalArgumentException into the definition.
• In both examples, note that the final part of the closure body is an expression.

That's it... It should be fairly easy to mock up these examples with the prototype. With that foothold, it is easier to understand the more complex examples and discussions.

Jargon Watch: PragProgged

PragProgged:

To joyfully convey a novel idea with respect to software development, only to discover, via research or conversation, that said thought is merely a variation of a concept already eloquently expressed in that damn book, The Pragmatic Programmer.

E.g. Cool, but my friend, isn't that the same theme as in that book? I think you've been PragProgged.


ps. A pox on your houses, Andrew Hunt and Dave Thomas! That's right: may you suffer plain old XML for the rest of your days!

Thieves in the Temple: Vandals, Dragons and Warp Guns on the JVM

Here are some motivators for this post.

1. Wired recently featured a piece on a new game called Portal. (Be sure to check out the awesome illustration here!) Essentially, players have warp guns that can bend the laws of physics: the guns create entry and exit portals though which the players can jump/dive. I'm not a puzzle gamer but this one looks neat. And, hey, it wowed Valve.

2. At the recent No Fluff Just Stuff conference, Jeff Brown gave a talk on Groovy's Meta Object Protocol. It dawned on me, and perhaps others, that just because things compile to JVM bytecode doesn't mean that all things are equal. For example, if you add a method to a class using Groovy, what does it mean if you access it via Java? By the end of the talk, you could sense everyone struggle with these warp portals on the JVM. We literally had to warp -- er, wrap -- our mind around the multiple dimensions of bytecode.

3. Eric Burke recently posted some Java/reflection behaviour that he found surprising (in the spirit of the excellent book, Java Puzzlers, by Joshua Bloch and Neal Gafter).

All of this adds up to some recent experimentation with Java, Groovy, and the SecurityManager.

These examples use Java 1.5 and Groovy 1.1 RC 1. If you want to follow along, the CLASSPATH contains only the local dir "." and the Groovy jar, groovy-all-1.1-rc-1.jar.

No Warp: Straight Java

To add some fantasy to this article, the JVM is a Temple, which contains a Dragon. We are thieves and seek to capture some of the Dragon's breath, for immeasurable riches and formidable power. However we must elude the dreaded Gatekeepers, who stand vigilantly on guard. Here is a POJO:

public class Dragon {
    private String catchBreath() {
        return "essence of hot magic";
    }
}

Clearly, with straight-up Java, the compiler prevents us from attaining the goal.

As Eric pointed out, we can do some reflection magic:

import java.lang.reflect.*;

public class PureJavaQuest {
    public static void main(String[] args) throws Exception {
      Method catchBreath =
      Dragon.class.getDeclaredMethod("catchBreath");
      catchBreath.setAccessible(true);
      Dragon dragon = new Dragon();
      System.out.println( catchBreath.invoke(dragon) );
    }
}

This works fine. But if the Gatekeepers invoke the Spell of Java Security, we're stymied:

java -Djava.security.manager PureJavaQuest

Exception in thread "main" java.security.AccessControlException:
access denied (
java.lang.reflect.ReflectPermission suppressAccessChecks)
[ stack trace snipped ]

As an aside, the following security policy would allow us to succeed:

grant {
    permission java.lang.reflect.ReflectPermission
                  "suppressAccessChecks";
};

Used like so (if the above were in my.security.policy):

java -Djava.security.manager
     -Djava.security.policy=my.security.policy
     PureJavaQuest

Warp One: Using Java with Groovy

Let's write a Groovy script to use the Dragon class directly:

println "Are gatekeepers using spell? "
    + System.getProperty("java.security.manager")

Dragon dragon = new Dragon()
println dragon.catchBreath()

Here's the run:

$ groovy EasyQuest.groovy
Are gatekeepers using spell? null
essence of hot magic

It works! I believe this is due to a bug in Groovy with respect to private access. (I'm ok with this because I suspect that it is a well-known issue that will be fixed soon). (Is this really a problem in Groovy 1.1 ? Am I missing something?)

Can we exploit this somehow? What happens if the Gatekeepers use the Spell of Java Security:

$ groovy.bat -Djava.security.manager EasyQuest.groovy
Are gatekeepers using spell? true
essence of hot magic

Hmm. The script implies that security is on, but it didn't seem to work. Though Groovy documents security here, perhaps I'm missing something on the setup. Groovy does sophisticated class loading and on-the-fly mojo. I suspect that it might be using a privileged block, and reaching that because of a mistake of mine. (If you know what I've done wrong, please comment or email me at codetojoy @t gmail)

Warp Two: Groovy to Java

Can we exploit this behaviour using Java? Here's the Groovy class that is effectively the same as the above script:

public class Quest {
    public String engage() {
       Dragon dragon = new Dragon()
       return dragon.catchBreath()   
    }

    public static void main(String[] args) {
       Quest quest = new Quest()
       println quest.engage()   
    }
}

Two steps: we use groovyc to compile Quest to a class file. Then, we'll warp and run it with java (with classpath as described).

groovyc Quest.groovy
java Quest
essence of hot magic

It works... Not really surprising, given the previous behaviour. Let's try it with the Gatekeepers' spell:

java -Djava.security.manager Quest
Exception in thread "main"
java.security.AccessControlException: access denied (
java.lang.RuntimePermission accessDeclaredMembers)
[ stacktrace snipped ]

Blammo. Busted by the Gatekeepers. Do you feel the warp here? A Groovy class is compiled to bytecode and run via the java command. Clearly Groovy's magic ultimately boils down to reflection, and Java Security cuts to the chase by simply shutting down the call, no matter from whence it originated.

Hyper Warp: Adding Methods

One more tactic... A standard trick for theft is to cause a commotion. Distract, then act.

With blatant abuse of the ExpandoMetaClass, we can do just that.

The ExpandoMetaClass allows us to add (or override) class methods in Groovy. Prepare to warp. The Temple is gonna get crazy.

If you look at the main() method below, it seems like an innocuous print statement. However, the static block is replacing the toString method for the String class. This commotion throws a NPE with the truly weird comment that the JVM has crashed.

public class Commotion {
    static {
        ExpandoMetaClass.enableGlobally()
        String.metaClass.toString = {
             ->
             String msg = "JVM crash. Contact sys admin"
             throw new NullPointerException(msg);
        }
    }

    static public void main(String[] args) {
        String s = new String("All is well in the Temple");
        println s.toString();
    }
}

Game on. Bring it, Gatekeepers!

Here is the output, when run as Groovy:

groovy Commotion
Caught: java.lang.NullPointerException: JVM crash. Contact sys admin
    at Commotion$__clinit__closure1.doCall(Commotion.groovy:6)
    at Commotion.main(Commotion.groovy:12)

As it turns out, the behaviour is the same as Warp Two: when compiled to bytecode, Java will throw the same nefarious error when run without the Security Manager (aka AccessContoller). With Java Security on, Java catches the reflection attempt.

Conclusions

Did we ultimately steal the Dragon's breath? To be honest, I don't think so: certainly not with the Java Security turned on. In pure Groovy, there are either some bugs or perhaps my configuration is not quite right (I'll report as soon as I find out what is going on.)

Also, remember that we had a Groovy jar (the big, embeddable jar) in the classpath. So things aren't quite as 'fluid' as these output snippets may imply.

What we have done is seen the power of Java Security and, hopefully, bent some holes in your perception of the JVM.

Finally, all experimentation aside, we should strive to be Gatekeepers.

But beware: there are thieves in the Temple. And they have warp guns.

Grok Like an Egyptian: The Weirdest Language

These days, it seems that the older languages are new again.

Lisp, Smalltalk, and Erlang are getting a lot attention (yes, they are all much different in age).

I have wondered if another language would be brought out of the attic, and sure enough, it is getting some attention.

I'm talking about APL which is easily the most strange, wonderfully weird language I have ever used (though it has been years).

Consider this:

• It required a special keyboard, and uses characters that are not ASCII. It uses a highly symbolic, mathematical syntax that has been compared to hieroglyphics.
  
• From what I (vaguely) remember, no explicit loops or conditionals.
• It is common to combine functions using higher-order operators in manner that is so terse, it makes Perl look like English. Think I'm kidding? How about a one-liner that computes Conway's Game of Life.
  
• Kenneth Iverson won a Turing award for his contributions (of which APL was a major part).

At the risk of alienating the small, rabid crowd of APLers, I should point out that APL does not run on the JVM (to my knowledge), does not form the basis for a popular web framework, and may not introduce Java people to new constructs such as closures and DSLs.

But if you are looking for a new way to think about problems, this is a fascinating option.

Which leads me to a question:

Is there a good, free APL development environment for Windows?

I've been looking for one for some weeks now, but haven't really found anything. Help!

Time permitting, I'd be interested in getting back into APL and possibly blogging a bit (if I can render the character set on here).

Happy 9th, Bug 4152790

A colleague pointed out this bug, as filed against the JDK.

A Happy (belated) 9th to Bug 4152790! (Check out the comments)

Delightful...

This is not a campaign to fix this bug, btw. Just an appreciation for benevolent wit. Though one can imagine that this one might be frustrating. I wonder how much, if at all, the "danger threshold" for the size of the graph/stack has changed across JDKs. e.g. Java 1.3 versus Java 6

Exposition as Genius

Recently, a colleague stated an eloquent view of genius:

An accomplishment is not truly ingenious if the person cannot explain it.

I love this quote, vis-a-vis software:

• If one considers code as an explanation of what the computer is doing, then this quote has inherent disrespect for clever hacks and tricks.
• More generally, a good, thoughtful explanation is quite an accomplishment, and a gift: it requires articulation, respect, and an intuitive feel for the audience. Over my career, I have learned that a critical attribute of a work environment is: do I feel comfortable asking questions? This spans technology and domain knowledge.
  
• This quote says nothing about the creative spark or _how_ the person came up with the idea. That's a separate issue, and a tall order. I doubt that Mozart could tell anyone how he wrote; i.e. how his genius worked. But he could tell you why a particular piece of music works.
  

Great stuff.

Pathological Development (The World According to Garthe)

Alex Miller has an interesting post on the concept of the Garthe JVM. Named after the evil twin of a TV show protagonist, the Garthe JVM would support the Java Memory Model but be otherwise evil (e.g. nefarious thread scheduling and bytecode re-ordering).

An entire JVM isn't realistic for me (those Terracotta guys aim high), but the notion of injecting some evil into your project, for testing, can be really illuminating, and even fun. (Note: this is app testing, not testing the memory model semantics.)



The Original Pathogen

Many moons ago, I worked on a web project that used a home-grown HTML templating framework. This was pre-JSP, but they used macros which were a precursor to JSP taglibs. The dev team had one pathological macro that would explicitly throw an exception. Needless to say, it was undocumented.

However, it was very useful. The app was in C++ so memory leaks and exception safety were key concerns. When testing for these factors, one shouldn't wait for evil to happen: one has to be evil. One has to be Garthe.

Garthe Injection

These days, with IoC containers and mock testing objects, there are variety of tools that one can use to inject some pathological behaviour into an app.

How about this world, according to Garthe:

• an implementation of a ORM layer where the DB goes down every 5 minutes
• network latency is most definitely not zero in Garthe's world: it can be infinite, and there are network-based objects to prove it
• an exception of some type T really are thrown from strange corners of the design
  
• a homegrown (undocumented!) Swing/UI component creates its own thread and attempts to process UI events on that thread, and not on the EDT

Madness, you say! What evil behaviour!

Precisely: how does your app handle it? If you asked the team, just as a thought experiment, would they know? If Person A and Person B have different speculations on the outcome, what does that mean?

This pathological development is a great conversation starter, and in terms of role play, it's fun. Just be sure to mark the code very clearly or, better yet, have a "skull and crossbones" jar/project that contains the source.