Two new Eclipse projects: Xpect and Xsemantics

Yesterday, two new Eclipse projects had been created:

• Eclipse Xpect
• Eclipse Xsemantics

Both project are around for quite some time and they both are based on Eclipse Xtext, the famous framework to create editors with all state-of-the-art features (parser, linker, validators, content assist, etc.) for your own textual DSLs, simply based on a grammar.

Xpect is written by Moritz Eysholdt, who is also a committer of Xtext. It is a unit- and integration-testing framework to be used for Xtext-based languages. Instead of writing fiddly JUnit tests, you can simply write things like

// XPECT errors --> "cannot divide two strings"
"hello" / "world"

assuming your language supports division, strings, and types.

And types is the topic of Xsemantics, written by Lorenzo Bettini, author of the book "Implementing DSLs with Xtext and Xtend". It is a DSL (implemented in Xtext itself) for writing type systems, reduction rules, interpreters and general relation rules for languages implemented in Xtext. So you can write rules like that:

rule subtypeUnion_Left
 G|- UnionTypeExpression U <: TypeRef S
from {
 U.typeRefs.forall[T| G |- T <: S]
}

Well, you probably need to know type theory a little bit to see the beauty in that.

Both of these frameworks are heavily used by Eclipse N4JS: it contains more than 10.000 Xpect tests (e.g., all specification tests) and a Java 8 like type system defined in Xsemantics, and this is also why I act as kind of "assisting project lead" to help the original authors to bring these great tools to Eclipse. The authors and the N4JS team at enfore are now working on bringing the code to Eclipse as soon as possible, to simplify the usage of these frameworks for all Xtext users!

Keep on modelling!

I once said that modelling without graphical editors is like Tour de France without mountains. Well, I have to correct myself: a great test suite and a complicated type system are just as exciting ;-)

When your MWE2 workflow is not working...

MWE2 is a kind of Ant for model related tasks. Xtext is using MWE2, and I also use it to run my own Xpand generator templates. It's a nice tiny tool, and one of the nice things is that it runs in its own JVM, so you can easily extend MWE2 with new components which resides in your project. When MWE2 is started, the project settings, i.e. the classpath (including all information from the plugin dependencies), are passed to the workflow. Unfortunately, this may produce problems which are really hard to find, mostly because the error message do not really tell you where to find the actual problem. Here is a list of some problems I have run into several times (using Eclipse 3.6 and Xtext/Xpand 1.0.1). Note that all the problems mentioned and fixed below may be caused by other reasons, requiring other fixes.

Problems instantiating module

Error message:

Error message in console:

1    [main] ERROR mf.mwe2.launch.runtime.Mwe2Launcher  - Problems instantiating module ...
...
Caused by: org.eclipse.emf.mwe.core.ConfigurationException: The platformUri location '......' does not exist 

Possible fix:

Fix projectName in MWE2 file.

I run into this problem after renaming a project. Ensure the project name defined in your MWE2 file

var projectName = ".."

matches the actual project name. This line is present in Xtext related MWE2 files.

Couldn't find module with name

Error message:

Error message in console:

ERROR mf.mwe2.launch.runtime.Mwe2Launcher  - Couldn't find module with name ...

Possible fix:

Create missing src-gen folder.

I run into this problem after checking out a project from a code repository. Since we do not add generated code to the repository, the src-gen folder was not added to the repository. Hence, it was not present after having checked out the project. However, it is configured in the build.properties. This seems to lead to a problem when the classpath is computed, so that the src folder is not added to the classpath either. Since the MWE2 file resides in the src folder, is is not found and, consequently, the module is not found. I was able to fix this problem by simply creating the src-gen folder. In order to not cause this problem again, I have added the src-gen folder to the repository and ignore only its content.

Workflow definition is ignored

Error message:

None. However, the selected workflow is completely ignored. It seems as if another workflow is executed.

Possible fix:

Ensure module name of workflow, that is the first line in MWE2 file

module ..

is unique.

This seems to be a typical copy- and paste error. Actually, in deed another workflow is executed. Although one can run an MWE2 file via "Run as../MWE2 workflow", the launcher does not directly call the actual workflow file. Instead, the name of the module is read and then the internal representation of this module is been executed. If you define two of more workflows with the same module name, only one of these modules is actually present (there seems to be some kind of map from module name to module).

Couldn't resolve reference to JvmType 'Workflow'.

Error message:

Error in MWE2 workflow file:

Couldn't resolve reference to JvmType 'Workflow'.

When you try to run the workflow, the following message appears:

Please put bundle 'org.eclipse.mwe2.launch' on your project's classpath.

Possible fix:

Ensure Plug-in Dependencies are correctly added to classpath.

This error can be caused by at least one of the following problems:

• your project is not an OSGi/Plug-in project. This can be fixed by converting the (Java) project to a Plug-in project.
• as printed in the dialog, ensure 'org.eclipse.mwe2.launch' to be listed in the plug-in dependencies

While these are obvious reasons for the error, in my case I had configured the project as plug-in project and 'org.eclipse.mwe2.launch' was defined in the dependency list. However, it still didn't worked. This may be caused because I renamed the project, and probably something had gone wrong. I noticed the problem only because the Package Explorer didn't showed the entry "Plug-In Dependencies". I assume there may be some corrupt cache entries. I only was able to fix this problem by creating a new plug-in project and moving the content of the broken project into the new one.

Weird errors when generating the parser etc.

Error message:

When running the Xtext MWE2 workflow to generate the code from your grammar, weird errors occur indicating problems in your grammar.

Possible fix:

Increase memory in MWE2 runtime configuration.

I stumbled over this problem twice, and it is really annoying since it is very hard to find the reason. If the parser generator runs out of memory, it crashes at some arbitrary position and randomly creates error messages. Since it usually happens in the parser generator, the error messages always indicate problems in your grammar, although your grammar may be perfectly ok. So, if you got any grammar problems which are not that obvious real grammar problems, ensure to set VM argument "-Xmx1024m" in the MWE2 runtime configuration.

Disclaimer: This is more a personal note, and some problems may be fixed in the meantime. Feel free to tell me if I got something wrong here :-)

Implement toString with Xtext's Serializer

Xtext uses EMF to generate the model API of the abstract syntax tree (or graph) of a DSL. For all implementation classes, the toString() method is generated. For a simple model element, this default implementation returns a string looking similar to this:

my.dsl.impl.SomeElement@67cee792 (attr1: SomeValue)

Well, this is not too bad. However this looks completely different to my DSL syntax, which may look like this:

SomeValue { 
    The content;
}

Especially for debugging and logging, I prefer that DSL like output. Since Xtext does not only generates a parser for reading such a text, but also a seralizer for creating the text from a model, I was wondering if that mechanism could be used for the toString method as well. (Actually, Henrik Lindberg pointed out the serializer class -- thank you, Henrik!)

In the following, I describe how to do that. Actually, this is a little bit tricky, and it will cover several aspects of Xtext and the generation process:

• use the generated serializer for formatting a single model element
• tweak the generation process in order to add a new method
• define the body of the newly added method

We will do that by adding a post processor Xtend file, which adds a new operation to the DSL model elements. The body of the operation is then added using ecore annotations. But first, we will write a static helper class implementing the toString method using the serializer.

Use the serializer

Xtext provides a serializer class, which is usually used for writing a model to an Xtext resource. The Serializer class (in org.eclipse.xtext.parsetree.reconstr) provides a method serialize(EObject obj), which returns a String---this is exactly what we need. This class requires a parse tree constructor, a formatter and a concrete syntax validator. Thanks to google Guice, we do not have to bother about these things. Xtext generates everything required to create a nicley configured serializer for us. What we need is the Guice injector for creating the serializer:

Injector injector = Guice.createInjector(new  my.dsl.MyDslRuntimeModule());
Serializer serializer = injector.getInstance(Serializer.class);

Now we could simply call the serialize method for a model element (which is to be an element of the DSL):

String s = serializer.serialize(eobj);

Since this may throws an exception (when the eobj cannot be successfully serialized, e.g., due to missing values), we encapsulate this call in a try-catch block. Also, we create a helper class, providing a static method. We also use a static instance of the serializer.
Since this helper class is only to be used by the toString methods in our generated implementation, we put it into the same package.

package my.dsl.impl;

import org.eclipse.emf.ecore.EObject;
import org.eclipse.xtext.parsetree.reconstr.Serializer;
import com.google.inject.Guice;

public class ToString {
 private static Serializer SERIALIZER = null;

 private static Serializer getSerializer() {
  if (SERIALIZER == null) { // lazy creation
   SERIALIZER = Guice.createInjector(new my.dsl.MyDslRuntimeModule())
        .getInstance(Serializer.class);
  }
  return SERIALIZER;
 }

 public static String valueOf(EObject eobj) {
  if (eobj==null) {
   return "null";
  }
  try {
   return getSerializer().serialize(eobj);
  } catch (Exception ex) { // fall back:
   return eobj.getClass().getSimpleName()+'@'+eobj.hashCode();
  }
 }

}

Post processing

Now we have to implement the toString() method of our model classes accordingly. That is, instead of the default EMF toString method, we want to call our static helper method for producing the String.

A generic solution, which can not only be applied for adding the toString method but for all kind of operations, is to use a post processor extension (written in Xtend) to add new operations to the generated ecore model. The overall mechanism is described in the Xtext documentation. We have to write an Xtend extension matching a specific naming convention: <name of DSL>PostProcessor.ext. In our exampel that would be MyDslPostProcessor.

The easy thing is to add a new operation to each classifier:

import ecore;
import xtext;

process(GeneratedMetamodel this) :
 this.ePackage.eClassifiers.addToStringOperation();

create EOperation addToStringOperation(EClassifier c):
    ... define operation ... ->
 ((EClass)c).eOperations.add(this);

For defining the operation, we need:

• the return type of the operation
• the body of the operation

The return type is an EString (which will result in a simple Java String). In EMF, we have to set the type via EOperation.setEType(EClassifier). That is, we need the classifier of EString. With Java, this would be no problem: EcorePackage.eINSTANCE.getEString().
Unfortunately, we cannot directly access static fields from Xtend. At least, I do not know how that works. Fortunately, we can substitute EcorePackage.eINSTANCE with calling a static method of EcorePackageImpl. This static method can then be defined as a JAVA extension in Xtend:

EPackage ecorePackage(): 
 JAVA org.eclipse.emf.ecore.impl.EcorePackageImpl.init();

Note that we return an EPackage instead of the EcorePackage. I assume this is necesssary because we use the EMF metamodel contributor and EcorePackage is not available then. We can now set the EString classifier as return type of the operation: setEType(ecorePackage().getEClassifier("EString"))

Now, we need the body of the operation. Ecore does not directly support the definition of a body, that is there is no field in EOperation for setting the body. Fortunately, we can exploit annotations for defining the body. The default EMF generator templates look for annotations marked with the source value "http://www.eclipse.org/emf/2002/GenModel". The key of the annotation must be "body", and the value of the annotation is then used as the body of the operation. In the body, we simply call our static helper method for producing the DSL-like string representation.

The complete post processor extensions looks as follows:

import ecore;
import xtext;

process(GeneratedMetamodel this) :
 this.ePackage.eClassifiers.addToStringOperation();

EPackage ecorePackage(): 
 JAVA org.eclipse.emf.ecore.impl.EcorePackageImpl.init();


create EOperation addToStringOperation(EClassifier c):
 setName("toString") ->
 setEType(ecorePackage().getEClassifier("EString")) ->
 eAnnotations.add(addBodyAnnotation(
  'if (eIsProxy()) return super.toString(); return ToString.valueOf(this);')) ->
 ((EClass)c).eOperations.add(this);

create EAnnotation addBodyAnnotation(EOperation op, String strBody):
 setSource("http://www.eclipse.org/emf/2002/GenModel") ->
 createBody(strBody) ->
 op.eAnnotations.add(this);
 
create EStringToStringMapEntry createBody(EAnnotation annotation, String strBody): 
 setKey("body")->
 setValue(strBody) ->
 annotation.details.add(this);

If you (re-) run the GenerateMyDSL workflow, the EMF toString() implementations are replaced by our new version. You can test it in a simple stand alone application (do not forget to call doSetup in order to configure the injector):

public static void main(String[] args) {
 MyDslStandaloneSetup.doSetup();
 MyElement = MyDslFactory.eINSTANCE.createElement();
 e.setAttr1("Test");
 e.setAttr2("Type");
 System.out.println(e.toString());
}

Closing remarks

You probably do not want to really replace all toString methods with the serializer output, as this would create rather long output in case of container elements. In that case, you can add the new operation only to selected classifiers, or use the (generated) Switch-class to further customize the output.

Although the solutions looks straight forward, it took me some time to solve some hidden problems and get around others:

1. How to create the serializer using the injector -- and how to create the injector in the first place
2. How to access a static Java method from Xtend without too much overhead. Would be great if static fields could be accessed from Xtend directly.
3. How to use the post processor with the JavaBeans metamodel contributor. If I switch to the JavaBeans metamodel, my extension didn't get called anymore.
4. I'm still wondering where "EStringToStringMapEntry" is defined. I "copied" that piece of code from a snippet I wrote a couple of months ago, and I have forgotten how I found that solution in the first place.
5. Sorry, but I have to say it: The Xtend version 1.0.1 editor is crap (e.g., error markers of solved problems do not always get removed). But I've heard there should be a better one available in version 2 ;-)

Extract Xtext Project Wizard

Besides a nice parser and a powerful editor, Xtext can also generate a project wizard and a generator plugin. In this little posting I will explain how to extract the project wizard related code from the generated UI plugin -- and explain why you would want to to that in the first place. First of all, what does the project wizard? The project wizard is useful to set up an initial project for your DSL, e.g., for creating initial templates, workflow files and to configure the project and its dependencies. The nice thing about the Xtext generated wizard is, that the template files are simply created using Xpand. It is very easy to provide custom templates and other stuff by simply modifying the Xpand file. You will find that Xpand template in the UI plugin, if you have enabled the wizard fragment in your DSL generator workflow:

// project wizard (optional)
fragment = projectWizard.SimpleProjectWizardFragment {
 generatorProjectName = "${projectName}.generator"
 modelFileExtension = file.extensions
}

This code is uncommented by default. So, Xtext generates a nice project wizard. This project wizard is found in the generated ui project. Let's say, my language is called sample.mydsl, then the wizard is created in sample.mydsl.ui, together with the powerful editor. Now, why would I want to extract this editor from the ui plugin. Actually, because I want to use the wizard during development of my language. Sounds silly... OK, here is the longer explanation: I find the wizard extremely useful especially if I have a project which uses code generation. Although you can use any kind of code generation technologies, you probably will often use Xpand and Xtend, as it is nicely integrated with Xtext. That is, Xtext can also generate a generator plugin with some template files and most notably a workflow file (MWE2). Actually, the generated generator plugin looks almost similar to a plugin project created by the generated project wizard. That is, both, the generator plugin and a project wizard created DSL project, contain a folder src/model with a sample model, and also a sample MWE2 generator file. Also, both projects contain a src-gen folder, and the plugin-dependencies are set accordingly. This is no coincidence: If you use code generation for your DSL, and if you use Xpand to perform the generation, using an MWE2 workflow to trigger the generation is a very easy solution. Alternatively, you will have to write a new action or something, but the MWE2 workflow is much easier to set up. The model provided in the generator plugin is usually only used for testing purposes. That is, you write your Xpand (and Xtend) templates in the generator plugin and you can quickly test them by applying them on the models provided in the generator model folder. Later, when the generator plugin is installed, the user of the generator won't see the Xpand templates in the workspace, however they can be accessed by the MWE2 workflow of the DSL project. During development, you probably run into the same situation as I did: You write the templates, then some things are changed in your DSL and you have do adjust the templates, or new features should be implemented, or you have forgotten some weird constellations, or you haven't written templates for all model elements. Most important: You probably will have several different models and you do not want to get the generated code to be generated in your generator plugin (as it should only define the templates, and should not contain some weird generated Java or whatever files). In my case, other guys on the project write the DSLs and I have to maintain the templates. I rarely use the generated DSL editor myself, but I often have to use the generator (and adjust the templates). Since I needed an extra project for each different case, I found myself copying the generator plugin (without the templates, but with my nicely configured MWE2 workflow) over and over again. I always had to adjust the plugin dependencies and so on. Well, a wizard would be nice in that situation. Now, you see my point? The Xtext generated project wizard is exactly what I needed -- but I need it in an Eclipse instance in which I do not have my DSL editor (and neither the DSL parser) installed, as this is the very instance I develop these things. But the wizard would come quite comfortably. So, the idea is as follows: I extracted the wizard into a separate plugin. The wizard plugin has no dependencies to my DSL, so it can be installed without my DSL plugins installed. However, the generator, i.e. the MWE2 workflow, requires all my DSL stuff. But this is no problem, as the generator plugin is an opened project in my development workspace -- thus the MWE2 workflow (not the plugin code, but the workflow) can access the project. Here is how to extract the project wizard (which is not too complicated, however it may save you some minutes):

assumption

You have an existing Xtext project, I will call it "sample.mydsl" in the following, and, of course, a generator plugin "sample.mydsl.generator" created for your DSL. Inside the generator, you have configure the MWE2 workflow.

generate project wizard

Enable Project Wizard Fragment in your DSL workflow, e.g. in "GenerateMyDSL.mwe2" of your DSL project:

// project wizard (optional) 
fragment = projectWizard.SimpleProjectWizardFragment {
 generatorProjectName = "${projectName}.generator" 
 modelFileExtension = file.extensions
}

Now run the workflow "GenerateMyDSL.mwe2", and disable the project wizard fragment (as we do not want to have two wizards in the end).

create wizard plugin

Create new plugin project (e.g., sample.mydsl.ui.wizard) with and Activator, check "This plugin-in will make contributions to the UI".
Important: Use "sample.ui.wizard" as package for the Activator (without mydsl), in order to retrieve the same package names as in the Xtext generated ui project.

move wizard code to new plugin

Move the following classes from the generated ui plugin into the wizard -- this is the actual "extraction" of the wizard:
from src-gen:

• sample.ui.wizard.MyDSLNewProjectWizard
• sample.ui.wizard.MyDSLProjectCreator

from src:

• sample.ui.wizard.MyDSLProjectInfo
• sample.ui.wizard.MyDSLNewProject.xpt

and copy the following classes from the generated ui plugin into the wizard plugin, put them into the wizard package:

• sample.ui.MyDSLUiModule

from src-gen:

• sample.ui.MyDSLExecutableExtensionFactory

Also copy the plugin from the ui plugin into the wizard plugin and remove everything except the last extension with point "org.eclipse.ui.newWizards", adjust class name of extension factory according to the class in your wizard project (you will have to add a ".wizard" to the fully qualified name).

configure wizard project

In Manifest, set singleton directive to true (if not already set) and add missing plug dependencies, e.g.

  
 org.eclipse.xtext.ui,
 org.eclipse.ui.editors;bundle-version="3.5.0",
 org.eclipse.ui.ide;bundle-version="3.5.0",
 org.eclipse.xtext.ui.shared,
 org.eclipse.ui,
 org.eclipse.xtext.builder,
 org.antlr.runtime,
 org.eclipse.core.runtime,
 org.eclipse.core.resources,
 org.eclipse.xtend,
 org.eclipse.xpand

Depending on your project, you may have to add other dependencies as well, e.g. de.itemis.xtext.typesystem if you use the great Xtext type system by Markus Völter.

adjust the copied and moved java files

• MyDSLUiModule is to be replaced completely:

   
   public class MyDSLUiModule extends AbstractGenericModule {
   
   private AbstractUIPlugin plugin;
  
  
   public MyDSLUiModule(AbstractUIPlugin plugin) {
    this.plugin = plugin;
   }
   
   public void configureLanguageName(Binder binder) {
    binder.bind(String.class).annotatedWith(Names.named(Constants.LANGUAGE_NAME)).toInstance("sample.MyDSL");
   }
   
   public void configureFileExtensions(Binder binder) {
    binder.bind(String.class).annotatedWith(Names.named(Constants.FILE_EXTENSIONS)).toInstance("MyDSL");
   }
   
   @Override
   public void configure(Binder binder) {
    super.configure(binder);
    binder.bind(AbstractUIPlugin.class).toInstance(plugin);
    binder.bind(IDialogSettings.class).toInstance(plugin.getDialogSettings());
   }
   
   
   // contributed by org.eclipse.xtext.ui.generator.projectWizard.SimpleProjectWizardFragment
   public Class<? extends org.eclipse.xtext.ui.wizard.IProjectCreator> bindIProjectCreator() {
    return MyDSLProjectCreator.class;
   }
  }
  

• MyDSLProjectCreator can be reused, however you may want to add some dependencies to the getRequiredBundles method, depending on the dependencies of your generated classes:

    
  @Override
  protected List<String> getRequiredBundles() {
   List<String> result = Lists.newArrayList(super.getRequiredBundles());
   result.add(DSL_GENERATOR_PROJECT_NAME);
   result.add("org.eclipse.jface.text");
   result.add("org.eclipse.jdt.core");
   result.add("org.eclipse.equinox.common");
   result.add("org.eclipse.core.runtime");
   return result;
  }    
  

  Actually, this list is the list of required bundles of your generated code. That is, if you generate Java code which requires a plugin "my.super.plugin", you have to add the dependency here.
• MyDSLNewProjectWizard: you probably have to fix a problem in getProjectInfo, simply change the fully qualified name MyDSLProjectInfo to a simple name, as we have moved the info into the same package.
• MyDSLExecutableExtensionFactor: fix class name of plugin activator, change getInstance().getInjector("..") to getDefault().getInjector()
• Activator: Add initialization of Guice injector and add an injector attribute to the wizard's activator:

    
  public class Activator extends AbstractUIPlugin {
      ..
      Injector injector;
  
   public Injector getInjector() {
    return injector;
   }
  
   @Override
   public void start(BundleContext context) throws Exception {
    super.start(context);
    plugin = this;
  
    injector = Guice.createInjector(
    // Wizard:
     Modules.override(new MyDSLUiModule(this))
     // Workspace etc.:
      .with(new org.eclipse.xtext.ui.shared.SharedStateModule()));
  
   }
   ..
  }
  

remove obsolte code from ui plugin

• remove method bindIProjectCreator in AbstractMyDSLUiModule
• remove the wizard extension point definition from the ui plugin.xml, as you probably do not want to have two wizards.

You can now run the wizard in the Eclipse runtime (i.e. the Eclipse started from within your initial, vanilla, installation). Of course, you can modify the Xpand template, e.g. I have simply copied and pasted the workflow of my generator plugin into that Xpand file. You may also add other adjustments as well. Now, you can export the wizard as a plugin and install it to the dropins folder of your Eclipse installation. After restarting that instance, the wizard is available in the workspace in which you develop your DSL. As it has no dependencies to the DSL, you can create new projects, which are configure as you specified above. The workflow is working, although it probably has dependencies to your DSL (e.g., it uses the generated DSL parser), at the DSL project is an opened project in your workspace. Side effect: Actually, you have extracted the Guice infrastructure as it is used by Xtext generated editors. So, even if you do not use your wizard as often as expected, you may have learned some stuff about this better-then-factory-pattern technology.

Quick'n Dirty Tutorial on Modelling with Eclipse

A friend of mine asked me how to modelling with Eclipse. I gave him a quick tour, demonstrating how to create a model with EMF, how to create an instance of that model, how to query the model with OCL, and how to create a text editor for that model with Xtext. It's pure modelling, i.e. no Java programming at all! I thought that maybe other people would be interested in that topic as well, so I created this little tutorial showing how all these Eclipse modelling "ingredients" can work together.

Preparations

Install Eclipse 3.5 (Galileo) with all the modelling tools, that is the Galileo Eclipse Modeling Tools package. Additionally, we will need the OCL interpreter console, which can be installed using the MDT update site:
http://download.eclipse.org/modeling/mdt/updates/releases/
The OCL console actually is an example, so you will have to install that specific example.

The Runtime Example

Let's assume a small IT company. The employees of that company develop software for other companies. Of course, they use models for that purpose! In order to better organize their work, they want to document who is working on what. That is, they want to assign certain task to developers. A task could be the implementation of a use case or writing a test case for a class. Let's see if we can use models for that...

Company Model with EMF

First of all, we have to create a model describing the company. Let's start from scratch:

1. create a new project:
   File / New / Project... / Eclipse Modeling Framework /Empty EMF project
2. create a new ecore diagram (we want to graphically design the model) in the newly created model folder: Right click on model folder and select
   New / Other... / Ecore Tools / Ecore Diagram
3. enter the Domain file name:: company.ecore and press Finish.

Now, we can "draw" our model using the graphical editor. Create four classes (EClass), add a name attribute (EAttribute) to one of them, create references (EReference) between the classes, and set the properties (in the properties view, this view can be activated by right-clicking in the diagram and select Show Properties View. The result should look like Figure 1.

Figure 1: The domain model of a company

Some remarks:

• The model is an instance of the ecore model, which is quite similar to class models in UML. All elements in ecore start with an "E", so it is EClass, EAttribute, or EString.
• The EType of the attribute name is EString.
• The upper bound of all references is "*" (you can enter "-1", which actually is the same as "*").
• You will need a class containing all your elements later. In the example, the Company serves as a container. Make sure the Is Containment flag is set for the two references employees and skills (see Fig. 1)

Now that we have created a model of our company, we can create an instance of that model. Later, we will generate an editor, but right now we want to quickly create an instance to get a feeling for our model. An instance of our model actually is an instance of our container element, that is the Company class. There is a generic editor available, which can create an instance of a model (or its elements) directly without the need of code generation. All you need is the ecore model. This is how to activate it:

4. Close the ecore diagram
5. Open the ecore model, this time use the Sample Ecore Model Editor. Usually, this editor is used if you double-click the ecore-file, but to be sure use the Open With... context menu entry.
6. Select the Company class (the class, not the package!), and select Create Dynamic Instance... from its context menu. This is shown in Figure 2.
   Figure 2: Create a dynamic instance.
7. enter the File name: Company.xmi and press Finish. The instance will be created in the model folder.
8. edit the model instance using the context menus New Child of the elements in the model.
9. edit the elements properties in the properties section of the editor, see Figure 3.
   Figure 3: Edit properties with the Generic EMF Form Editor

Now that we have create a model and an instance of that model, we want to "work" with that model instance. E.g., we can use OCL to query the model:

10. Activate the console view and open the Interactive OCL console with the button on the left of the console, see Figure 4.
    Figure 4: Open OCL interpreter console
11. Select an element in the editor, This selected element is the context of the OCL query.
12. Enter your OCL query in the console (in the lower section of the OCL interpreter console). E.g. query the name of the selected element with self.name. The result will be displayed in the upper section of the OCL console as shown in Figure 5. We will demonstrate other queries later on.
    Figure 5: Query the model instance with OCL

Task Model with Xtext

Now that we have a model of our company, we want to assign tasks to the employees. We could create a new ecore model just as demonstrated above, but we want to try something new. So, let's try to not only create a model, but a text editor as well. For that, we will use Xtext. Based on a grammar, Xtext can create an ecore model and a text editor with nice features. The grammar is an annotated EBNF grammar, I do not want to go into the details here (otherwise it wouldn't be a q'n d-tutorial ;-) ). So, we have to create a new project and enter a grammar:

13. Create a new Xtext project:
    File / New / Project... / Xtext /Xtext Project
14. SetMain project name, the Language name (to de.feu.Tasks), and the DSL-File extension (to tasks).
15. Open Tasks.xtext (this is the EBNF-like grammar) and enter your grammar according to Figure 6.
    Figure 6: Xtext grammar defining our task model along with a concrete textual syntax
    Here is the grammer (for copy & paste):

    grammar de.feu.Tasks with org.eclipse.xtext.common.Terminals
    
    import "http://www.eclipse.org/emf/2002/Ecore" as ecore
    import "platform:/resource/de.feu.company/model/company.ecore" as company
    
    generate tasks "http://www.feu.de/Tasks"
    
    Tasks :
     (developers+=Developers)*
     (domainmodels+=DomainModels)*
     (tasks+=Task)*;
     
    Developers :
     'developer' importURI=STRING;
     
    DomainModels :
     'domainmodel' importURI=STRING;
    
    
    Task: 'task' element=[ecore::EObject] 
       'by' developer=[company::Developer] 
       ':' description=STRING;
    

    This grammar defines a container element Tasks. Inside, we can "load" a list of developers and domain models. Eventually, tasks can be defined by assigning an element from a domain model (see element=[ecore::EObject]) to a developer (developer=[company::Developer]) and add a description of that task.
16. Save the gammar and generate the ecore model along with the text editor using the MWE-workflow. For that, select file GenerateTasks.mwe and run the workflow via its context menu Run As / MWE Workflow

Notes:

• In Xtext, a model is called a DSL. Experts like to use different names for "model", sometimes it's cooler to call a model "meta-model" (or, even cooler, "meta-meta-model"), sometimes they call it "domain specific language", abbreviated with a nice TLA (three letter acronym): DSL. Of course, there are good (and sometimes not so good) reasons for doing so, but usually it's easier to simply call a model a model (and an "instance of a model" an "instance of a model") .
• The grammar shown in Figure 6 actually uses a lot of nice Xtext features and you will have to read the Xtext documentation for details. I only want to explain one thing here, which is a little bit advanced. You can import existing models into the grammar (import ...). In the example, we import the ecore model and our previously created company model. We import these models in order to be able to define inter-model references later on. Our Task element will refer to an element, which can be any ecore::EObject, and the developer is to one developer defined in our company model (company::Developer). EMF supports inter-model-references, and Xtext generated editors support that feature as well! The import statements in our grammar only import the models, but later on we want to actually import existing model instances. For that, we need the importURI feature of Xtext to define what instance to import in our actual task model instance.

Although we haven't written a single line of Java code yet, a lot of code has been generated automatically. In order to "activate" that code, which actually defines a new Eclispe plugin, we have to start a new runtime instance of Eclipse, that is we start Eclipse from within Eclipse. In the long run, you will have to get used to Eclipse plugin development anyway... As we have to run a new instance anyway, we can generate an editor for our company model as well. Instead of using a dynamically created instance using the "Generic EMF Form Editor", EMF can generate a Java implementation of the model and an editor as well. We will do that now:

17. Select the company.ecore file and choose New / Other... / Eclipse Modeling Framework / EMF Generator Model from its context menu. Use the suggested file name, select Ecore Model in the next wizard page. Then, press Load in the next page, and eventually Finish on the last wizard page. This creates a generator model, which basically adds some information to the original ecore model which is necessary in order to actually generate code (e.g., the name of the plugins and so on). We do not want to change anything here, but we still need that model.
18. In that model, select the very first element (Company) and select Generate All from its context menu (see Figure 7).
    Figure 7 Generate the model and editor code

Work with Multiple Models

Now that we have generate a lot of code, we want to use the newly created tools. Start a new Eclipse runtime instance (e.g. by selecting Run As / Eclipse Application from a project context menu) and do the following:

19. Create a new project (in the runtime instance):
    File / New / Project... / General /Project
20. Select the project and choose from its context menu:
    New / Other... / Example EMF Model Creation Wizard / Company Model
    This activates our previously generated editor (and a wizard) for creating a company model instance. We can edit a company instance just as we did it with the generic editor. In the wizard, select the Model Object Company and then create a new company as shown in Figure 8. (Don't forget to save the model ;-) ).
    Figure 8: A company model, edited with the generated editor
21. Now, create a new file (File / New / File) inside the project and call it project.tasks. The Xtext generated editor is opened automatically and we can now edit the task model instance.
22. In order to "simulate" a project, we create a simple UML use case model. Simply create a new use case diagram via File / New / Other... / UML 2.1 Diagrams / Use Case Diagram, give it a name (e.g. project_usecase) and draw some use cases, a sample is shown in Figure 9.
    Figure 9: A sample use case diagram
23. Switch back to project.tasks and "import" our company model and the sample use cases. Add a task using content assist (Ctrl-Space) just as shown in Figure 10. Just play around, add three or four tasks in order to be able to follow the next steps.
    Figure 10: Content assist, demonstrating access to imported models
24. Just as at the beginning, we want to execute some OCL queries on our model, but this time on our project.task model instance. For OCL, we always need a context, but unfortunately we cannot select a context (or model element) in the text editor. So we have to reopen the project.task with the generic EMF editor: Choose from its context menu Open With / Other... / Generic EMF Form Editor. You will now see the very same model as in the text editor, but this time you see a tree-based version of the model.
25. Open the OCL console just as above, select an element (here Tasks) and enter a query. For example, we want to know how much tasks are assigned to a specific developer: self.tasks->select(developer.name='Jens')->size(). You can see that in Figure 11
    Figure 11 A little more sensible OCL query

Conclusion

IMHO it is impressing what you can do with all these cool Eclipse modelling tools, without writing a single line of (Java) code. We saw how to

• create an EMF ecore model with the Ecore Tools diagram editor
• open the same ecore model with the Generic EMF Form Editor
• create instances of a model without the need to generate code and without starting a new runtime instance, using the "Create Dynamic Instance" feature (and the generic EMF editor again)
• generate a text editor and a model without a single line of Java code with Xtext
• generate a tree based editor and a Java based model implementation with EMF
• create a UML diagram with the UML Tools diagram editor
• query your models with OCL from the OCL project

We didn't bothered about how to store our models or model intances (it's all XMI), we were using OMG standards like UML (the EMF based implementation is provided by the UML2 project, EMF's ecore is an EMOF-like model, we were using OCL, other implementations are available, too (e.g., BPMN2, SBVR, or SPEM used by EPF). And if you do not find an Eclipse project, you probably will find a third party project providing an EMF based implementation ;-) With tools like GEF or GMF you can create (or generate) diagram editors for your models, and (well, I couldn't resist) with GEF3D you can even create 3D diagram editors, e.g. for visualizing inter-model connections. And there are many more tools out there, partially for simplifying the use of existing tools, for model transformations and code generation and so on. And of course you can adapt and modify the code generated above to suit your need.

Warning: Thin Ice!

While it is very easy to do impressive things (with the help of a tutorial), it is very hard to get into all these frameworks and tools. There are a lot of traps hidden everywhere! Just two examples from the tutorial:

• In the Xtext grammar we used here, the ecore model was imported (see Fig. 6). I tried the very same using the UML2 model, and I got a weird error when generating the code (actually I couldn't generate in that case).
• A NullPointer-Exception is thrown (you can see that in the console of the original Eclipse instance) when opening the project.tasks file with the Generic EMF Form Editor. Fortunately it still is possible to select an element (for the OCL query), but it's a little bit weird.

Well, I probably should file a bug report at least for the first problem... So be warned: Even if the tutorial gives you the impression as if modelling with Eclipse tools is very easy, it sometimes isn't. In general, modifying generated code as well as using tools like GMF is not that simple. But at least there are the newgroups, tutorials, and even books (e.g. the EMF book (2nd edition!) or the GMF book).

Disclaimer

This is a quick and dirty tutorial article. I simply documented an example modelling session by taking some screenshots, and then wrote some explanations. So, there probably are some things I've missed to tell you or errors in my text. Please leave me a comment if you find a bug here ;-) (or if you like the tutorial). May the Model be with you!

User Report: Migrate from oAW Xtext to TMF Xtext

With Eclipse 3.5 comes the new version of Xtext, called TMF Xtext. Since I had to switch to Eclipse 3.5 in order to keep GEF3D running on the latest Eclipse version, I decided to migrate from the old openArchitecture Xtext (oAW Xtext) to the new TMF version. Fortunately the Xtext documentation contains a section about migrating from oAW Xtext to TMF Xtext. Frankly, I was a little bit afraid (never change a running system). But at least I hadn't implemented any content assist or other editor extensions yet, so I hoped that it wouldn't be too hard. Before you read on, please note that I'm only an Xtext user. I don't know any internal details about Xtext. Maybe some of my problems can be solved more efficiently, if that is the case, I would be more then happy if you leave me a comment. My first intention was to migrate my already running grammar as fast as possible, with as few modifications as possible.

The Setting

I have written a model transformation language, and it consists of an Xtext grammar (with about 90 rules), an interpreter for executing the language, and other things which I didn't expected to be involved in the migration, as a debugger component or other user-interface stuff. An Xtend-based M2M transformation exists which modifies the generated ecore model in order to set some default values, add some workarounds for existing oAW bugs, and do other things. In order to migrate my application, I had to

1. migrate the grammar
2. adapt the M2M-transformation
3. adapt client code which uses the model and the Xtext API

In the following old versions are highlighted with a light red back and the new ones with a light green background.

Migrating the Grammar

As described in the Xtext documentation, I had to replace "Enum" with "enum", and add two new lines at the beginning of the grammar. Since "with" has become a keyword. Thus I had to rename "with" in my grammar :

RuleXY: ... ( "with" with=Block)? ...

into

RuleXY: ... ( "with" _with=Block)? ...

This also caused some small changes in the model and the client code. Is there a way to use Xtext keywords as a reference names?

Backtracking

Apparently the parser generation (or some settings) has changed, too. E.g., the following rule worked with the oAW Xtext version:

ParameterReference: QualifiedParameterReference|SimpleParameterReference;
SimpleParameterReference: type=Type (name=ID)?;  
QualifiedParameterReference: type=Type name=ID;

The idea of these rules is to force the definition of a name in case of qualified parameter references and to make it optional in simple ones. Unfortunately these rules are not working with the new version. Since ANTLR is a LL-parser, there are some constraints on how to define the grammar. There is a document explaining how to remove backtracking from a grammar, unfortunately some of the described techniques cannot be used with Xtext. In some cases I was able to fix the problem by rewriting my grammar, as in this example: I simply had to reorder the first rule:

ParameterReference: SimpleParameterReference|QualifiedParameterReference;
SimpleParameterReference: type=Type (name=ID)?;  
QualifiedParameterReference: type=Type name=ID;

In some cases I couldn't find a solution (maybe it is possible, but I'm not an LL/LR-expert). Actually I'm not the first one with this problem, and there is a posting in the TMF newsgroup about this. If you cannot rewrite your grammar, you need backtracking (and lookahead). Thanks to this posting, I was able to fix the problem. I had to edit the MWE-file from

<fragment class="org.eclipse.xtext.generator.AntlrDelegatingFragment" />

to

<fragment class="de.itemis.xtext.antlr.XtextAntlrGeneratorFragment">
   <options k="2" backtrack="true" memoize="true"/>
</fragment>

Note that you have to install the itemis generator fragment from http://download.itemis.com/updates as described in the Xtext documentation (page 62). Maybe I'm going to "left factor" my grammar in a future version, but for now I only wanted to get my application running.

Lexer Rules

In the oAW version, lexer rules were defined with the keyword "native", which has been changed to "terminal". Not mentioned in the migration guide yet: INT was redefined in the new version. I had rewritten the definition in the old version in order to remove the optional sign (as my language is handling this itself). This is the old INT definition:

Native INT: ('-')?('0'..'9')+ 

And here's the new one:

terminal INT: ('0'..'9')+ 

In my case I could remove my own definition since the new one matches my requirements. In other cases, this might be a trap. According to the documentation, it should be possible to define the type of a terminal rule now. I tried this as follows:

terminal FLOAT returns ecore::EFloat: ('0'..'9')*'.'('0'..'9')+;

As a matter of fact, the code is generated as expected, but the parser still returns a String, which leads to a ClassCastException. So I removed the type definition and added a convenient method to my model, as described below. I assume this is a bug or something, I will file a report later on ;-). Since the old version always returned Strings, my code was prepared for that, anyway.

OrmPackage missing

I don't know exactly why or when, but at some point in time my grammar file contains got an error marker:

- WrappedException: java.lang.ClassNotFoundException: com.sun.java.xml.ns.persistence.orm.OrmPackage"

This bug seems to be fixed already, fortunately it is possible to generate the model and code from the grammar, even with this error.

Unassigned Actions

In the oAW Xtext version, a model element was assigned whenever a non-abstract rule was hit. My grammar contained the following rule:

NullLiteral: "null";

This rule is not working with the new TMF Xtext version. As found in the documentation "by default the object to be returned by a parser rule is created lazily on the first assignment". Although I've read that passage, I was not aware of the consequences. In the new version, an element is not automatically created when the rule is hit, but only when an assignment is to be executed. That is, a rule which contains only terminals does not create a model element, as in the example above. While this first example is obvious, the problem is sometimes hidden. Here is another more tricky example:

Block: "{" (statements+=BlockStatement)* "}";

If no "BlockStatements" are specified, that is in case of an empty block ("{}"), no block element is created at all. I posted that problem to the newsgroup, and Sebastian Zarnekow immediately solved my problem (Thank you very much, Sebastian!). The solution is to use unassigned actions, which forces the creation of an element of the specified type. So I had to change the rules above as follows:

NullLiteral: {NullLiteral} "null";
Block: {Block} "{" (statements+=BlockStatement)* "}";

I figure this is a bitchy trap, especially if there is an optional assignment.

Model Changes

I'm already using an Xtend M2M-transformation to adjust the generated ecore model. TMF Xtext supports this mechanism, I simply had to rename my Xtend extension into MitraPostProcessor.ext (Mitra is the name of my language, by the way).

Enumeration NULL values

There are some adjustments necessary due to changed behaviour of Xtext. Most notable: There is no "NULL" value generated for enumerations. Also, it is not possible to define a hidden enumeration in the grammar, which might could have solved that problem. IMHO, this is a major drawback, as this makes a postprocessing of the created ecore file necessary. But they can be added to the M2M transformation. Unfortunately, I didn't got this problem solved with Xtend, I posted that to the TMF newsgroup. This is at least one solution using Java, although I would prefer a pure Xtend solution:

process(ecore::EPackage p):
 addNullValue(p, "VisibilityModifier") ->
 ... ;

Void addNullValue(ecore::EPackage p, String toEnum):
 doAddNullValue( (ecore::EEnum) p.getEClassifier(toEnum) );
 
Void doAddNullValue(ecore::EEnum e):
 JAVA de.feu.MitraPostProcessorHelper.addNullValue(org.eclipse.emf.ecore.EEnum);

And this is the Java extension:

public class MitraPostProcessorHelper {

 public static void addNullValue(org.eclipse.emf.ecore.EEnum e) {
  for (EEnumLiteral literal : e.getELiterals()) {
   literal.setValue(literal.getValue() + 1);
  }
  EEnumLiteral nullLiteral = EcoreFactory.eINSTANCE.createEEnumLiteral();
  nullLiteral.setName("NULL");
  nullLiteral.setLiteral("NULL");
  nullLiteral.setValue(0);
  e.getELiterals().add(0, nullLiteral);
 }
}

In order to make a literal the default value, it must be the first defined literal. This is why e.getELiterals().add(0, nullLiteral); is required, simply adding the literal with e.getELiterals().add(nullLiteral); doesn't work.

Java Body Annotation in the Ecore Model

In the old version, Xtext didn't create the genmodel and no code. I added this feature and I also edited the generated code after is was created using oAW extension. Since Xtext is doing the code generation now automatically, I didn't feel that comfortable with manually editing the generated model code anymore. So I decided to add Java annotations to the ecore model in order to add some tiny helper methods. This is how body annotations are added to an ecore model in Xtend:

Void addJavaBodies(ecore::EPackage p):
 let ecorePackage = p.getEcorePackage():
 p.getEClassifier("Bounds").addOperation("isMany",
     ecorePackage.getEClassifier("EBoolean"),
       'return upper==-1 || upper>1;') ->
   ...;
/*
This is a hack in order to retrieve the ecore package itself. This is usually
done with a Java extension, but I did not want to add a Java extension here.
The trick is to retrieve the package by navigating to the container of an ecore
type (here a String).
*/
cached ecore::EPackage getEcorePackage(ecore::EPackage p):
  (ecore::EPackage)
  ( ((ecore::EClass) (p.getEClassifier("MetamodelDeclaration")))
    .getEStructuralFeature("type").eType.eContainer
  );
 

create ecore::EOperation addOperation(ecore::EClassifier c, String strName, ecore::EClassifier type,  String strBody):
 setName(strName) ->
 setEType(type) ->
 eAnnotations.add(addBodyAnnotation(strBody)) ->
 ((ecore::EClass)c).eOperations.add(this);


create ecore::EOperation addOperation(ecore::EClassifier c, String strName, ecore::EClassifier type, 
 List parameters,
 String strBody):
 setName(strName) ->
 setEType(type) ->
 parameters.collect(e|((List)e).setParameter(this)) ->
 eAnnotations.add(addBodyAnnotation(strBody)) ->
 ((ecore::EClass)c).eOperations.add(this);

create ecore::EParameter setParameter(List parameter, ecore::EOperation op):
 setName((String)parameter.get(0)) ->
 setEType((ecore::EClassifier) parameter.get(1)) ->
 op.eParameters.add(this)
;

create ecore::EAnnotation addBodyAnnotation(ecore::EOperation op, String strBody):
 setSource("http://www.eclipse.org/emf/2002/GenModel") ->
 this.createBody(strBody) ->
 op.eAnnotations.add(this);

create ecore::EStringToStringMapEntry createBody(ecore::EAnnotation anno, String strBody):
 setKey("body") ->
 setValue(strBody) ->
 anno.details.add(this);

API Changes

So far I found three API changes which required changes in my code. These changes are rather simple and briefly explained in the following.

NodeUtil.getNode(..)

First of all, NodeUtil.getNode(EObject); has changed. There is no type Node anymore. I had to change my code from

Node node = NodeUtil.getNode(eobj);

to

NodeAdapter adapter = NodeUtil.getNodeAdapter(eobj);
AbstractNode node = null;
if (adapter != null) {
 node = adapter.getParserNode();
}

Resources and Setup

For (JUnit) tests, I have to call a setup method in order to register Xtext's resource implementations and my model. This is now simply achieved via a generated setup method. This is the setup method I'm using now in JUnit tests:

public static void setup() {
   System.setProperty("org.eclipse.emf.ecore.EPackage.Registry.INSTANCE",
   "org.eclipse.emf.ecore.impl.EPackageRegistryImpl");
   MitraStandaloneSetup.doSetup();

No DSL-specific Editor Class

A little bit surprising at first is the absence of a DSL specific editor class, in my case MitraEditor. Instead, the general Xtext editor class is used, DSL specifics are injected using Google Guice as described in the Xtext manual. Most surprising is the new syntax found in the plugin.xml which is used to inject things. Here is the declaration of my editor using the new notation (the plugin.xml is generated):

<extension point="org.eclipse.ui.editors">
   <editor
       class="de.feu.MitraExecutableExtensionFactory:org.eclipse.xtext.ui.core.editor.XtextEditor"
       contributorClass="org.eclipse.ui.editors.text.TextEditorActionContributor"
       default="true"
       extensions="mitra"
       id="de.feu.Mitra"
       name="Mitra Editor">
   </editor>
</extension>

The attribute "class" defines the Guice module to be created and the editor class. Unfortunately I need the editor class in order to add breakpoint markers (for my debugger plugin). The old version looks like this:

public class MitraBreakpointAdapterFactory implements IAdapterFactory {
 public Object getAdapter(Object adaptableObject, Class adapterType) {
  if (adaptableObject instanceof MitraEditor) {
    ...

How do I identify the mitra editor in the new version? First of all, instead of a generated DSL specific editor, an XtextEditor is used (this class is found in plugin org.eclipse.xtext.ui.core). Then the current language can be retrieved via getLanguageName(). This is the new version:

public Object getAdapter(Object adaptableObject, Class adapterType) {
 if (adaptableObject instanceof XtextEditor) {
  XtextEditor xtextEditor = (XtextEditor) adaptableObject;
  if ("de.feu.Mitra".equals(xtextEditor.getLanguageName())) {
  ...

Note that the name of the editor class has to be changed in the plugin.xml of the debug plugin accordingly.

Summary

After applying the changes described above, my language worked as expected (at least all the tests are green and my application is running). Since I haven't written any extensions for content assist or other UI things, I only had to adjust the grammar, my M2M-transformation and the code a little bit. If you do not have an M2M-transformation, you may have to add it in order to add the missing NULL values for enumerations -- I assume this is the case quite often. TMF Xtext adds some very nice features (grammar mixings, improved linking/cross references feature with ILinkingService, serialization with formatter, return types, and tree rewrite actions). These features will simplify my hand written code immensely, and I'm looking forward using them. Last but not least, since Xtext is now part of the Eclipse modeling package, it is much easier to install an Xtext based DSL and editor. Last but not least: Kudos to the Xtext team! The new version looks really nice! And many thanks for the great and fast (newsgroup) support!