Modular Moose | Blog

Building and sharing Moose models with MooseNexus

Wed, 21 Oct 2026 00:00:00 GMT

Building a Moose model is often only the first step of an analysis. We also need to know which source project produced it, which dependencies were used, which tools extracted it, and how another person can build or retrieve the same model.

MooseNexus brings these steps into one repeatable workflow. It builds Moose models, records how they were produced, stores them as versioned artifacts, and can share them through an artifact registry.

This post first uses the MooseNexus API from a Moose image, then performs the same kind of build with the command-line interface. The final section explains how an artifact registry can store and distribute the resulting models and images.

What MooseNexus does

MooseNexus sits between a source project and the Moose model that we want to analyze.

From source project to versioned model artifact: MooseNexus records the result locally and can publish it to a shared artifact registry.

A build starts with two pieces of information:

the directory containing the source project;
coordinates that identify it: a group, a name, and a version.

MooseNexus then imports the project metadata, resolves the dependencies needed for extraction, runs the model extractor, and records the result. For Java projects it can recognize Maven and Gradle projects. Projects without a supported build descriptor can be described explicitly as unmanaged projects.

The result is more than a model file. A model artifact also records its source-project identity, dependency-resolution result, and build provenance, including the Moose, Pharo, MooseNexus, and extractor versions. This information makes it possible to inspect how a model was produced and to select a compatible environment later.

By default, projects and artifacts are kept in a local repository below ~/.moose. The important principle is similar to Maven coordinates: com.example:demo:1.0.0 identifies one version of a project independently of the machine on which it was analyzed.

Build a model with the API

The main API entry point is MooseNexusBuildSpec. A build specification describes the intended result; MooseNexus handles the individual import, dependency, extraction, and storage steps.

1. Load MooseNexus

For a Java project, load the Java runtime group in a Moose image:

Metacello new
  githubUser: 'moosetechnology'
  project: 'MooseNexus'
  commitish: 'v1.x.x'
  path: 'src';
  baseline: 'MooseNexus';
  load: 'Java'.

The floating v1.x.x tag selects the newest compatible version in the MooseNexus 1.x series. Use an exact release tag when the build must be fully reproducible.

2. Identify the source project

Let us build a model for a project located at /path/to/demo. We first give it stable coordinates:

coordinates := MooseNexusCoordinates
  group: 'com.example'
  name: 'demo'
  version: '1.0.0'.

The coordinates identify the project in the local repository and, later, in an artifact registry. They do not need to be the same as Maven or Gradle coordinates, although using the same identity can be convenient.

3. Describe the build

Create a build specification from the coordinates and source directory:

spec := MooseNexusBuildSpec
  coordinates: coordinates
  sourceDirectory: '/path/to/demo' asFileReference.

spec
  modelName: 'demo-model';
  modelComment: 'Model used for the architecture analysis'.

The source directory can be anywhere; it does not have to be a Git checkout. For a Maven or Gradle project, MooseNexus detects the appropriate importer from the directory. The explicit coordinates remain the identity of the MooseNexus artifact.

This separation is useful: the build tool describes how the software project is assembled, while the build specification describes the model artifact that we want MooseNexus to produce.

4. Execute the build

Execute the specification in the default repository:

result := spec execute.

This performs the complete workflow and records the project below the default ~/.moose repository. The result gives access to three useful objects:

result project.
result modelArtifact.
result dependencyResolutionResult.

Use executeIn: with a MooseNexusRepository when the artifact should be stored elsewhere:

repository := MooseNexusRepository
  localDirectory: '/path/to/model-repository' asFileReference.

result := spec executeIn: repository.

5. Load the model in Moose

Building records the artifact but does not force it into the current image. Import it explicitly when it is time to analyze it:

model := result project importModel: result modelArtifact.

The imported model is registered in MooseModel root. Keeping the build and import steps separate also lets headless jobs produce artifacts without retaining a full analysis image.

For most clients, execute or executeIn: is the API to keep. MooseNexus also exposes the individual build phases, but those are mainly useful when an application needs custom orchestration.

Build from the command line

The MooseNexus CLI runs the same ideas without requiring us to prepare a Moose image by hand. It creates an isolated workspace, provisions a compatible Moose/Pharo runtime, loads MooseNexus, executes a build specification, and stores the result.

Install it with npm. The CLI requires Node.js 22 or later:

npm install --global @moosetechnology/moosenexus

For Java extraction, the default setup also requires Docker. We can check the available tools before starting a build:

moosenexus doctor

The interactive wizard is the easiest way to discover the commands:

moosenexus --wizard

For a Maven or Gradle project, a direct build can be as small as:

moosenexus build-model \
  com.example:demo:1.0.0 \
  /path/to/demo

This creates a model artifact and installs it in the local MooseNexus repository. To create a ready-to-open Moose image containing the imported model instead, use build-image:

moosenexus build-image \
  com.example:demo:1.0.0 \
  /path/to/demo \
  --adopt

The built image is first stored as an immutable artifact. --adopt also copies it to the standard Pharo images directory, where it becomes a working image that we can open and modify.

The two outputs serve different needs:

a model artifact is smaller and can be imported into a compatible existing Moose image;
an image artifact is a ready-to-open environment containing the model and its compatible runtime.

What an image artifact contains

An image artifact is selective rather than a copy of the complete build workspace. Its portable ZIP contains:

at its root, the .image file, its .changes file when available, the available Pharo .sources files, meta-inf.ston, pharo.version, and a CLI build report;
under pharo-local/MooseNexus/, the recorded project metadata, model artifact, and a copy of the analyzed source project.

Other image-local directories, such as pharo-local/iceberg, and the Pharo VM are not included. MooseNexus does not scan these files for credentials or remove secrets: the source copy, saved image, or changes file may contain sensitive information. Review an artifact before publishing it.

Reuse an image artifact

An installed image artifact can be adopted several times, much like making working copies from a template. Each adoption creates a separate writable image while leaving the repository artifact unchanged:

moosenexus adopt-image com.example:demo:1.0.0
moosenexus adopt-image com.example:demo:1.0.0 --adopt-as demo-alice

By default, these copies are placed in the standard Pharo images directory and appear in Pharo Launcher. This is not a Launcher template: MooseNexus copies the image files and launcher metadata from an installed artifact. The recorded project and source copy remain in the local MooseNexus repository rather than being duplicated into every adopted image.

Use moosenexus artifacts to see what is installed locally. Build options can also be kept in a YAML file and passed with --config, which is preferable for repeatable scripts and continuous integration. --dry-run shows the resolved workflow without executing it, and --json produces machine-readable output for automation.

The important need here is artifact storage and distribution: a shared place where versioned Moose models and images can be published, found, and retrieved.

MooseNexus currently implements this with the Open Container Initiative distribution conventions. Although OCI is best known for container images, its registries can store other versioned artifacts as well. Reusing that standard gives MooseNexus an existing upload, download, authentication, and access-control ecosystem instead of inventing a Moose-specific artifact server.

An organization can therefore use infrastructure it may already have—such as Harbor, Sonatype Nexus, Artifactory, or another OCI-compatible registry—to distribute Moose models and images. MooseNexus defines what goes into the artifact, while ORAS performs the standard registry upload and download.

A MooseNexus model bundle contains:

the exported model;
an archive of the analyzed sources;
the project metadata;
the model manifest, including build and extractor provenance.

Including the sources makes the artifact inspectable and helps reproduce the model, but it is also important to remember before publishing: a model bundle can contain proprietary source code. Use a registry and access policy appropriate for the project.

First authenticate ORAS with the target registry:

oras login registry.example.com

Then add a registry and namespace to the build command:

moosenexus build-model \
  com.example:demo:1.0.0 \
  /path/to/demo \
  --registry registry.example.com \
  --namespace team/models

MooseNexus still installs the artifact locally and also publishes it under a reference derived from the project coordinates, for example:

registry.example.com/team/models/moosenexus/com.example/demo:1.0.0

Another machine can retrieve the model with the same coordinates:

moosenexus pull-model \
  com.example:demo:1.0.0 \
  --registry registry.example.com \
  --namespace team/models

pull-model installs the artifact in the local MooseNexus repository. It can then be imported through the API shown earlier. Use pull-image ... --adopt when the published artifact is a Moose image and a ready-to-open copy is wanted instead.

Publication is always explicit: building locally does not silently upload anything. MooseNexus also keeps registry credentials outside its own configuration and relies on the authenticated ORAS installation.

API or CLI?

Both interfaces use the same project coordinates, build specification, repository, and artifact concepts.

Use the API when…	Use the CLI when…
the build is part of a Moose application	the build should start from a clean, isolated runtime
an image needs custom import or extraction policy	a shell script or CI job should run the build
the resulting model will be analyzed immediately	a model or ready-to-open image should be produced and shared

The API is the stable foundation, and the CLI automates the surrounding runtime setup. Starting with the CLI is often easier; moving to a Smalltalk build specification gives more control without changing the underlying model-artifact workflow.

MooseNexus therefore turns a one-off model extraction into a named, versioned, and shareable build result. That becomes especially valuable when a model must outlive the image in which it was first created—or when another person, machine, or CI job needs to produce the same analysis input.

Setup MooseCI on your repository with GitHub Actions

Mon, 05 Oct 2026 00:00:00 GMT

Setup MooseCI on your repository with GitHub Actions

What is MooseCI?

MooseCI is a tool that runs automatic analyses on your code. It looks for code quality problems and writes a report for you.

MooseCI is built on top of Moose. It can run in headless mode, so you can use it on a CI server.

You can use MooseCI in two ways:

as a command line tool, with Docker
inside a CI, with its GitHub Action for example

For now, MooseCI supports Java and Python.

MooseCI runs its analyses on the Famix model of your project, not on the FAST model.

Run MooseCI with Docker

The easiest way to try MooseCI is with Docker.

First, pull the image for your language. For Java, use the latest image, and for Python, use the python image:

docker pull ghcr.io/moosetechnology/moose-ci:latest
docker pull ghcr.io/moosetechnology/moose-ci:python

Go to the folder you want to analyze, and create a configuration file with the init command:

docker run -v "$(pwd):/src" ghcr.io/moosetechnology/moose-ci:latest init

This creates a moose-ci.ston file. You can edit it to choose the rules and the metrics.

Then run the analysis:

docker run -v "$(pwd):/src" ghcr.io/moosetechnology/moose-ci:latest analyze

MooseCI prints the report in the console and writes a JSON report in .moose-ci/report.

Run MooseCI in a CI with GitHub Actions

MooseCI is also available as a GitHub Action. Add it to a repository, and it runs on every pull request.

The Action:

runs MooseCI on your project
uploads the report as an artifact
comments on the pull request with a download link and a short summary of the analysis

Example with VerveineJ

Let’s take VerveineJ as an example. VerveineJ is a Java project that parses Java code and exports it to JSON/MSE for Moose.

We added MooseCI to VerveineJ. You can see the result in this pull request.

First, create the configuration file. Go to the folder you want to analyze, and run init:

cd app/src/main/java/fr
docker run -v "$(pwd):/src" ghcr.io/moosetechnology/moose-ci:latest init

This creates app/src/main/java/fr/moose-ci.ston. The file looks like this:

MooseCIConfig {
  #projectLanguage : #java,
  #metrics : [
    #packages,
    #classes
  ],
  #rules : [
    #too_many_parameters : 10,
    #large_class : 20,
    #unused_local_variable,
    #unused_parameter
  ],
  #outputFormats : [
    #json
  ],
  #outputPath : '.moose-ci/report',
  #visualizations : [ ],
  #isRemote : false
}

In this file, you can:

set the language with #projectLanguage
choose the metrics to compute with #metrics
choose the quality rules with #rules, and give a threshold to some rules (for example #too_many_parameters : 10)
choose the report format and location with #outputFormats and #outputPath

Then add the workflow. Create .github/workflows/moose-ci.yml:

name: MooseCI
on: pull_request
permissions:
  contents: read
  actions: write
  pull-requests: write
jobs:
  analyze:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: moosetechnology/setup-MooseCI@v1.0.0
        with:
          project-path: ./app/src/main/java/fr
          project-language: java

Some details:

on: pull_request runs the workflow on every pull request.
actions: write is needed to upload the report.
pull-requests: write is needed to write the comment.
project-path is the folder to analyze. It must contain moose-ci.ston.
project-language selects the image. Use java or python.

On each pull request, MooseCI comments with the analysis. For VerveineJ, the summary looks like this:

The report also contains the list of rule violations, with the location of each problem in the source code.

Use MooseCI in another CI

MooseCI is not limited to GitHub. In any other CI, you can run the same Docker image and the same commands as the CLI, for example:

docker run -v "$(pwd):/src" ghcr.io/moosetechnology/moose-ci:latest analyze

The report is written in the MooseCI report folder (.moose-ci/report). Your CI can then publish it as an artifact.

Conclusion

MooseCI is easy to use. You can run it locally with Docker, or on every pull request with the GitHub Action. You only need a configuration file and a small workflow.

Implementing a Static Single Assigment (SSA) in FAST-Python

Tue, 29 Sep 2026 00:00:00 GMT

Implementing the SSA of Python in FAST-Python

What is SSA?

Static Single Assignment is a program representation where every variable is assigned exactly once. The same logical variable x is renamed into several versions x_1, x_2, … — one per assignment — and each use points to the version that can actually provide its value.

The goal is to make data flow explicit in the AST: instead of “this x is assigned somewhere before”, we get “this x is exactly the value produced by that assignment”.

x = 2        # x_1
print(x)     # uses x_1

x = 3        # x_2
print(x)     # uses x_2

The hard part starts when the value of a variable can come from several assignments depending on the control flow:

if z > 3:
    x = 2     # x_1
else:
    x = 3     # x_2

print(x)      # uses phi(x_1, x_2)

After the merge of the two branches there is no single assignment that produced x: we say x has a phi version phi(x_1, x_2), a virtual assignment stating “the value is x_1 if we came from the first branch, x_2 if we came from the second”. An analysis consuming the SSA can then explore both possible values.

For us, the SSA is the third stage of the pipeline, sitting on top of the two previous stages:

Import → Local resolution → CFG → SSA

It has a hard prerequisite: we must know, for each use, which declaration it refers to (the local resolution stage) and which control-flow paths exist (the CFG stage). If you have not read about the FAST CFG and its visitor, I strongly recommend the Visiting the control-flow graph article first — this blog post is the direct continuation.

Architecture: one visitor on top of the CFG

The whole transform lives in a single class, FASTPythonSSAVisitor. Its declaration shows the trick that makes it work: it combines two visiting abilities—

FASTPythonVisitor << #FASTPythonSSAVisitor
  traits: 'FASTCFGTVisitor';
  slots: { #newVariablesMap. #localDeclarations };
  tag: 'CFG/LocalResolver/SSA';
  package: 'FAST-Python-Tools'

the superclass FASTPythonVisitor brings the language visitor (the visitFASTPyXxx: methods generated from the model, through FASTPyTVisitor);
the trait FASTCFGTVisitor brings the CFG visitor (the visitCFGXxxBlock: methods that walk the blocks of the CFG, plus the conditional-branch hooks).

So a single accept: can traverse the CFG (visitCFGBlock: and friends) and, when a block contains Python statements, dispatch back to the language-specific visit methods (visitFASTPyAssignment:, visitFASTTCanBeVariable:, …).

The trait is generic and lives in the base FAST project (FAST-Core-Tools), which means any language model can reuse it the same way. It is the same trait-based composition idea as the local resolver, but applied to the CFG.

The state

The visitor has only two instance variables:

localDeclarations — an IdentitySet of every local declaration that received at least one SSA version while walking the entity. It is the source of the “current active versions” snapshot (see below).
newVariablesMap — an IdentityDictionary used only during the merge of conditional branches, to accumulate, per branch, the versions that were created inside that branch.

And the entry point is a class-side convenience:

FASTPythonSSAVisitor class >> resolve: aPythonEntity
    ^ self new resolve: aPythonEntity

The three SSA value classes

The versions are not strings, they are first-class objects living in FAST-Core-Tools (again reusable across languages), and they are Moose entities: each version is added to the model, so it is persisted and exported with it.

FASTAbstractVariableVersionSSA — abstract root. It holds the localDeclaration the version belongs to, and offers the queries we need once the SSA is built: nodes (the accesses directly linked to this version, the bidirectional opposite of ssaVersion), localUses, readAccesses, writeAccesses (restricted to this version), name, isPhi, ssaVariables.
FASTVariableVersionSSA — a basic version for one assignment. It carries an integer version number, so name answers x_1, x.y_1…
FASTVariablePhiVersionSSA — a phi version. It carries choices, the collection of versions being merged, so name answers phi(x_1, x_2), and isPhi answers true.

FASTVariableVersionSSA >> name
    ^ String streamContents: [ :s |
          s
              nextPutAll: self localDeclarationName;
              nextPut: $_;
              nextPutAll: version asString ]

FASTVariablePhiVersionSSA >> name
    ^ 'phi(' , ($, join: (choices collect: #name)) , ')'

Nested merges flatten themselves: the phi’s addChoiceToPhi: re-dispatches each choice into the receiving phi, so a phi inside a branch ultimately contributes its own choices to the outer phi.

The links carried on the model

The SSA result is attached to the model through two links defined in FAST-Core-Tools on the trait FASTTCanBeLocalDeclaration (used by FASTTEntity, so available on every FAST entity):

entity ssaVersion — on any access (read or write) to a variable, the version it is linked to. This is a bidirectional relation: each version exposes the accesses linked to it through its nodes collection.
declaration activeVersion — on the local declaration, the current version at the current point of the traversal. This is the internal “state variable” of the algorithm: it is bumped at each assignment and replaced by a phi at each merge. It stays a plain attribute.

FASTTCanBeLocalDeclaration >> activeVersion
    ^ self attributeAt: #activeVersion ifAbsent: [ nil ]

FASTTCanBeLocalDeclaration >> ssaVersion
    <FMProperty: #ssaVersion type: #FASTAbstractVariableVersionSSA opposite: #nodes>
    ^ self attributeAt: #ssaVersion ifAbsent: [ nil ]

The SSA is then a traversal that, at every write, creates a version, adds it to the model, and sets it as activeVersion; at every read, copies activeVersion into ssaVersion of the access.

The pipeline inside `resolve:`

FASTPythonSSAVisitor>>#resolve: is deliberately buildable in your own project step by step:

FASTPythonSSAVisitor >> resolve: aFASTBehaviouralEntity
    "First we do a local resolution so that each entity is linked to its
     local declaration."
    FASTPythonLocalResolverVisitor resolve: aFASTBehaviouralEntity.

    "Then we build the CFG to know the different paths in the entity."
    cfg := aFASTBehaviouralEntity cfg.

    "And lastly, I can build the SSA now that I have the CFG and the
     local declarations."
    self visit: cfg

Three stages, three small lines. The local resolution is done inside the SSA visitor only for convenience; you can equally run it beforehand.

Producing the SSA versions

The core: `visitFASTTCanBeVariable:`

Just like the local resolver, the SSA logic is concentrated on the trait FASTTCanBeVariable, caught in a single visit method for every node kind that can be a variable (identifier, attribute access, subscript, walrus):

FASTPythonSSAVisitor >> visitFASTTCanBeVariable: aTCanBeVariable

    aTCanBeVariable isVariableWriteAccess
        ifTrue: [ self handleNewAssignemntTo: aTCanBeVariable ]
        ifFalse: [
            aTCanBeVariable localDeclaration isNonLocalDeclaration ifFalse: [
                "Subscript is a special case since it is not directly assigned.
                 So the first read acts as a new declaration."
                (aTCanBeVariable isSubscript
                     and: [ aTCanBeVariable localDeclaration activeVersion isNil ])
                    ifTrue: [ self handleNewAssignemntTo: aTCanBeVariable ]
                    ifFalse: [ aTCanBeVariable ssaVersion: aTCanBeVariable localDeclaration activeVersion ] ] ].

    super visitFASTTCanBeVariable: aTCanBeVariable

The rule is a direct SSA translation of the local resolver’s “write declares, read binds”:

write access → create a new version (handleNewAssignemntTo:);
read access → copy the declaration’s activeVersion into the access’s ssaVersion;
read of an unresolved name (localDeclaration isNonLocalDeclaration) → nothing. print, len, imported functions and variables have no declaration in the model, so they get no version (ssaVersion stays nil);
first read of a subscript whose declaration has no version yet → it behaves like a declaration: the first x[3] read is treated as an assignment (mirroring the local-resolver convention), so its version is created there.

Creating a version

FASTPythonSSAVisitor >> handleNewAssignemntTo: aFASTEntity
    aFASTEntity ssaVersion: ((self createVariableVersionFor: aFASTEntity)
                 newVersionNumber;
                 yourself)

FASTPythonSSAVisitor >> createVariableVersionFor: aFASTEntity
    | newSSA |
    newSSA := FASTVariableVersionSSA for: aFASTEntity.
    aFASTEntity mooseModel add: newSSA.
    aFASTEntity lastSSAVersion ifNotNil: [ :lastSSA | newSSA version: lastSSA version ].

    aFASTEntity localDeclaration activeVersion: newSSA.
    localDeclarations add: aFASTEntity localDeclaration.
    ^ newSSA

FASTVariableVersionSSA for: aFASTEntity records aFASTEntity localDeclaration as the declaration the version belongs to. Then:

the new version is added to the model — the versions are Moose entities, so this makes them persistent and exported with the model;
the version number is seeded from lastSSAVersion — the active version of the previous declaration of the same name, chased through the local-resolver shadowing chain: it is what makes x_1 then x_2 even when an unrelated def x(): was declared in between;
the new version becomes the activeVersion of the declaration;
the declaration is added to localDeclarations (so the “snapshot” of active versions, used for the branches, sees it);
newVersionNumber bumps the integer — version := version + 1 — and the result is stored as the write access’s ssaVersion.

FASTPyEntity >> lastSSAVersion
    "scrolling the shadowing chain of the declaration to find a previous version"
    | entity |
    entity := self localDeclaration.
    [ entity isNotNil ] whileTrue: [
        entity activeVersion ifNotNil: [ :version | ^ version ].
        entity := entity shadowing ].
    ^ nil

Reordering the visits: the order matters for SSA

The generated visitor visits the children of a node in metamodel order. For the SSA, that order is often wrong because a read must be resolved against the versions before the node, and a write must only bump the version after its right-hand side has been read. We overwrite the generated visit methods in four places.

Assignments: the right side comes first

This is the most important one. In the metamodel, the left of an assignment comes before its right. For the SSA we must invert:

visitFASTPyAssignment: anAssignment
    "Visiting the right field before the left because the variables at the
     right should be linked to the previous assignment."
    self visitFASTPyTAssignable: anAssignment.
    self visitFASTPyExpression: anAssignment.

    self visitEntity: anAssignment right.
    self visitEntity: anAssignment left.
    self visitEntity: anAssignment type

Consider x = x + 1. In metamodel order we would visit the left x first: it would create x_2 and set it as activeVersion, and then the x in x + 1 would read — wrongly — the new x_2. By visiting right before left, the read sees x_1 and the write bumps afterwards.

Function definitions: parameters before the body

Parameters are declarations that the body reads, so they have to be visited before the body:

visitFASTPyFunctionDefinition: aFunctionDefinition
    "Reordering so parameters are visited before the statements, in order
     to have all the declarations."
    self visitFASTTWithParameters: aFunctionDefinition.
    self visitEntity: aFunctionDefinition returnType.
    self visitFASTPyTDefinition: aFunctionDefinition.
    self visitFASTPyTWithTypeParameters: aFunctionDefinition.
    self visitFASTPyStatement: aFunctionDefinition

visitFASTPyMethodDefinition: aMethodDefinition
    "We resolve in the same way as functions."
    self visitFASTPyFunctionDefinition: aMethodDefinition

Definitions rebuild their own CFG

A function definition appearing as a statement of the module is inside the module CFG. But the function’s body is a separate control-flow world. When we reach a definition, we therefore rebuild a fresh CFG and restart the traversal on it, so each definition is turned into its own SSA form:

visitFASTPyTDefinition: aTDefinition
    "We need to redo the CFG for definitions."
    | cfg |
    self visitFASTTNamedEntity: aTDefinition.
    self visitCollection: aTDefinition decorators.

    cfg := aTDefinition cfg.
    self visit: cfg

Building the phi versions

Where do phi versions come from? The CFG visitor. When FASTCFGTVisitor visits a conditional block, it visits all the branches before the merge, and during that it calls four hooks you can override. Their default implementation in the trait is empty:

preConditionalsBranchesVisitOf: aConditionalBlock          "before the first branch"
preConditionalsBranchVisitOf: block conditional: aConditionalBlock   "before each branch"
postConditionalsBranchVisitOf: block conditional: aConditionalBlock  "after each branch"
postConditionalsBranchesVisitOf: aConditionalBlock         "after all branches, before the merge"

That is exactly the machinery we fill in to build phis. The idea is a three-step protocol:

before the branches: snapshot the active versions;
after each branch: record which new versions were created by that branch;
after all branches: merge, per variable, one version per branch (plus the snapshot version for branches that did not assign it) into a phi.

Step 1 — snapshot before the branches

FASTPythonSSAVisitor >> preConditionalsBranchesVisitOf: aConditionalBlock
    "When we start to visit the branches of a conditional, we create an
     entry for this conditional in the new variables map and we save the
     active versions present before visiting the children."
    newVariablesMap at: aConditionalBlock put: (IdentityDictionary with: #previousVariables -> self currentActiveVersions)

currentActiveVersions is a snapshot of every declaration’s active version at this point:

currentActiveVersions
    ^ localDeclarations collect: #activeVersion

Step 2 — record the new versions of each branch

FASTPythonSSAVisitor >> postConditionalsBranchVisitOf: firstBranchBlock conditional: aConditionalBlock
    "Once we visited a conditional branch, we save the current active
     versions for this branch."
    | mapForConditional |
    mapForConditional := newVariablesMap at: aConditionalBlock.
    mapForConditional at: firstBranchBlock put: (self currentActiveVersions difference: (mapForConditional at: #previousVariables))

The difference is atomic: subtracting the snapshot (#previousVariables) from the current active versions leaves exactly the versions created inside this branch (the ones that were not there before). We store them per branch block.

Step 3 — build the phis and merge

FASTPythonSSAVisitor >> postConditionalsBranchesVisitOf: aConditionalBlock
    "Once we are done visiting a conditional block we can clean the map and build the Phi variables."

    | mapForConditional previousVariables |
    mapForConditional := newVariablesMap at: aConditionalBlock.
    previousVariables := mapForConditional at: #previousVariables.
    mapForConditional removeKey: #previousVariables.
    self producePhiVersionBasedOnPreviousVariables: previousVariables andNewVariableMap: mapForConditional.
    newVariablesMap removeKey: aConditionalBlock

The heavy lifting is in producePhiVersionBasedOnPreviousVariables:andNewVariableMap::

FASTPythonSSAVisitor >> producePhiVersionBasedOnPreviousVariables: previousVariables andNewVariableMap: newVariablesMap
    | declarationsToMerge |
    declarationsToMerge := newVariablesMap values flatten collectAsSet: #localDeclaration.

    declarationsToMerge do: [ :localDeclaration |
        | allVersions |
        "First we collect all the versions to use to produce a new Phi version
         for the current local declaration."
        allVersions := Set new.
        newVariablesMap valuesDo: [ :variables |
            variables
                detect: [ :variable | variable localDeclaration = localDeclaration ]
                ifFound: [ :variable | allVersions add: variable ] ].

        "If not all branches create a new version, we need to add the previous
         version also to the list of variables for the Phi variable. Note that
         it is possible it did not exist before."
        allVersions size = newVariablesMap size ifFalse: [
            previousVariables
                detect: [ :variable | variable localDeclaration = localDeclaration ]
                ifFound: [ :variable | allVersions add: variable ] ].

        self createPhiVersionFor: localDeclaration versions: allVersions ]

Decoding it:

declarationsToMerge — the set of distinct local declarations that received at least one new version in some branch;
for each such declaration, walk the branch map and gather the one version per branch that belongs to it;
if not every branch created a version for it, a branch may fall straight through with the old value — so we also merge in the version that was active before the conditional (from the previousVariables snapshot), if it existed;
createPhiVersionFor:versions: then decides.

FASTPythonSSAVisitor >> createPhiVersionFor: localDeclaration versions: allVersions
    "No need to add it to the versions since it was already added by the
     previous versions."
    | phiVersion |
    allVersions size < 2 ifTrue: [ ^ self ]. "No phi if we do not have at least 2 versions"

    phiVersion := FASTVariablePhiVersionSSA for: allVersions.
    allVersions anyOne mooseModel add: phiVersion.
    localDeclaration activeVersion: phiVersion.
    ^ phiVersion

Two guard rails:

if a single candidate version remains (e.g. only one branch assigns, and merging the previous version produces a single version) there is nothing to merge, so no phi is created — the single version just stays active;
otherwise a FASTVariablePhiVersionSSA is created with the candidate versions as choices, and becomes the new activeVersion of the declaration — so the reads right after the merge pick it up via the normal “copy activeVersion into ssaVersion” rule.

No need to “attach” anything else: the phi inherits from the version classes, so it knows its localDeclaration and exposes localUses, readAccesses, writeAccesses restricted to the merged versions.

Traced example

if y < 2:
    x = 4     # x_1
else:
    x = 5     # x_2

print(x)      # phi(x_1, x_2)

Walked by the visitor:

preConditionalsBranchesVisitOf: → previousVariables snapshot is empty (no version yet).
then branch: x = 4 → handleNewAssignemntTo: creates x_1, becomes active.
postConditionalsBranchVisitOf: → branch recorded with {x_1}.
else branch: x = 5 → creates x_2, becomes active.
postConditionalsBranchVisitOf: → branch recorded with {x_2}.
postConditionalsBranchesVisitOf: → declaration merged: {x_1} from branch 1, {x_2} from branch 2 → both branches assigned, so no previous version added → allVersions = {x_1, x_2} → phi created, becomes active.
print(x) is read after the merge → ssaVersion := activeVersion → phi(x_1, x_2). The use is linked to both possible values.

And the case where a branch does not assign the variable:

x = 3       # x_1
if y < 2:
    function()   # no assignment of x
else:
    x = 4   # x_2

print(x)    # phi(x_1, x_2)

Step 6 here: branch 2 introduced x_2 only, and branch 1 introduced none for x → allVersions starts as {x_2}, its size (1) is not the number of branches (2) → we add the previous version x_1 from the snapshot → allVersions = {x_1, x_2} → phi. Correct: if the first branch is taken, x still holds the old value.

If the assignment happens in a single branch and no previous version exists, then allVersions ends up with a single element → no phi, and the reads just use that single version — matching the (correct) semantics that the variable is simply that assignment in all reachable paths.

Querying the result

Once the SSA is built, the model can be queried directly:

access ssaVersion — the version of the access: a FASTVariableVersionSSA or a FASTVariablePhiVersionSSA (or nil for unresolved names and non-local declarations).
access ssaName — its pretty name, e.g. x_1, x.y_2, phi(x_1, x_2).
version nodes — the accesses directly linked to this version through ssaVersion (the bidirectional opposite).
version localUses / version readAccesses / version writeAccesses — the accesses for this specific version (in the phi case, over all its choices).
model allSSAVersions / model allResolvedVariableVersions — all the versions of the model, the latter without duplicates.
variable allSSAVersions / variable allSSABasicVersions — all versions of one variable, the latter without the phis.

(printCall arguments first) ssaVersion name.          "phi(x_1, x_2)"
(printCall arguments first) ssaVersion localUses size.

The full API is documented in the Exploiting the SSA section of analysis.md.

Known limits

The SSA inherits the limitations of the two stages it sits on:

Instance variables (self.x) cannot be treated correctly: there is no declaration of them, and we do not know the order in which methods are invoked.
Subscripts are matched by their source code, so x[a] and x[b] are conflated — and the whole “first read acts as a declaration” convention is a workaround, not a real answer.
Non-local declarations get no version: unresolved and imported names are invisible to the SSA.
Attribute access chains are not tracked (a = x.y then a.z is not linked to x.y.z).

You will find these documented in the Limitations section.

Quick start

"Import"
model := FASTPythonImporter parseFile: aFile.

"Full pipeline: local resolution is done inside"
FASTPythonSSAVisitor resolve: model allFunctionDefinitions first.

"Or explicitly, step by step"
model := FASTPythonImporter parseFile: aFile.
FASTPythonLocalResolverVisitor resolve: model module.
model allFunctionDefinitions first cfg.
FASTPythonSSAVisitor resolve: model allFunctionDefinitions first.

The SSA requires Python 3 scoping, like the local resolver it builds on.

Advice for implementing SSA in your own FAST project

Reuse the generic machinery. FASTCFGTVisitor, FASTVariableVersionSSA, FASTVariablePhiVersionSSA and the version queries live in the base FAST project (FAST-Core-Tools). You should only have to write the language-specific part: the visitX: rules for your nodes and the branch hooks.
The write/read rule is your whole core. One visit on the trait “can be a variable” (read → copy active version, write → create version) is enough, provided the local resolution already linked uses to declarations.
Drive phis from the CFG hooks, not by scanning. The four pre/postConditionals* hooks give you the branch boundaries for free. Snapshot before, record per branch, merge after — that protocol is generic.
Order your visits so reads see the old version. The metamodel order is a trap: reorder assignments (right before left), functions (parameters before body), and anything that mixes declaration and use.
Give each access a version and each declaration a “current version” attribute. A ssaVersion relation (bidirectional with nodes) and an activeVersion attribute on FASTTCanBeLocalDeclaration were enough. Make the versions Moose entities and add them to the model so they are persisted and exported.
Make the versions first-class objects, not strings. Give them a localDeclaration, name, and ssaVariables/choices; the query API (localUses, readAccesses, …) then comes almost for free on the abstract root.
Handle your “no declaration” case explicitly. In Python, unresolved/imported names simply get no version. Decide early what your language’s equivalent is (a non-local placeholder, an explicit global scope…).

The FAST-Python sources you will want to look at: FASTPythonSSAVisitor (this post is a guided read of it), the FASTCFGTVisitor trait and the FAST*VersionSSA classes in FAST-Core-Tools, and the SSA tests in FASTPythonSSATest.

Moving this implementation to FAST

This implementation currently lives in FAST-Python, but almost everything in it could move to the base FAST project in the future.

The language-specific part is surprisingly thin. Given a CFG, a local resolution, and a model whose variables use the FASTTCanBeVariable trait, this SSA implementation gives you almost everything for free:

the version classes (FASTVariableVersionSSA, FASTVariablePhiVersionSSA) and the FASTCFGTVisitor trait already live in FAST-Core-Tools;
the write/read rule is a single visit method on FASTTCanBeVariable;
the phi building runs on the generic CFG hooks.

So, for a new language, there is almost nothing more to do: mostly override the few visitX: methods where the metamodel order is wrong for SSA — assignments (right before left), functions (parameters before body), and definitions (rebuild their own CFG) — to reorder the visits.

Implementing the Local Resolver of Python in FAST-Python

Wed, 16 Sep 2026 00:00:00 GMT

Implementing the Local Resolver of Python in FAST-Python

The goal of this blog post is to explain how I implemented a local resolver for FAST-Python. FAST-Python is on of the hardest project to implement this kind of algo, so reading this should help implementing any other local resolver for other languages.

What is local resolution?

Local resolution is a symbol resolution pass: it links each named entity to the entity that declares it. In other words, for every occurrence of a name — read, write, call, import — it answers the question “where does this name come from?”.

In FAST-Python, after the model is imported from source, running the local resolver produces a model where every name usage points to its local declaration:

x = 1        # this is the declaration of `x`
print(x)     # this use of `x` is linked to the declaration above

It is called local because it only resolves declarations that live in the same file/model (in contrast to a global/system resolution that would also resolve calls to external libraries, standard modules, etc.). Once we went through the file, a name that is not declared anywhere in the file gets bound to a special FASTNonLocalDeclaration placeholder — it is a declaration for the model, but we know it refers to something outside.

Local resolution is the foundation on top of which our other analyses are built such as the Static Single Assignment (SSA). Without it, you cannot even tell whether two x in the code are the same variable. You can’t even say if a FASTPyIdentifier is a variable or not.

The API

The result of the resolution is exposed through two accessors available on every FAST entity (they live in the FAST-Core-Tools package of the base FAST project, so they are reusable across languages):

entity localDeclaration — from any use of a name, returns the entity that declares it (the first assignment, a FASTPyFunctionDefinition, a loop variable, an import…). If the name was not declared in the file, it returns a FASTNonLocalDeclaration.
declaration localUses — from a declaration, returns all the entities that resolve to it (usages and other declarations that share it).

This is a bidirectional relationship: localUses is the exact inverse information of localDeclaration.

On top of those two, FAST-Python adds a set of convenience queries that only make sense once the resolution is done — and only for nodes that represent variables:

access allAccesses / allReadAccesses / allWriteAccesses — all the read/write accesses to the same variable.
access internalAccesses — the attribute accesses and subscripts on the variable (x.y, x[3]).
access allNodesUsingMe / access allStatementsUsingMe — the statements that use the variable.
variable isResolvedVariable — true if the node resolves to a local variable declaration (and is not a function, method, import or unresolved name).
node usedVariables — all the entities in the subtree of the node that resolve to local variables.

The full list is documented in the analysis.md “Querying local resolver information” section.

Implementation: a Moose visitor

The resolver is implemented as a Moose/FAST visitor: FASTPythonLocalResolverVisitor. It visits a FAST model (a FASTPythonVisitor built on the generated visitor trait FASTPyTVisitor) and inspects/annotates the nodes in place.

If you are not familiar with how the visitor is generated from the model, I strongly recommend reading Improving the Visitor Generator first: it explains how FASTPyTVisitor is produced and how you can override the generated visitX: methods.

The Python problem: there is no declaration of variables

Before we can write a single line of the algorithm, we have to deal with the first (and deepest) Python quirk: there is no declaration of variables.

In Java you get a real declaration:

int x = 1;   // I declare and type `x`

In Python, an assignment is the declaration:

x = 1        # this IS the declaration of `x`

There is no keyword, no type, no way to distinguish “declare a fresh variable” from “reassign an existing one” syntactically — the same syntax x = ... does both. The resolver has to decide, and the decision is purely lexical reasoning about scopes and order, with no syntactic marker to lean on. Concretely:

The first assignment in a scope is the declaration. The resolver must therefore know the order in which things appear, and which construct “creates” a scope.
Parameters are declarations: def f(x): ... declares x as the function’s parameter. They are semantically the first assignment in the function’s scope.
Walrus operators (x := ...) declare x inline, even inside an expression or an if condition.
A for ... in ... loop declares its target variable — but only the first time; it must be a declaration when the variable is not already known in the scope.
Augmented assignments (x += 1) are uses + writes to the same variable.
For clauses in a list comprehension.

In our implementation this all boils down to a single rule expressed in visitFASTTCanBeVariable: (see below): a write access declares and bind, a read access binds. Everything else is scope management around that rule.

A scope stack

The whole state of the algorithm is a single instance variable — and the class itself is tiny. Here is the complete class declaration:

FASTPythonVisitor << #FASTPythonLocalResolverVisitor
  slots: { #namesContext };
  tag: 'CFG/LocalResolver/SSA';
  package: 'FAST-Python-Tools'

The superclass is FASTPythonVisitor (from the generated FASTPyTVisitor trait), so all the visitX: methods come for free: we only override the ones where the metamodel visit order differs from the resolution order.

namesContext is a stack of scopes (a Stack of Dictionarys, mapping name -> declaration). It is initialized in initialize:

initialize
    super initialize.
    namesContext := Stack new

The class entry point is trivial:

FASTPythonLocalResolverVisitor class >> resolve: aModule
    ^ self new resolve: aModule

and the instance resolve: wraps everything in one scope and runs the visitor:

resolve: aFASTBehaviouralEntity
    self useNewScopeDuring: [
        "flush attributes in case this is not the first resolution"
        aFASTBehaviouralEntity withAllContainedEntities do: [ :entity | entity resetLocalResolution ].
        aFASTBehaviouralEntity accept: self ] "<== Launch the resolution on the tree"

Scopes are pushed/popped around the constructs that define a new scope (module, function, comprehension), through useNewScopeDuring::

useNewScopeDuring: aBlock
    namesContext push: Dictionary new.
    [ aBlock value ] ensure: [ namesContext pop ]

The core operations

The algorithm is three small operations combined:

get — look a name up in the stack of scopes (from top to bottom):

declarationNamed: aName
    namesContext do: [ :scope |
        scope at: aName ifPresent: [ :declaration | ^ declaration ] ].
    ^ nil

set / ensure a declaration — put a declaration in the current scope, handling re-declaration and shadowing (see Shadowing):

ensureDeclarationOf: aName declaration: aFASTNode in: scope
    scope
        at: aName
        ifPresent: [ :declaration |
            declaration localResolverKind = aFASTNode localResolverKind
                ifTrue:  [ ^ declaration ]
                ifFalse: [ declaration shadowedBy: aFASTNode.
                           aFASTNode ensureLocalUses.
                           scope at: aName put: aFASTNode ] ]
        ifAbsentPut: [ aFASTNode ensureLocalUses; yourself ]

bind — link a use to the existing declaration (creating a FASTNonLocalDeclaration if none is found):

bind: aFASTNode toDeclarationNamed: aName
    ^ self bind: aFASTNode toDeclarationNamed: aName
              ifAbsentUse: [ self ensureNonlocalDeclarationNamed: aName ]

bind: aFASTNode toDeclarationNamed: aName ifAbsentUse: aBlock
    | declaration |
    declaration := (self declarationNamed: aName) ifNil: [ aBlock cull: aFASTNode cull: aName ].
    aFASTNode localDeclaration: declaration.
    declaration addLocalUse: aFASTNode

What happens when no declaration is found

When a read has no matching declaration in any scope, bind:...ifAbsentUse: falls back to ensureNonlocalDeclarationNamed::

ensureNonlocalDeclarationNamed: aName
    ^ self
          ensureDeclarationOf: aName
          declaration: (FASTNonLocalDeclaration new
                   name: aName;
                   yourself)
          in: namesContext first

Note the scope: namesContext first is the bottom of the stack, so a FASTNonLocalDeclaration behaves as if it were declared at module level — reads of an undeclared name in a function and at module level end up in the same place. Also note that the fallback for attribute accesses and subscripts is slightly different (see the heart): declarationForUndeclaredNode:named: creates a local declaration on the first read of a subscript when the receiving variable has a declaration, and only resorts to the non-local declaration otherwise:

declarationForUndeclaredNode: aNode named: aName
    "In case of a subscript, if the receiving variable exists, we consider
     that the first read access to the subscript is the declaration."
    aNode isSubscript ifTrue: [
        aNode value
            localDeclarationifPresent: [ :decl |
                ^ self ensureDeclarationOf: aName
                                declaration: aNode
                                in: namesContext top ]
            ifAbsent: [ "Nothing, just let the non local declaration." ] ].
    ^ self ensureNonlocalDeclarationNamed: aName

The heart: `visitFASTTCanBeVariable:`

Everything that can be a variable in FAST-Python implements the trait FASTTCanBeVariable (identifiers, attribute accesses, subscripts, walrus…). The visitor intercepts them in one place, and this is where “write declares / read binds” lives:

visitFASTTCanBeVariable: aTCanBeVariable
    aTCanBeVariable isVariableWriteAccess
        ifTrue:  [ self ensureDeclarationOf: aTCanBeVariable
                            named: aTCanBeVariable localDeclarationName
                            declaration: aTCanBeVariable ]
        ifFalse: [ self bind: aTCanBeVariable
                            toDeclarationNamed: aTCanBeVariable localDeclarationName
                            ifAbsentUse: [ :node :name |
                                self declarationForUndeclaredNode: node named: name ] ].

    super visitFASTTCanBeVariable: aTCanBeVariable

Two details make Python specific here:

localDeclarationName is the name that will be used for scope lookup. It is not always the identifier text: an attribute access and a subscript use their source code as their name, so x.y and x[3] are treated as first-class “variables” (with known limits, see Limitations). Every named entity gets a default from the base FAST project (FASTTNamedEntity>>localDeclarationName returns #name); the Python classes only override it where the name is not the identifier:

"from FAST, the default for any named entity:"
FASTTNamedEntity >> localDeclarationName [ ^ self name ]

"Python overrides:"
FASTPyAttributeAccess >> localDeclarationName [ ^ self sourceCode ]
FASTPySubscript       >> localDeclarationName [ ^ self sourceCode ]

isVariableWriteAccess decides whether a node is a write access or a read access. It comes from FAST (FASTTEntity>>isVariableWriteAccess), but it delegates to variableDeclaration which is an explicitRequirement — every FAST project must implement it for the node kinds that can be variable write accesses. In FAST-Python, each relevant class provides its own logic to walk up the AST and check whether it sits in a write position:

"FAST core — the API:"
FASTTEntity >> isVariableWriteAccess [
    ^ self variableDeclaration isNotNil
]

FASTTEntity >> variableDeclaration [
    "If I am a node representing a write access, I return the node
     assigning me. Else I return nil."
    ^ self explicitRequirement
]

"FAST-Python — an identifier checks whether it is the left side of
 an assignment, a for-loop target, etc.:"
FASTPyIdentifier >> variableDeclaration [
    | assignedNode |
    assignedNode := self selfOrTopmostAssignableCollection.
    assignedNode parentAssignmentLeft ifNotNil: [ :assignment | ^ assignment ].
    assignedNode parentForStatementLeft ifNotNil: [ :for | ^ for ].
    assignedNode parentForInClauseLeft ifNotNil: [ :clause | ^ clause ].
    ^ super variableDeclaration
]

"Parameters and walrus are their own declarator:"
FASTPyParameter >> variableDeclaration [ ^ self ]
FASTPyWalrus    >> variableDeclaration [ ^ self ]

"Attribute accesses and subscripts check for parentAssignmentLeft:"
FASTPyAttributeAccess >> variableDeclaration [
    self selfOrTopmostAssignableCollection parentAssignmentLeft
        ifNotNil: [ :assignment | ^ assignment ].
    ^ super variableDeclaration
]

"The default on FASTPyEntity returns nil (not a write access):"
FASTPyEntity >> variableDeclaration [ ^ nil ]

The interesting bit: for identifiers, variableDeclaration does not just check the immediate parent — it first walks through tuple/destructuring parents via selfOrTopmostAssignableCollection, so that in (a, b) = (1, 2) the individual a and b correctly return the tuple assignment as their declaration.

Reordering the visit, or dealing with Python scoping

Because the visitor is generated (see the visitor generator blog post), it visits the children of a node in the metamodel order. For a resolver, metamodel order is not always resolution order: a construct may declare a name in a part of the syntax that the generated visitor reaches too late (or too early) relative to its uses.

Our main tool is therefore: override the generated visitX: with a manual ordering of a few visitEntity:/visitCollection: calls — and be very careful about when to open a scope, because Python does not create a scope where you would expect one.

Block scoping is inconsistent: `if` and `while`

In most block-structured languages you expect:

if (cond) { let y = 3; }
use(y);   // Compile error: y does not exist

Python does not create a scope for blocks. An if, a while, a for body do not introduce a new scope:

if cond:
    y = 3
print(y)      # 3, perfectly valid Python

Combine that with “assignment is the declaration” and you get leaks: a variable assigned inside an if branch or a loop is visible after the block. For the resolver this means no useNewScopeDuring: around branches — a name assigned in the then must be visible in the else and after the whole statement. What we do control is the order, which is important for shadowing: in an if, the semantics are

the condition (can use outer variables, and walrus operators can even declare variables!),
the then clause,
the elif clauses (in order),
the else clause.

visitFASTPyIfStatement: anIfStatement
    "visit then - elif(s) -> else"
    self visitFASTTConditionalStatement: anIfStatement.
    self visitFASTPyStatement: anIfStatement.
    self visitEntity: anIfStatement thenClause.
    self visitCollection: anIfStatement elifClauses.
    self visitFASTPyTWithElseClause: anIfStatement

Note that we do not call super here: the point is precisely to produce our own child order instead of the metamodel one.

while statements are similar, but the else clause has to go last (it runs when the loop condition becomes false):

visitFASTPyWhileStatement: aWhileStatement
    self visitFASTTConditionalStatement: aWhileStatement.
    self visitFASTTStatementBlock: aWhileStatement.
    self visitFASTPyStatement: aWhileStatement.
    self visitFASTPyTWithElseClause: aWhileStatement

`for` loops leak their variable

for i in range(10):
    pass
print(i)      # 9 — the loop variable is still there!

The for ... in ... statement declares its loop variable in the enclosing scope (there is no loop scope). i survives after the loop. So a for is simultaneously a scoping and a declaring construct: the resolver must declare the target (left) in the current scope — no push — and, since that is the declaration, visit it before the iterable and the body:

visitFASTPyForStatement: aForStatement
    self visitEntity: aForStatement left.
    self visitEntity: aForStatement right.
    self visitFASTPyTWithElseClause: aForStatement.
    self visitFASTTStatementBlock: aForStatement.
    self visitFASTPyStatement: aForStatement

Function and method definitions: parameters are declarations

The generated visitor does not visit the parameters before the body — but the parameters are declarations that the body uses. So visitFASTPyFunctionDefinition: manually visits parameters first, wrapped in a new scope, and does not use super:

visitFASTPyFunctionDefinition: aFunctionDefinition
    self ensureDeclarationOf: aFunctionDefinition
        named: aFunctionDefinition name
        declaration: aFunctionDefinition.
    self useNewScopeDuring: [ "do not use super: parameters must be visited before the body"
        self visitFASTTWithParameters: aFunctionDefinition.
        self visitFASTPyTWithTypeParameters: aFunctionDefinition.
        self visitFASTPyStatement: aFunctionDefinition.
        self visitEntity: aFunctionDefinition returnType.
        self visitFASTPyTDefinition: aFunctionDefinition ]

visitFASTPyMethodDefinition: simply delegates to the function one.

Comprehensions: a scope that leaks itself, but not its body

Comprehensions are Python 3’s attempt at “introduce an expression-level scope”, and they only partially succeed:

[x for x in coll]
print(x)      # NameError in Python 3 — x does not leak OUT of the comprehension

In Python 3, the comprehension has its own scope, so the loop variable of its for clause does not escape. But the for clause(s) and the if condition(s) can still see and use variables from the enclosing scope, and the comprehension’s own variable is in scope through the whole comprehension (the body and the conditions can refer to the for clause variables).

That gives you a scope that is neither fully lexical like a function, nor absent like a block. In our implementation, visitFASTPyComprehension: opens a dedicated scope and reorders the visit so that the for clauses are processed before the conditions and the body (otherwise the conditions/body would be visited against the wrong scope):

visitFASTPyComprehension: aComprehension
    self useNewScopeDuring: [
        self visitFASTPyTSplatExpression: aComprehension.
        self visitFASTPyExpression: aComprehension.
        self visitCollection: aComprehension forClauses.
        self visitCollection: aComprehension conditions.
        self visitEntity: aComprehension body ]

We also had to choose a Python version:

Other constructs that declare names

Imports declare the imported name (or its alias) in the current scope — visitFASTPyImport: ensures a declaration for each imported entity (using alias if present, source code otherwise).
Walrus operator (x := ...) declares x — visitFASTPyWalrus: ensures its declaration (it can even appear in an if condition).

`global` and `nonlocal` statements

Python is one of the rare languages where a function can explicitly opt out of local scoping:

x = 1

def f():
    global x      # `x` is the module-level x, not a local one
    x = 2

def g():
    y = 1
    def h():
        nonlocal y    # `y` is the `y` of `g`, not a new local
        y = 2

These two statements redirect resolution away from the current scope:

a global x means: “in this scope, x is the module-scope x”. Writes here target the global declaration, they must not create a new local declaration.
a nonlocal x means: “x is the variable of the nearest enclosing function that defines it”. It is similar to global but with a different target scope.

In the model, a FASTPyGlobalStatement (resp. FASTPyNonlocalStatement) contains a collection of variables, and each FASTPyVariable keeps the inverse parentGlobalStatement / parentNonlocalStatement pointer. That is what the visitor checks in FASTPythonLocalResolverVisitor>>#visitFASTPyVariable: — when visiting a variable that is the target of one of these statements, we change which scope the write will land in:

visitFASTPyVariable: aVariable
    "We handle two specific cases here.
     - global: the variables impacted should act as if they were in the
       global scope. I ensure the variable is in the bottom scope and add
       a copy in the current scope (so that if we assign it, it goes in
       the global and does not create a new local).
     - nonlocal: the variables impacted should act as if they were in the
       first parent scope defining the variable."
    aVariable parentGlobalStatement ifNotNil: [
        namesContext top
            at: aVariable localDeclarationName
            ifAbsentPut: [ self ensureDeclarationOf: aVariable localDeclarationName declaration: aVariable in: namesContext last ] ].

    aVariable parentNonlocalStatement ifNotNil: [
        namesContext allButFirst
            detect: [ :scope | scope includesKey: aVariable localDeclarationName ]
            ifFound: [ :scope | namesContext top at: aVariable localDeclarationName put: (scope at: aVariable localDeclarationName) ]
            ifNone: [ self error: 'Non local statement points a variable that was never defined.' ] ].

    super visitFASTPyVariable: aVariable

Let’s unpack the two branches:

global — we want a write x = 2 inside the function to target the module-level x. So we look at the bottom scope (namesContext last, the module scope) and, if x is not there yet, we declare it there with the current writing variable as its declaration. Then we copy a reference into the top scope (namesContext top). From then on, any write in this scope simply finds x present in the top scope — ensureDeclarationOf: sees the same kind (#variable = #variable) and keeps the same declaration, so no new local is created and the write silently targets the global one. The copy trick makes the subsequent writes “resolve” without us having to special-case every write site.
nonlocal — the target scope is not the module but the nearest enclosing function that already defines the name. So we scan namesContext allButFirst (everything except the bottom scope) and re-bind x in the top scope to the existing declaration found there. If no enclosing scope defines the name, this is a Python error (SyntaxError at compile time in real Python), so we raise an error too.

Both branches rely on a nice property of our data structure: the top entry of the stack is always the current scope, and putting a reference to an existing declaration (rather than a fresh node) in the current scope is exactly what redirects future uses.

Shadowing in Python

Shadowing is another direct consequence of “no declarations”: in Python, anything named can shadow anything named, in the same scope:

x = 1              # Declaration 1: variable
from os import x   # Declaration 2: import (shadows Declaration 1)
def x():           # Declaration 3: function (shadows Declaration 2)
    pass
print(x)           # links to Declaration 3

The rule we implemented

In ensureDeclarationOf: (see core operations), when a name is re-declared in the same scope, the resolver does not decide based on types (there are none) but on localResolverKind:

if the new entity has the same kind (e.g. re-assigning a variable): we keep the same declaration and do nothing special. Two x = ... in a row share one declaration. (This is why ensureLocalUses initializes the uses instead of resetting them: several nodes can be declarations of one shared declaration.)
if the kind is different (a variable shadowed by an import or a function): we create a new declaration, and link the two declarations together.

localResolverKind is not used anywhere else in the resolver — it exists only to drive this decision. Each node kind that can act as a declaration returns its kind, and the base FAST project provides no default because it is purely language-specific:

FASTPyIdentifier          >> localResolverKind [ ^ #variable ]
FASTPyParameter           >> localResolverKind [ ^ #variable ]   "can be reassigned"
FASTPyWalrus              >> localResolverKind [ ^ #variable ]
FASTPyAttributeAccess     >> localResolverKind [ ^ #variable ]
FASTPySubscript           >> localResolverKind [ ^ #subscript ]
FASTPyFunctionDefinition  >> localResolverKind [ ^ #function ]
FASTPyMethodDefinition    >> localResolverKind [ ^ #method ]
FASTPyImport              >> localResolverKind [ ^ #import ]

Two subtleties we hit:

Parameters are #variable, not a dedicated kind, because they can be reassigned inside the body: def f(x): x = 2 must share the declaration with the parameter, not shadow it.
FASTPyEntity>>localResolverKind raises an error by default, which catches any node kind that is used as a declaration without having declared its kind — a cheap safety net while extending the metamodel.

The shadowing API

To make shadowing queryable, two back-links were added on the declarations (FASTPyEntity, in FAST-Python-Model, generated via the metamodel generator):

declaration shadowing — returns the declaration it shadows, or nil.
declaration shadowedBy — returns the declaration that shadows it, or nil.

Together they form a linked list from the first declaration to the most recent one. Each declaration keeps its own localUses (the set of entities that resolve to it, not to its successors), and usages always resolve to the most recent declaration.

Going back to the example:

varDecl := model module statements first left.      "FASTPyVariable"
importStmt := model module statements second.       "FASTPyImportFromStatement"
funcDecl := model module statements third.          "FASTPyFunctionDefinition"

varDecl shadowedBy.        "=> FASTPyImportFromStatement"
importStmt shadowedBy.     "=> FASTPyFunctionDefinition"
importStmt shadowing.      "=> FASTPyVariable"
funcDecl shadowing.        "=> FASTPyImportFromStatement"

varDecl localUses size.     "1  (just the assignment)"
importStmt localUses size.  "1  (just the import)"
funcDecl localUses size.    "2  (the definition + print(x))"

A caution about “same kind”

Same-kind re-declaration shares the declaration, which also means shadowing’s chain and local uses are independent of the SSA versioning: if you need to know which assignment impacts a particular use, local resolution is not enough — combine it with the SSA pass (it produces one version per assignment). The two analyses are designed to be composed in this pipeline order: Local resolution → CFG → SSA.

Known limits and what is left to do

We implemented local resolution for Python, but we focused on variables first, because that is what the CFG/SSA and the reachability analyses needed. Functions, methods and imports are handled in the scope bookkeeping (localResolverKind, reordering in the visitor), but if your mission is a complete Python symbol table, expect more work there (call graph resolution, self/class attributes, closures and bound variables…).

Specific known weaknesses (documented in analysis.md → Limitations):

Attribute access chains: x.y.z = 3; a = x.y; print(a.z) — a.z should resolve to x.y.z, but it does not (only the source code of the attribute access is used as the name).
Subscripts are compared by source code: y[x] with different x values are conflated; x[0:4] and x[:4] (semantically equal) are seen as different. Matching the expression instead of the string would fix it.
Instance variables (self.x) cannot be handled correctly without knowing the order in which methods are invoked.
Python 2 scoping is not supported (comprehension variables leak in Python 2; we implement Python 3).
global/nonlocal are handled, but nonlocal errors out if the variable was never defined in an enclosing scope.

Also, one of the future step zould be to make some parts, such as the context stack, generic and push it to FAST so that it can be reused in other FAST projects.

Quick start

"Import"
model := FASTPythonImporter parseFile: aFile.

"Resolve"
FASTPythonLocalResolverVisitor resolve: model module.

"Query"
(model allFunctionDefinitions first) localDeclaration.  "a FASTPyFunctionDefinition"

The recommended pipeline for analysis:

model := FASTPythonImporter parseFile: aFile.
FASTPythonLocalResolverVisitor resolve: model module.
model allFunctionDefinitions first cfg.                          "CFG"
FASTPythonSSAVisitor resolve: model allFunctionDefinitions first. "SSA (after resolution)"

The local resolver and SSA require Python 3 scoping.

Advice for implementing it in your own FAST project

Use the generated visitor, override visitX: selectively. You rarely need to reorder everything — only where the metamodel order differs from the declaration-before-use order (if, for, while, functions, comprehensions).
Model a scope stack explicitly. One Stack of Dictionarys was enough for the whole algorithm. Wrap “new scope” sites in a useNewScopeDuring:/[ensure: pop] pair so the stack is popped even on error.
Make “what declares a name” explicit and language-aware. For Python: write access declares; read access binds; for target declares in the enclosing scope; comprehensions open a scope; global/nonlocal redirect.
Add a localDeclarationName per node kind (it is sourceCode for attribute accesses/subscripts, the identifier name otherwise) — do not hard-code “the name is the text” everywhere.
Add a localResolverKind and use it to drive shadowing. It made the Java-free, type-free Python shadowing tractable and gave us a cheap way to keep-or-split declarations.
Reset your attributes before re-resolving. We flush localDeclaration/localUses on all contained entities at the start of resolve: so the resolver is idempotent on a model.

The FAST-Python sources you will want to look at: FASTPythonLocalResolverVisitor, the localResolverKind/localDeclarationName extensions in FAST-Python-Tools.

Mixing up analysis of Project Architecture (Famix) and its history (Mining Software Repository)

Thu, 27 Aug 2026 00:00:00 GMT

Famix is the meta-model family of Moose allowing one to analyse the dependencies of a project and its global architecture. But sometimes, you want more, you want to see what are the part of the code that are modified most, are critical, etc. It is for instance what people do when using Sonar, Coverage analysis, and so on.

To explore the history of existing projects, Mining Software Repositories (MSR) is the way to go. Moose also includes a set of tools to explore the history of a project, and to combine it with the Famix analysis. The project is named GitProjectHealth, and the full documentation is available here.

But let’s today perform an example. I have this nice HR application that I want to investigate. It is large, and I want to retrieve information between the architecture and the history of the project.

So let’s do a step by step blog about this story.

Creating the Famix model

I am working with Moose12. But this story should last for future version (I mean, I don’t know how long, but it should not change a lot) First, I clone the project, and then, I install its dependency with maven mvn install.

Then, I build a nice model using the docker version of VerveineJ.

docker run -e JAVA_XMX=-Xmx16G -v /path/to/hr:/src -v /path/to/.m2/repository:/dependency ghcr.io/evref-bl/verveinej:v4.2.6 -anchor assoc -alllocals -format json

I use the version 4.2.6 of VerveineJ, because I am with Moose12, but don’t hesitate to use the latest version of VerveineJ if you are with a more recent version of Moose.

Then, I load the model in Moose, and I can start to explore the architecture of the project. To load the model, I simply drag and drop the json file in Moose, and then I can start to explore the model.

Retrieving the history of the project

To retrieve the history of the project, I use the GitProjectHealth tool. This project is not included in the Moose distribution by default, but you can download it by executing the following command in a Moose playground:

Metacello new
  repository: 'github://moosetechnology/GitProjectHealth:main/src';
  baseline: 'GitLabHealth';
  onConflict: [ :ex | ex useIncoming ];
  onUpgrade: [ :ex | ex useIncoming ];
  onDowngrade: [ :ex | ex useLoaded ];
  load

Then, because it is a large HR application, I will perform the retrieving of the history locally using git.

rhModel := GLHModel new.
repository := GLHRepository new
  cacheAt: #localImporterReference
  put: '/path/to/hr' asFileReference;
  yourself.
project := GLHProject new.
project repository: repository.

rhModel add: repository.
rhModel add: project.

localImporter := GitLocalModelImporter new.
localImporter withFiles: true.
localImporter withCommitDiffs: true.
localImporter withDiffRanges: false.
localImporter glhModel: rhModel.
localImporter withCommitsSince: 30 years.
localImporter importRepository: repository.

rhModel flush.
rhModel entities do: #flush.

You’ll notice several things in the code above.

I created a fake project and repository to store the history of the project. This is to help the local importer and might be improved.
I use the withFiles option to retrieve the files of the project, and the withCommitDiffs option to retrieve the diffs of the commits. The withDiffRanges option is set to false because I don’t need it for now.
I use the withCommitsSince: 30 years option to retrieve all the commits of the project. You can change this value to retrieve only the last commits if you want.
I use the flush method to remove cache that is computed by the local importer, this is for storage efficiency.

Once it is done, I have the full history of the project, but not connected to the exsiting model of the code.

Connecting the models

We want now to connect the GitProjectHealth model, with the Famix model. There is a tool dedicated for this usage in GitProjectHealth, it is called GPCGitToFamixConnector.

The project links the types of the Famix model with the files retrieved by the GitProjectHealth model based on the files path. To use it, simply execute the following code:

GPCGitToFamixConnector new
  famixModel: myFamixModel;
  glhProject: project;
  connect.

it is possible that GPCGitToFamixConnector is not loaded in your Moose image. It is is the case, load the missing package from Iceberg, or check in the baseline of GitProjectHealth if it is included in the version you are using, there should be a group named “Famix”

And, that’s it… yeah, it is quite easy to connect the two models, and now you can explore the history of the project in the context of its architecture.

You want to search for the types the most modified? One classic Moose query will do the job:

(myFamixModel allUsing: FamixTType) sorted: [ :type | type appliedDiffs size ] descending

Building an ownership cartography of a system - Part 2

Thu, 13 Aug 2026 00:00:00 GMT

When you share your work with others, such as the previous blog post, you can get feedback and suggestions for improvement. And this is exactly what happened.

And by Distribution Map, we are talking about a research paper accepted at ICSM 2006 So, let’s not reinvent the wheel anymore and look at how to use the already existing Distribution Map in Moose to visualize the ownership of a system.

One color for each developer

The first visualisation that comes to mind is to use one color for each developer. Using the Degree-of-Authorship (DOA) formula, we can compute the ownership of each file and retrieve the main developer for each file.

In moose, we can then create a tag that will associate a color to each developer and then use this tag to color the files in the system. We assign a tag to each file that corresponds to the main developer of the file.

((glhModel allWithSubTypesOf: GLHFile)
    select: [ :file | file diffs size ~= 0 ])
  do: [ :file |  file tagWithName: ((file degreeOfAuthorship) keyAtValue: 1.0) name  ].

We first search for all the files that have been modified, then we compute the degreeofAuthorship for every possible user, we select the one with the best score, which is 1, and we create a tag with the author name. Once the tag is created for every file, it is possible to propagate the files to the Distribution Map.

Propagate to Distribution Map

To propagate the files to the distribution, we first retrieve the files in our model. Then, we get the parent folder of the files.

appsFolder := (glhModel allWithSubTypesOf: GLHFileDirectory) detect: [ :fi | fi path = 'src' ].
files := (appsFolder allToScope: GLHFileBlob).
"Find parent folder"
folders := (files collect: [ :file | file directoryOwner ] as: Set).
folders

Based on the parent folders, the Moose Distribution Map will be able to recalculate the children, and so the files of interest. We select the folders variable and propagate with Cmd + Shift + M, Cmd + Shift + P.

Then, we open the Distribution map from the Moose Menu.

We first get a visualisation without any color. So, in the settings of the visualisation, we select the tags to display. We select every authors and get the last visualisation.

Again, the visualisation well presents the different packages and the one with major author. The gray square corresponds to file less modified in the system the last two years.

Building an ownership cartography of a system

Tue, 11 Aug 2026 00:00:00 GMT

Context

When managing a software system, it is important to know who is responsible for which part of the system. This information can be useful for various reasons, such as identifying potential bottlenecks, understanding the impact of changes, and improving collaboration among team members.

One major threat could, for example, come from a developer leaving the team. This is often referred to as the “bus factor” or “truck factor”. But how to detect such a situation?

Research have shown that the ownership of a system can be inferred from its version control history. To do so, one approach is to first investigate the ownership of each file.

Two nice papers, not from the Moose community, are:

Computing the ownership of a file

Looking at the research paper, we can find a formula that can help build a first version of ownership. It consists for each file of a git repository to compute a degree of authorship for each potential author using this formula.

Given a file $f$ with path $f_p$ , the degree-of-authorship of a developer $d$ whose Git user has been mapped to $m_d$ is given by.

DOA(m_d, f_p) = 3.293 + 1.098 × FA(m_d, f_p) + 0.164× DL(m_d, f_p)−0.321 ×ln(1 + AC (m_d, f_p))

With the following definitions:

DOA depends on three factors: (i) first authorship ( $FA$ ): if $m_d$ originally created $f$ , $FA$ is 1; otherwise it is 0; (ii) number of deliveries ( $DL$ ): number of changes in $f$ made by $m_d$ ; and (iii) number of acceptances ( $AC$ ): number of changes in $f$ made by any developer, except $m_d$ .

Looking at the source code of the Truck-Factor tool, we can see thet $DL$ corresponds to a commit that modifies a file. The file can be added, modified, and renamed.

$AC (m_d, f_p)$ is computed by the total number of modifications of a file minus the number of modifications made by the developer $m_d$ .

Moose Implementation

Moose comes with the GitProject Health project to represent a Git repository and analyze its history. So, I wanted to implement the DOA formula in Moose.

Making it faster

One concern is that GitProject Health is firstly made to analyze Git Social Platform and not only git repository with the truck factor in mind. Thus, it is not optimized for this use case.

The major concern comes when loading the model that rely on several calls to the Git Platform API. In order to avoid this, I decided to implement a companion importer named GitLocalModelImporter that can load a model from a local git repository.

To go faster, it relies on the FFI LibGit2. This project includes a LGitRevwalk, a walker that can iterate over the commits of a git repository. Thus, we can iterate over the commits and retrieve the files, the diffs, the commits, and the authors in a repository.

This version is still not optimized to compute truck factor, but it can be used to do lot more than just computing truck factor. It comes as a new classic importer for GitProject Health and so can be used side by side with other importers for optimization.

Implementing the DOA formula

To implement the DOA formula, I created a simple but effective class named GLPDegreeOfAuthorshipAnalyzer. This class is now part of the GitProject Health project set of metrics. It reuses the exact same algorithm as the Truck-Factor tool to compute the degree of authorship for each file in a git repository.

We note several additionnal considerations to compute the DOA formula. These considerations are mention because they enable analysis of a project without considering the history of the project, which is long to consider.

The first authorship ( $FA$ ) is computed by looking at the first commit that added or modified a file to the repository.
GitProject Health does not consider renaming of files yet, so it does not consider file renaming.

Even though not considering rename can create issues, this work has been performed on an industrial project and considering renaming would not drastically change the results of the analysis.

Vizualizing the ownership of a system

The last step I wanted to investigate is to visualize the ownership of a system. I did not create a vizualization dedicated to this point yet. But let’s investigate a running example for the pharo project.

After cloning the project, I create a Moose image with the GitProject Health project installed. Then, I load the pharo project.

glhModel := GLHModel new.
repository := GLHRepository new
    cacheAt: #localImporterReference
    put: '/path/to/MSR/pharo' asFileReference;
    yourself.

glhModel add: repository.

localImporter := GitLocalModelImporter new.
localImporter withFiles: true.
localImporter glhModel: glhModel.
"I will import the last two years"
localImporter withCommitsSince: 2 years.
localImporter importRepository: repository.

After a possibly long execution, we get the repository loaded. It is thus possible to investigate it using our tools.

We want to build a cartography of the application to identify the coverage of files with an author. To do so, I decided to build a TreeMap using the Roassal framework.

First, I need to load the Roassal TreeMap support

Metacello new
    baseline: 'Roassal';
    repository: 'github://pharo-graphics/Roassal:Pharo13';
    onConflictUseIncoming;
    load: #( Full )

Then, start the fun.

A quick view of the project

We build a dictionary with the degree of authorship for everyfile.

filesAuthorShip := ((glhModel allWithSubTypesOf: GLHFile)
    select: [ :file | file diffs size ~= 0 ])
    collect: [ :file |
        "This uses our analyser behind the scene"
        file -> file degreeOfAuthorshipAuthors ]
    as: Dictionary.

We look for the folder that is the root of the tree map, and we compute the max number of authorship that will be used for coloring the vizualisation.

appsFolder := (glhModel allWithSubTypesOf: GLHFileDirectory)
    detect: [ :fi | fi path = 'src' ].

max := (filesAuthorShip collect: #sum) max.

Finally, we build the tree map.

palette := RSQualitativeColorPalette qualitative set39.
b := RSTreeMap new.
b inset: 2 asPoint.
b boxShape borderColor: Color black.
b boxShape borderWidth: 0.05.

color := NSScale linear domain: { 0. max };
    range: #( white red ).
b
    leafWeight: [ :f | 1 ];
    explore: appsFolder nesting: [ :directory |
        directory files select: [ :f | (f isKindOf: GLHFileDirectory) ] ]
    leaves: [ :directory | directory files reject: [ :f | f isKindOf: GLHFileDirectory ] ].

b shapes
    do: [ :box |
        box borderWidth: 1.
        box isSLeaf
            ifTrue: [ box color: (color scale: (filesAuthorShip at: box model ifPresent: [ :dic | dic sum ] ifAbsent: [ 0 ])) ]
            ifFalse: [
                | path |
                path := (box model path splitOn: '/').
                path size > 1 ifTrue: [
                    box color: (palette scale: (box model path splitOn: '/') second) ] ] ].

b shapes @ (RSPopup text: [:f | f path, String crlf, ((filesAuthorShip at: f ifPresent: [ :dic | dic sum ] ifAbsent: [ 0 ]) printString) ]).

b build

Using this script, we built a vizualisation with different color for each package, and a red scale for know packages. It ouput the following image

Looking at the image we see that some package have author from the 2 last year modification whereas other were not modified at all.

More than showing the Truck Factor, this vizualization highlights the code that have been updated by people that gain or conserve knowledge on the codebase.

One improvment could be to highlight the core contributors.

highlighting the core contributors

To highlight the core contributors, we first create a color palette for them.

coreContributors := (filesAuthorShip values flatCollect: #keys) asSet.
paletteContributor := RSQualitativeColorPalette qualitative flatui120.

Then, we update the first b shapes statements by:

b shapes
    do: [ :box |
        box borderWidth: 1.
        box isSLeaf
        ifTrue: [
            filesAuthorShip at: box model ifPresent: [ :dic |
                | author |
                author := (dic associations sorted: [ :a :b | a value > b value ]) first key.
                box color: (paletteContributor scale: author) ] ]
        ifFalse: [
            | path |
            path := (box model path splitOn: '/').
            box color: Color white ] ] ].

This attributes the color of the core contributors to each box. We present below a final computed visualization that highlight one contributor who seems to became the expert of the Refactoring package.

Improving the generation of visitors

Wed, 20 May 2026 00:00:00 GMT

Context

Recently, I needed to implement a feature in FAST-Python requiring a visitor for my metamodel. People knowing me knows that I if I can automatize a task to save time and to keep the code coherant with the model, I’ll go for it. Nicolas Anquetil and Clotilde Toullec recently started to implement a visitor generator, so I tried it. Since it was a POC, I encountered some problems.

I’ll explain in this blog post how to use this visitor generator, what problems I encountered and the solutions I proposed.

Using the visitor generator

Once my changes will be integrated, using the visitor generator is as simple as adding one method on the class side of your generator. For example, in FAST-Python:

FASTPythonMetamodelGenerator class>>metamodelToolGenerators

  ^ { FamixVisitorGenerator }

This will tell the generator to also generate a visitor trait.

It is possible to also customize the package in which the visitor is generated by overridin #packageNameForVisitor:

FASTPythonMetamodelGenerator class>>packageNameForVisitor

  ^ #'FAST-Python-Visitor'

Problems I faced

While using this visitor generator I faced a few problems.

Dependency on sub metamodels

Problem

The first one I encountered is that FAST-Python is depending on FAST that is depending on Famix that is depending on MooseQuery.

In order to have the FAST-Python visitor working, all dependencies needed to have a visitor themselves which was not the case. So I had to generate a FAST visitor for example. I could do it since I am part of the moosetechnology organization, but this is a hassle.

Solution

The solution I found was to generate standalone visitors. Instead of using the visitors of the sub metamodels, now we generate a visitor containing the visit of all entities of the model and the visit of the remote trait used in the model.

This can seems like a lot of useless code at first, but it has two advantages:

Visitors and now standalone
This allow to customize the visit of remote traits for the language we are currently working on. This will be important for one of the other problems I encountered.

Infinit loops in visitors

Problem

The second problem I faced was that the generated visitor had infinit loops while I tried to use it. The reason is simple, when visiting an entity, we were visiting all its relations, but Moose is a cyclic graph since we have relations to the contained entities and relations to the containers of the entity.

Solution

Since most analysis that we are doing are “top down” (meaning that we start by the top level entities of our project and we visit their children) and we do not often need a “bottom up” visitor (for this we can just iterate on parents), I decided to exclude the visit of parent entities from the visit.

Double visit of some relations

Problem

In some cases I noticed the we visited two times the same nodes. This is due to the fact that Famix model mix the usage of superclasses and traits. Here is a simple case to understant what is happening:

Here we see that a FASTPyReturnStatement inherits from FASTPyStatement. This class is using FASTTStatement. FASTPyReturnStatement also uses the trait FASTPyReturnStatement, but this trait also uses FASTTStatement.

Now when we visit FASTPyReturnStatement, we end up visiting two times FASTTStatement. Once via its superclass and once via its trait composition.

Solution

The solution I proposed is that when we visit a trait composition, we check the users of the trait and if some of them already visit the trait in their superclasses, we skip the visit of this relation.

With this, #visitFASTPyReturnStatement in the context of FAST-Python become:

visitFASTTReturnStatement: aTReturnStatement

  <generated>
  "We should visit FASTTStatement but all its users in this language already visit it in their superclasses so we skip the call here.".

  self visitEntity: aTReturnStatement expression

In some cases, we need to skip the visit for some entities, but not all of them. For example, with conditional statements. In that case we generate a method like this:

visitFASTTConditionalStatement: aTConditionalStatement

  <generated>
  "We do not visit all behaviorals because some classes already visit it in their superclasses in this language implementation. Visiting them here also would cause a double visit of this trait."
  ({ FASTPyIfStatement . FASTPyWhileStatement } includes: aTConditionalStatement class)
    ifFalse: [ self visitFASTTStatement: aTConditionalStatement ].

  self visitFASTTWithCondition: aTConditionalStatement

This is possible only because we generate standalone visitors now.

Conclusion

With those changes I’m hoping it will be easier to generate and use visitors with Famix. The big advantage is that if we do not need to touch the visitor by hand, it will follow all the evolutions of the metamodel.

Have fun with this :)

Introducing Java initializers

Tue, 17 Feb 2026 00:00:00 GMT

In Java, we can define behavior that is executed exclusively at the initialization of an instance. For now, our metamodel represented these behaviors as methods. This evolution represents them as Initializers.

What are initializers?

We consider as initializers the following elements:

Constructors: they are called when creating a new instance. When a constructor is called, if no explicit call is defined, it implicitly calls the default no-argument constructor, that calls the no-argument constructor in the superclass. We do not represent implicit constructors and these invocations.
Initialization blocks: blocks that are executed when a new instance is created. They are copied by the Java compiler into each constructor and avoid code duplication. We do not represent this implicit invocation.
In Famix: the <Initializer> method: we create a method to hold all attribute definitions in a type.

Why do we change?

The main motivation for this change is to adapt the metamodel to the needs of building call graphs. Call graphs must be able to create the implicit invocations described above and to distinguish between the 3 types of initializers.

Another motivation is to differentiate between initializers and actual methods. In analyses, we often need to focus on methods and initializers can add noise when treated as actual methods, especially the <Initializer> method.

What’s changing

We introduce FamixJavaInitializer, a subclass of FamixJavaMethod. An Initiliazer has 2 properties:

#isInitializationBlock: boolean, false by default.
#isConstructor: boolean, derived. In java, a constructor is an initializer with the same name as its parent type, with no declared type (or void as declared type).

We do not merge all initializers as we did before, but we still merge similar initializers, that will always be called together. In a Java model, a type (TWithMethods) can define a maximum of 4 initializers (besides constructors):

An instance initialization block, that is the merge of all instance initialization blocks.
An instance-side <Initializer> method, similar to the one we created before.
A static initialization block, merge of all static initialization blocks (isClassSideis true).
A static <Initializer>method (isClassSideis true) for static attributes definition.

When inspecting a Java model, initializers can now be found under Initializers and Model initializers.

Speed up models creation: application to JSON/MSE parsing

Tue, 18 Nov 2025 00:00:00 GMT

Context

In order to be able to work with Moose there is a prerequisite we cannot avoid: we need a model to analyze. This can be archieved in 2 principal ways:

Importing an existing JSON/MSE file containing a model
Importing a model via a Moose importer such as the Pharo importer or Python importer

While doing this, we create a lot of entities and set a lot of relations. But this can take some time. I found out that this time was even bigger than I anticipated while profiling a JSON import.

Here is the result of the profiling of a JSON of 330MB on a Macbook pro M1 from 2023:

Form this profiling we can see that we spend 351sec for this import. We can find more information in this report:

On this screenshot we can see some noise due to the fact that the profiler was not adapted to the new event listening loop of Pharo. But in the leaves we can also see that most of the time is spent in FMSlotMultivaluedLink>>#indexOf:startingAt:ifAbsent:.

This is used by a mecanism of all instance variables that are FMMany because those we do not want duplicated elements. Thus, we check if the collection contains the element before adding it.

But during the import of a JSON file, we should have no duplicates making this check useless. This also explains why we spend so much time in this method: we always are in the worst case scenario: there is no element matching.

The optimization

In order to optimize the creation of a model when we know we will not create any duplicates, we can disable the check.

For this, we can use a dynamic variable declaring that we should check for duplicated elements by default, but allowing to disable the check during the execution of some code.

DynamicVariable << #FMShouldCheckForDuplicatedEntitiesInMultivalueLinks
  slots: {};
  tag: 'Utilities';
  package: 'Fame-Core'

FMShouldCheckForDuplicatedEntitiesInMultivalueLinks>>#default
  ^ true

And now that we have the variable, we can use it:

FMSlotMultivalueLink >> unsafeAdd: element
  (self includes: element) ifFalse: [ self uncheckUnsafeAdd: element ]
  FMShouldCheckForDuplicatedEntitiesInMultivalueLinks value
    ifTrue: [ (self includes: element) ifFalse: [ self uncheckUnsafeAdd: element ] ]
    ifFalse: [ self uncheckUnsafeAdd: element ]

FMMultivalueLink >> unsafeAdd: element
  (self includes: element) ifFalse: [ self uncheckUnsafeAdd: element ]
  FMShouldCheckForDuplicatedEntitiesInMultivalueLinks value
    ifTrue: [ (self includes: element) ifFalse: [ self uncheckUnsafeAdd: element ] ]
    ifFalse: [ self uncheckUnsafeAdd: element ]

And the last step is to disable the check during the MSE/JSON parsing:

FMMSEParser >> basicRun
  self Document.
  self atEnd ifFalse: [ ^ self syntaxError ]
  FMShouldCheckForDuplicatedEntitiesInMultivalueLinks value: false during: [
      self Document.
      self atEnd ifFalse: [ ^ self syntaxError ] ]

Result of the optimization

Now let’s try to import the same JSON file with the optiwization enabled:

We can see that the import time went from 351sec to 113sec!

We can also notice that we do not have one bottleneck in our parsing. This means that it will be harder to optimize more this task (even if some people still have some ideas on how to do that).

Use this optimization in your project

This optimization has been made for the import of JSON but it can be used in other contexts. For example, in the Moose Python importer, the implementation is sure to never produce a duplicate. Thus, we could use the same trick this way:

FamixPythonImporter >> import
  FMShouldCheckForDuplicatedEntitiesInMultivalueLinks value: false during: [ super import ]

Testing your algo on a java project

Wed, 08 Oct 2025 00:00:00 GMT

When developping algorithm on top of the Moose platform, we can easily hurt a wall during testing.

To do functional (and sometimes unit) testing, we need to work on a Moose model. Most of the time we are getting this model in two ways:

We produce a model and save the .json to recreate this model in the tests
We create a model by hand

But those 2 solutions have drawbacks:

Keeping a JSON will not follow the evolutions of Famix and the model produce will not be representative of the last version of Famix
Creating a model by hand has the drawback of taking the risk that this model will not be representative of what we could manipulate in reality. For example, we might not think about setting the stubs or the source anchors

In order to avoid those drawbacks I will describe my way of managing such testing cases in this article. In order to do this, I will explain how I set up the tests of a project to build CallGraph of Java projects.

The idea

The idea I had for testing callgraphs is to implement real java projects in a resources folder in the git of the project. Then, we can parse them when launching the tests and manipulate the produced model. This would ensure that we always have a model up to date with the latest version of Famix. If tests breaks, this means that our famix model evolved and that our project does not work anymore for this language.

Basic setup

Create your java code

The first step to build tests is to write some example java code.

I will start with a minimal example:

public class Main {
    public static void main(String[] args) {

        System.out.println("Hello World!");
    }
}

I’ll save this file in the git repository of my project under Famix-CallGraph/resources/sources/example1/Main.java.

Now that we have the source code, we need a way to access it in our project.

Use GitBridge

In order to access our resources, we will use GitBrigde.

You can install it by executing:

Metacello new
  githubUser: 'jecisc' project: 'GitBridge' commitish: 'v1.x.x' path: 'src';
  baseline: 'GitBridge';
  load

But we should add it to our baseline:

BaselineOfFamixCallGraph >> #gitBridge: spec

  spec baseline: 'GitBridge' with: [ spec repository: 'github://jecisc/GitBridge:v1.x.x/src' ]

BaselineOfFamixCallGraph >> #baseline: spec

  <baseline>
  spec for: #common do: [
    "Dependencies"
    self gitBridge: spec.

    "Packages"
    spec
      package: 'Famix-CallGraph';
      package: 'Famix-CallGraph-Tests' with: [ spec requires: #( 'Famix-CallGraph' 'GitBridge' ) ]. "<== WE ADD GITBRIDGE HERE!"
     ].

  spec for: #NeedsFamix do: [
    self famix: spec.

    spec package: 'Famix-CallGraph' with: [ spec requires: #( Famix ) ] ]

Now that we have the dependency running, we can use this project. We will explain the minimal steps here but you can find the full documantation here.

The usage of GitBridge begins with the definition of our FamixCallGraphBridge:

GitBridge << #FamixCallGraphBridge
  slots: {};
  package: 'Famix-CallGraph-Tests'

Now that this class exists we can access our git folder using FamixCallGraphBridge current root.

Let’s add some syntactic suggar:

FamixCallGraphBridge class >> #resources

  ^ self root / 'resources'

FamixCallGraphBridge class >> #sources

  ^ self resources / 'sources'

We can now access our java projects doing FamixCallGraphBridge current sources.

This step is almost done, but in order for our tests to work in a github action (for example), we need two little tweaks.

In our smalltalk.ston file, we need to register our project in Iceberg (because GitBridge uses Iceberg to access the root folder).

SmalltalkCISpec {
  #loading : [
    SCIMetacelloLoadSpec {
      #baseline : 'FamixCallGraph',
      #directory : 'src',
     #registerInIceberg : true   "<== This line"
    }
  ]
}

Also, in our github action we need to be sure that the checkout action will get enough info for git bridge to run and not the minimal ammount (which is the default) adding a fetch-depth: option.

steps:
  - uses: actions/checkout@v4
    with:
      fetch-depth: '0'

Parse and import your model

Now we need to be able to parse our project. For this, we will use a Java utility thaht is directly in Moose: FamixJavaFoldersImporter.

We can parse and receive a model doing:

model := (FamixJavaFoldersImporter importFolders: { FamixCallGraphBridge sources / 'example1' }) anyOne.

Tests implementation

Now that we can access the model it is possible to implement our tests.

I’m starting by an abstract class:

TestCase << #FamixAbstractJavaCallGraphBuilderTestCase
  slots: { #model . #graph };
  package: 'Famix-CallGraph-Tests'

Now I will create a TestCase that needs my java model

FamixAbstractJavaCallGraphBuilderTestCase << #FamixJavaCHAExample1Test
  slots: {};
  package: 'Famix-CallGraph-Tests'

And now I will create a setup importing the model and creating a call graph:

FamixAbstractJavaCallGraphBuilderTestCase >> #setUp

  super setUp.
  model := (FamixJavaFoldersImporter importFolders: { self javaSourcesFolder }) anyOne.
  graph := (FamixJavaCHABuilder entryPoints: self entryPoints) build

FamixJavaCHAExample1Test >> #javaSourcesFolder
  "Return the java folder containing the sources to parse for those tests"

  | folder |
  folder := FamixCallGraphBridge sources / 'example1'.

  folder ifAbsent: [ self error: 'Folder does not exists ' , folder pathString ].

  ^ folder

And now you have your model available for the testing!

Optimization

I am using this technic to tests multiple projects such as parsers or call graph builders. In those projects I do touch my model and the setup can take time. So I optimize this setup in order to build a model only once for all the test case using a TestResource.

In order to do this we can remove the slots we added to FamixAbstractJavaCallGraphBuilderTestCase and create a test resource that will hold them

TestResource << #FamixAbstractJavaCallGraphBuilderTestResource
  slots: { #model . #graph };
  package: 'Famix-CallGraph-Tests'

Then we can move the setup to this class

FamixAbstractJavaCallGraphBuilderTestResource >> #setUp

  super setUp.
  model := (FamixJavaFoldersImporter importFolders: { self javaSourcesFolder }) anyOne.
  graph := (FamixJavaCHABuilder entryPoints: self entryPoints) build

Personally I’m also adding a tearDown cleaning the vars because TestResources are singletons and I do not want to hold a model in memory all the time.

Then I’m creating my test resource for the example1 project.

FamixAbstractJavaCallGraphBuilderTestResource << #FamixJavaCHAExample1Resource
  slots: {};
  package: 'Famix-CallGraph-Tests'

FamixJavaCHAExample1Resource >> #javaSourcesFolder
  "Return the java folder containing the sources to parse for those tests"

  | folder |
  folder := FamixCallGraphBridge sources / 'example1'.

  folder ifAbsent: [ self error: 'Folder does not exists ' , folder pathString ].

  ^ folder

And now we can declare that the TestCase will use this resource:

FamixJavaCHAExample1Test class >> #resources
    ^ { FamixJavaCHAExample1Resource }

The model then become accessible like this:

FamixJavaCHAExample1Resource >> #model

  ^ self resources anyOne current model

Simplify your life

Here is a few tricks I use to simplify even better the setting of my tests cases

Automatic java source folder detection

The first one is to make automatic the detection of the java source folder by using the name of the test cases:

FamixAbstractJavaCallGraphBuilderTestResource >> #javaSourcesFolder
  ^ self class javaSourcesFolder

FamixAbstractJavaCallGraphBuilderTestResource class >> #javaSourcesFolder
  "Return the java folder containing the sources to parse for those tests"

  | folder |
  folder := FamixCallGraphBridge sources / ((self name withoutPrefix: 'FamixJavaCHA') withoutSuffix: 'Resource') uncapitalized.

  folder ifAbsent: [ self error: 'Folder does not exists ' , folder pathString ].

  ^ folder

We can now remove this method from all subclasses! But makes sure the name of your source folder matches the name of the tests ressource ;)

Automatic test resource detection and access

We can do the same with the detection of the test resource in the test case.

FamixAbstractJavaCallGraphBuilderTestCase class >> #resources

  ^ self environment
      at: ((self name withoutSuffix: 'Test') , 'Resource') asSymbol
      ifPresent: [ :class | { class } ]
      ifAbsent: [ {  } ]

FamixAbstractJavaCallGraphBuilderTestCase class >> #sourceResource

  ^ self resources anyOne current

FamixAbstractJavaCallGraphBuilderTestCase >> #sourceResource
  "I return the instance of the test resource I'm using to build the sources of a java project"

  ^ self class sourceResource

FamixAbstractJavaCallGraphBuilderTestCase >> #model

  ^ self sourceResource model

Et voila ! Now adding a test case ready to use on a new java project is equivalent to create a test case:

FamixAbstractJavaCallGraphBuilderTestCase << #FamixJavaCHAExample2Test
  slots: {};
  package: 'Famix-CallGraph-Tests'

And the resource associated!

FamixAbstractJavaCallGraphBuilderTestResource << #FamixJavaCHAExample2Resource
  slots: {};
  package: 'Famix-CallGraph-Tests'

Nothing much.

Easily find the sources of the tested project

A last thing I am doing to simplify thing is to implement a method to access easily the sources.

FamixJavaCHAExample1Test >> #openSources

  <script: 'self new openSources'>
  self resources anyOne javaSourcesFolder openInOSFileBrowser

Other languages than Java

It is possible to do the same thing for other languages than java but maybe not exactly in the same way than in this blogpost for the section “Parse and import your model”. But this article is meant to be an inspiration!

I hope this helps improve the robustness of our projects :)

Generation of new FAST-Language metamodel using Pharo-Tree-Sitter project

Mon, 15 Sep 2025 00:00:00 GMT

If you’re here, you’re probably interested in creating a new FAST metamodel and expanding Moose to represent the AST (Abstract Syntax Tree) of an additional language. In this post, we explain to you how to generate a “First version” of a new FAST-Language metamodel using the project Pharo-Tree-Sitter. To be able to understand that, we assume you are already familiar with:

Tree-Sitter
Pharo-Tree-Sitter
FAST
Metamodel generators

Tree-Sitter is a parser generator tool and an incremental parsing library. It can build a concrete syntax tree for a source file and efficiently update the syntax tree as the source file is edited. It is able to parse a large variety of programming languages such as Java, C++, C#, Python and many others.
Pharo-Tree-Sitter is a project developed in Pharo that integrates the original Tree-Sitter parsers and allows visualizing their results (such as ASTs) directly in Pharo. It relies on the FFI protocol, which requires the corresponding libraries depending on the OS (.dll, .so, or .pylib) to be present in Pharo’s VM folders. The project supports parsing several languages, and for some of them (like Python, TypeScript, and C), the library generation is automated. You can find more details in the repository’s README. This is the project that we will use to generate a new FAST-Language metamodel, so you need to download it into your Pharo image.
FAST means Famix AST. Contrary to Famix that represent application at a high abstraction level, FAST uses a low-level representation: the AST. FAST defines a set of traits that can be used to create new meta-models compatible with Moose tools. When developing a new FAST-Language metamodel, you will rely on these FAST traits to structure your metamodel. However, this does not apply to the “First version” described in this post, but rather to the upgraded versions when you evolve and refine it.
Metamodel generator is a Pharo library used to create new metamodels such as FAST-Java, Famix-Java, or FAST-Fortran. The generation of any new version of a FAST-Language metamodel can only be achieved through the metamodel generator. As you will see in this post, Pharo-Tree-Sitter enables you to define a new metamodel generator. Once executed, it produces the corresponding FAST-Language metamodel. We will explain this process in more detail in the following sections.

Download Pharo-Tree-Sitter and get the correspondent libraries

First you need to create a Moose image and download Pharo-Tree-Sitter:

Metacello new
  baseline: 'TreeSitter';
  repository: 'github://Evref-BL/Pharo-Tree-Sitter:main/src';
  load.

Once downloaded, you need to make sure that Pharo-Tree-Sitter is able to parse the language that you intend to create the metamodel for. If it is not included, you need to follow the instructions in the readme file of this repository and add the new language. For this blog post we will assume that the language is already supported and we will continue with “Python” 🐍🐍🐍.

To be able to continue, and if this is the first time you’re using this project (Pharo-Tree-Sitter), you need to launch the tests of python in package “TreeSitter-Tests” class “TSParserPythonTest”. This is needed to launch the process of downloading the original tree-sitter and tree-sitter-python projects from GitHub, generating the correspondent libraries and moving them to the correspondent VM folder based on the image version you create: for example Moose 12. If you create another image of another version, you need to launch the tests again to make sure the libraries are again moved to the correspondent folder. Now that you have the libraries, you can parse python code and get an AST, but not FAST-Python model. So in the next step we explain how this can be possible.

Create the first version of the metamodel (FAST-Python in our example)

Don’t worry, not too much to be done, but a snippet of code needs to be written and executed. But we have to explain to you first how it is working.

Explaining package TreeSitter-FAST-Utils

This package contains two main classes: “TSFASTBuilder” and “TSFASTImporter”. For our task we will rely on the first one. The second is used to make the transition between an AST generated by TreeSitter and a FAST-Language model.

“TSFASTBuilder” contains a set of methods responsible for generating a new metamodel generator:

#tsLanguage: is used to set an instance of TSLanguage, which is TSLanguage python in our case.
#createMetamodelGeneratorClass is responsible for creating a new package and a class inside. By default, the class name will be “FASTLanguageNameMetamodelGenerator” which is “FASTPythonMetamodelGenerator” and the package name is “FAST-LanguageName-Model-Generator”. This method also calls another one “typesToReify”, which gets all the symbols from the initial TreeSitter project (using an FFI call), and add them as slots in the class definition. These symbols represent the nodes of the language in question like “class” for Python.
#addPrefixMethodIn: adds #prefix method on the class side of the metamodel generator class. By default it is FASTLanguage.
#addPackageNameMethodIn: adds #packageName method on the class side of the metamodel generator class. By default it’s ‘FAST-Language-Model’.
#addSubmetamodelsMethodIn: adds #submetamodels method on the class side of the metamodel generator class, and by default it contains FASTMetamodelGenerator.
#addDefineClassIn: adds #defineClasses method. In this method slots are defined, starting by #entity then all the symbols imported from TreeSitter.
#addDefineTraitsIn: adds #defineTraits method. By default FASTTEntity trait is created.
#addDefineHierarchyIn: adds #defineHierarchy method. By default only #entity relation is defined with FASTTEntity.
#addDefineRelationsIn: adds #defineRelations method. By default only #entity relations are defined with genericChildren and genericParent.

Voilà, now that you understand how it works, we will show you how to generate one for Python:

tsb := TSFASTBuilder new.
tsb languageName: 'Python'.
tsb tsLanguage: TSLanguage python.
tsb build.

This will generate the metamodel generator. Now that the generator is created you can use it to generate the metamodel:

FASTPythonMetamodelGenerator new generate.

Now you can access the packages and classes created: ‘FAST-Python-Model’ and ‘FAST-Python-Model-Generator’.

From now on you have to handle the metamodel manually. You have to add missing traits (including FAST Traits), properties that should be imported from TreeSitter… You benefit from the importer to handle the parsing on the metamodel side. You can create a package for tools having a #parse method doing this for example:

| parser tsLanguage importer |

Smalltalk image garbageCollect.

parser := TSParser new.
tsLanguage := TSLanguage python.
parser language: tsLanguage.

importer := TSFASTImporter new.
importer tsLanguage: tsLanguage.
importer languageName: 'Python'.
importer originString: string.

^ importer import: (parser parseString: string) rootNode "pay attention to #source: "

You can check FASTTypeScript for more details.

N.B: We recommend you to parse many python examples (you can find a lot in the main project of TreeSitter-Python), using Pharo-Tree-Sitter project. Once parsed you can inspect in Pharo the properties for each node using #collectFieldNameOfNamedChild and find the properties for each one. Then you can add them in #defineRelations of the metamodel.

That’s it for now!

Visualizing java dependencies between microservices

Fri, 12 Sep 2025 00:00:00 GMT

In July, I had to analyze the dependencies between microservices for Berger-Levrault. To do so, I chose to use the Moose tool.

Here is the simple but effective process I followed.

About the project structure

The backend I analyzed follows a common pattern. In the git repository, there is a folder api containing the microservices, and a folder lib with resources for each microservice. There is also an additional project called lib-common.

Thus, the microservice home is composed of a project named api-home and a project named lib-home.

src
- api
  - api-home
    src/
    …
- lib
  - lib-home
    src/
    …
  - lib-common/
    …

We wanted to check that dependencies were correctly implemented in the project:

no api project should directly depend on another api (API calls are allowed, but not classic Java dependencies)
each api project can depend on its equivalent lib project
lib projects can depend on lib-common

Let’s see how to perform this check with Moose.

Loading the project

To perform the analysis, I used Moose and followed these steps:

I installed the latest version of Moose.
I cloned the repository containing the backend to analyze.
I installed the project dependencies with:
```
mvn clean install
```

I used VerveineJ to generate a model of the code. To avoid version issues, I used the Docker version of VerveineJ, which gave me a model.json file:

docker run -v /path/to/my/project:/src -v /home/badetitou/.m2/repository:/dependency  ghcr.io/evref-bl/verveinej:v3.3.1 -alllocals -anchor assoc -format json -o model.json

I loaded the model into a Moose 12 image by drag-and-dropping the model.json file into the running Moose image.

Building a dependency visualization

Moose provides ready-to-use visualizations to represent dependencies. In my case, I chose to use the Architectural map. This visualization presents the entities of the model (packages, classes, methods) as a tree and displays the associations between them (i.e., the dependencies).

I first asked this visualization to display all the classes. It works, but does not allow us to distinguish the different microservices.

The main problem is that too much information is displayed and we cannot see the microservices. To fix this, I used Moose’s tag feature. A tag allows you to associate a color and a name to an entity.

So I tagged the classes of my system depending on their location in the repository.

To do this, in a Moose Playground, I used the following script (adapt it to your context 😉):

model allTaggedEntities do: [ :entity | entity removeTags ].

((model allWithSubTypesOf: FamixJavaType) reject: [ :type | type sourceAnchor isNil ]) do: [ :class |
    class sourceAnchor ifNotNil: [ :sa |
        (sa fileName beginsWith: './services/api-A') ifTrue: [ class tagWithName: 'A' ].
        (sa fileName beginsWith: './services/api-B') ifTrue: [ class tagWithName: 'B' ].
        (sa fileName beginsWith: './services/api-C') ifTrue: [ class tagWithName: 'C' ].

        (sa fileName beginsWith: './libraries/lib-A') ifTrue: [ class tagWithName: 'lib-A' ].
        (sa fileName beginsWith: './libraries/lib-common') ifTrue: [ class tagWithName: 'lib-common' ].
        (sa fileName beginsWith: './libraries/lib-B') ifTrue: [ class tagWithName: 'lib-B' ].
        (sa fileName beginsWith: './libraries/lib-C') ifTrue: [ class tagWithName: 'lib-C' ].
    ]
].

(model allWithSubTypesOf: FamixJavaType) reject: [ :type | type tags isEmpty ]

The result is not perfect yet because entities are not grouped by tag. To fix this, simply select the tag to add option in the architectural map settings.

You then get a clear visualization of the links between the microservice projects and the libraries they use. We see that no api is linked to an incorrect lib project. We also notice that microservice B is linked to lib-B as well as lib-common. Maybe this link to lib-common should be removed? But that’s another story…

Building a Famix importer with TreeSitterFamixIntegration

Mon, 11 Aug 2025 00:00:00 GMT

Analyzing source code starts with parsing and for this you need semantic understanding of how symbols in the code relate to each other. In this post, we’ll walk through how to build a C code importer using the TreeSitterFamixIntegration framework.

Prerequisites

Basic knowledge of Famix and Moose.
Basic knowledge of what Tree-sitter is.
Familiarity with the Visitor design pattern. You can check this blog post which explains the Visitor pattern in the context of tree-sitter ASTs.

Overview of TreeSitterFamixIntegration

The TreeSitterFamixIntegration stack provides tools to ease the development of Famix importers using tree-sitter. This package offers some great features for parsing such as (but not limited to):

Useful methods for source management (getting source text, positions, setting sourceAnchor of a famix entity).
Error handling to help catch and report parsing issues
a better TreeSitter node inspector (which is very helpful when debugging)
Utility to efficiently import and attach single-line and multi-line comments to their corresponding entities.
Context tracking for symbol scope (no more context push and pop 😁)

There is a detailed documentation you can check that explain every features.

Step 1: Setting up our environment

After creating a new Moose image, let’s start by loading the necessary packages.

The C Metamodel

First, we need to load the C metamodel. This metamodel provides the Famix classes that represent C entities such as functions, structs, variables, etc.

Metacello new
    baseline: 'FamixCpp';
    repository: 'github://moosetechnology/Famix-Cpp:main';
    load

The TreeSitterFamixIntegration project

Next, we need to load the TreeSitterFamixIntegration project. It provides both pharo-tree-sitter and SRSymbolResolver.

 Metacello new
    githubUser: 'moosetechnology' project: 'TreeSitterFamixIntegration' commitish: 'main' path: 'src';
    baseline: 'TreeSitterFamixIntegration';
    load

The project structure

Now that we have the necessary packages loaded, we can create our C importer.

Create a new package named Famix-C-Importer.

The minimum classes we will have to create inside are:

FamixCimporter: This class will be responsible for importing C files and parsing them using Tree-sitter.
FamixCVisitor: This class will walk through the parsed C syntax tree and create Famix entities.
FamixCCommentVisitor: This class will handle comments and attach them to the corresponding Famix entities.

The `FamixCimporter` class

The FamixCimporter class is the entry point for our importer. It will handle the parsing of C files into Abstract Syntax Trees (AST).

This class will inherit from FamixTSAbstractImporter (defined in the TreeSitterFamixIntegration project), which provides the necessary methods for importing and parsing C files using Tree-sitter.

FamixTSAbstractImporter << #FamixCImporter
    slots: {};
    package: 'Famix-C-Importer'

Now, let’s override some methods to set up our importer:

1. `treeSitterLanguage` method

FamixCImporter >> treeSitterLanguage
    "Should return a TreeSitter language such as  TSLanguage python"

    ^ TSLanguage cLang

This method returns the Tree-sitter language we want to use for parsing. In this case, we are using the C language. You can find the available languages in the Pharo-Tree-Sitter package.

2. `visitorClass` method

FamixCImporter >> visitorClass

    ^ FamixCVisitor

It returns the visitor class that will walk through the parsed syntax tree and create Famix entities. We will define this class later.

3. `importFileReference:` method

FamixCImporter >> importFileReference: aFileReference

    aFileReference isFile
        ifTrue: [
            (self isCFile: aFileReference) ifFalse: [ ^ self ].
            self importFile: aFileReference
        ]
        ifFalse: [
            aFileReference children do: [ :each |
            self importFileReference: each
        ].
    ]

This method calls importFile: on all C files recursively found in a directory. We will add more logic to this method later but for now, it serves as a starting point for our importer.

The isCFile: method checks if the file has a .c or .h extension.

FamixCImporter >> isCFile: aFileReferencemon
    ^ #( 'c' 'h' ) includes: aFileReference extension

The importFile: method is defined in the FamixTSAbstractImporter class (provided by the TreeSitter-Famix-Integration project). It parses the file content to create an AST and then passes the visitor (the FamixCVisitor that we previously defined) to walk through the AST.

The `FamixCVisitor` class

The FamixCVisitor class is responsible for walking through the parsed AST and creating Famix entities. It will inherit from FamixTSAbstractVisitor, which provides the necessary methods for visiting Tree-sitter nodes.

FamixTSAbstractVisitor << #FamixCVisitor
    slots: {};
    package: 'Famix-C-Importer'

For this class, we will just need to override one method:

`modelClass` method

FamixCVisitor >> modelClass

    ^ FamixCModel

It returns the Famix metamodel class that will be used to create Famix entities. In this case, we are using FamixCModel which is in the Famix-Cpp package.

Let’s test our importer so far

Now that we have our importer and visitor classes set up, we can already test it. To test our importer, we can create a simple C file and import it using the FamixCImporter class.

test.c

#include <stdio.h>

int aGlobalVar = 1;
int main() {
    int aLocalVar;
    aLocalVar = aGlobalVar + 2;
}

To import this file, we can use the following code in the Playground (cmd + O + P to open it):

Before running the above code, open the Transcript to see the logs (cmd + O + T to open it).

Then select all the code and run it by inspecting it (cmd + I or click the “Inspect” button). You will get something similar to this.

The above screenshot shows what is inside our model. We can see that there is pretty much nothing there yet apart from the SourceLanguages which is added by default by TreeSitterFamixIntegration.

Now if we look at the Transcript, we can see that the importer has imported the file but we didn’t implement the visitor methods yet for every node in the AST, so no Famix entities were created.

If you want to inspect the corresponding AST of our test file, you can do something similar to what is in this other blog post on tree-sitter.

Step 2: Our first Famix entities

In this section we are going to see some examples of visiting methods for creating compilation unit and function entities.

CompilationUnit entities

Let’s go back to our FamixCImporter class and from there we will create a CompilationUnit and HeaderFile entities. We need to do that there because we have to check if the file is a header file or a source file.

FamixCImporter >> importFileReference: aFileReference
    aFileReference isFile
    ifTrue: [
        | fileEntity |
        (self isCFile: aFileReference) ifFalse: [ ^ self ].
        fileEntity := aFileReference extension = 'c'
                        ifTrue: [
                         visitor model newCompilationUnitNamed: aFileReference basename.
                  ]
                        ifFalse: [
                         visitor model newHeaderFileNamed: aFileReference basename.
                  ].

        visitor
          useCurrentEntity: fileEntity
          during: [ self importFile: aFileReference ] ]

    ifFalse: [
        aFileReference children do: [ :each |
          self importFileReference: each
                    ].

        ^ self ]

We use the useCurrentEntity:during: to provide a context for the visitor. This is same as pushing the fileEntity to a context, visit children and then popping it from the context. And it will set the current entity to the fileEntity.

Now try importing a whole directory containing C files. You should see that the importer creates a FamixCHeaderFile for each header file and a FamixCCompilationUnit for each source file.

Source Anchors

To set the source anchor for any Famix entity, we can use the setSourceAnchor: aFamixEntity from: aTSNode method provided by the FamixTSAbstractVisitor class. This method takes a Famix entity and a Tree-sitter node.

We can use it to set the source anchor for our fileEntity . Go to visitTranslationUnit: in the FamixCVisitor class and add the following code:

FamixCVisitor >> visitTranslationUnit: aNode
  self setSourceAnchor: self currentEntity from: aNode.
  self visitChildren: aNode "for not cutting the traversal"

Now if we import our test.c file again, we will see that the CompilationUnit entity has a source anchor.

Function entities

Next, we will create FamixCFunction entities for each function declaration in the C file. We will do this in the visitFunctionDefinition: method of the FamixCVisitor class.

But first we need to know where the function name is located to create the FamixCFunction entity. Create the method and put a halt there to inspect the node.

visitFunctionDefinition: aNode
  self halt.
  self visitChildren: aNode.

If we look at the function definition node, we can see that the function name is in the identifier node, which is a child of the function declarator node.

To get that name, there are two ways:

visit the function_declarator until the identifier returns its name using self visit: aNode
get it by child field name using aNode _fieldName that returns the child node with the given field name. And you don’t need to implement the _fieldName method because it is already handled by the framework.

For simplicity, and to show other available features in the framework, we will use the second way.

Let’s inspect the function definition node to see what fields it has.

So if we do aNode _declarator it will return the function declarator node

And if we do aNode _declarator from the function_declarator it will give us the identifier that we want.

Now we can create the function entity and set its name and source anchor.

visitFunctionDefinition: aNode

    | declaratorNode identifierNode functionName entity |

    declaratorNode := aNode _declarator.
    identifierNode := declaratorNode _declarator.
    functionName := identifierNode sourceText.


    entity := (model newFunctionNamed: functionName) functionOwner: self currentEntity.

    self setSourceAnchor: entity from: aNode.

    self useCurrentEntity: entity during: [ self visitChildren: aNode ]

The self currentEntity returns the compilation unit entity which is the parent of the function entity.

And before visiting the children, we set the current entity to the newly created function entity using useCurrentEntity:during:. This will allow us to create other entities that are related to this function, such as parameters and local variables.

Local and Global Variables

The difference between local and global variables is that local variables are declared inside a function, while global variables are declared outside any function.

Implementation

To create the variable entities, we will create the visitDeclaration: method in the FamixCVisitor class. This method is called for each variable declaration in the C file.

FamixCVisitor >> visitDeclaration: aNode
  "fields: type - declarator"

  | varName entity |

  self visit: aNode _type.

  varName := self visit: aNode _declarator.

  entity := self currentEntity isFunction
              ifTrue: [
                  (model newLocalVariableNamed: varName)
                    parentBehaviouralEntity: self currentEntity;
                    yourself ]
              ifFalse: [
                  (model newGlobalVariableNamed: varName)
                    parentScope: self currentEntity;
                    yourself ].

  self setSourceAnchor: entity from: aNode.

The visitDeclaration: method does the following:

Visits the variable’s type. This will allow us to parse its type information.
Retrieves the variable name by visiting the declarator field. If the variable is initialized, this will be an init_declarator node; otherwise, it will be an identifier. We should implement visit methods for both cases to extract the name correctly.

FamixCVisitor >> visitInitDeclarator: aNode
  "fields: declarator - value"

  self visit: aNode _value.
  ^ self visit: aNode _declarator "variable name is in the declarator node"

FamixCVisitor >> visitIdentifier: aNode

    ^ aNode sourceText "returns the name of the variable"

Creates a variable entity, either a local variable or a global variable, depending on whether the current entity is a function or not.
Sets the source anchor for the variable entity using the setSourceAnchor:from: method.

Step 3: Symbol resolution

In this section, we will implement the symbol resolution for our C importer. This will allow us to resolve references to variables and functions in our C code.

As an example, we will resolve the reference to the local variable aLocalVar in the main function, which will be represented as a famix write access entity.

Implementation

Create the write access entity

To create the write access entity, we will implement the visitAssignmentExpression: method in the FamixCVisitor class. This method is called for each assignment expression.

visitAssignmentExpression: aNode
  "fields: left - right"
  | access leftVarName |

  leftVarName := self visit: aNode _left.

  access := model newAccess accessor: self currentEntity;
          isWrite: true;
          yourself.

  self setSourceAnchor: access from: aNode.

Using SRIdentifierResolvable

Add the following code to the visitAssignmentExpression: method to resolve the variable:

visitAssignmentExpression: aNode
  "fields: left - right"
  | access leftVarName |

  leftVarName := self visit: aNode _left.

  access := model newAccess accessor: self currentEntity;
          isWrite: true;
          yourself.

  self setSourceAnchor: access from: aNode.

  self
    resolve: ((SRIdentifierResolvable identifier: leftVarName)
          expectedKind: {
            FamixCLocalVariable.
            FamixCGlobalVariable };
          yourself)

    foundAction: [ :variable :currentEntity | access variable: variable ].

The resolve: aResolvable foundAction: aBlockClosure method is provided by the FamixTSAbstractVisitor class.

It takes two arguments:

aResolvable: an instance of SRIdentifierResolvable. This resolvable is created with the identifier (the variable name) and the expected kinds of entities (in this case, either a local variable or a global variable). The identifier: method sets the identifier to resolve, and the expectedKind: method sets the expected kinds of entities that can be resolved.
aBlockClosure: a block that will be executed when the resolvable is resolved (we found the variable). In this case we set the variable of the access entity to the resolved variable.

See the SRIdentifierResolvable documentation

Custom resolver

The SRIdentifierResolvable is a generic resolver that can be used to resolve identifiers. However, in some cases, we may need to create a custom resolver to handle specific cases. In that case, we can create a class that inherits from SRResolvable and override the resolveInScope:currentEntity: method to implement our custom resolution logic.

For more information about the symbol resolver, you can check the documentation.

Step 4: Parse comments

The TreeSitterFamixIntegration package provides a utility to parse comments and attach them to the corresponding Famix entities. This is done using the FamixCCommentVisitor class.

let’s add some comments to our test.c file:

test.c

#include <stdio.h>

int aGlobalVar = 1;

/*
entry point of our programm
*/
int main() {
    int aLocalVar; // a local variable
    aLocalVar = aGlobalVar + 2;
}

Implementation

To parse comments, we will create the FamixCCommentVisitor class that will inherit from FamixTSAbstractCommentVisitor. And we just need to override the visitNode: method.

FamixCCommentVisitor >> visitNode: aNode

  aNode type = #comment ifTrue: [
      (aNode sourceText beginsWith: '/*')
        ifTrue: [ self addMultilineCommentNode: aNode ]
        ifFalse: [ self addSingleLineCommentNode: aNode ] ].

  super visitNode: aNode

We use the addMultilineCommentNode: and addSingleLineCommentNode: methods provided by the FamixTSAbstractCommentVisitor class to add the comment to the model.

For a detailed explanation of how to use the comment visitor, you can check the documentation.

Last thing to do is to use the comment visitor somewhere in our importer. We can do that everytime we finish visiting every children of translation unit node.

FamixCImporter >>visitTranslationUnit: aNode

  self setSourceAnchor: self currentEntity from: aNode.
  self visitChildren: aNode.

  FamixCCommentVisitor visitor: self importCommentsOf: aNode

We use the visitor: aFamixVisitor importCommentsOf: aNode method to import the comments of the translation unit node.

Summary

In this blog post, we have seen how to build a Famix importer for C code using the TreeSitterFamixIntegration framework. We have covered the following topics:

Setting up the environment and creating the importer and visitor classes.
Creating Famix entities for compilation units, functions, and variables.
Implementing symbol resolution for local and global variables.
Parsing comments and attaching them to the corresponding Famix entities.

This is just a starting point for building an importer with this stack. You have to implement more tests and methods to handle other entities. The TreeSitterFamixIntegration framework provides a lot of other utilities we didn’t cover to help you with that.

Useful links

Transformation journey (3/3) : Visualize and apply code transformations

Fri, 13 Jun 2025 00:00:00 GMT

Once again, let me welcome you to the final blog post in those three blog posts about code transformations! During the first blog post, we used queries and tools to locate the entities we seek to transform, before implementing the actual transformation logic in the second blog post.

In this third post, we will use a visualisation window allowing us to check the source code of our entity before and after a transformation, and finally apply those transformation on the actual source files, therefore transposing this transformation from Moose to your actual software!

We are still following the scenario presented in the previous blog posts, on our use case, ArgoUML. In this system three classes define and use a method named logError. A class ArgoLogger has been defined that contains a static method logError.

The transformation task is to add a receiver node to each logError method invocation so that the method called is now ArgoLogger.logError().

Preface

As this blog post follows what was discussed and built in the first one of the series, a lot of information (how to build the image and model used for our use case, but also the use case itself) is provided in the previous posts.

If you haven’t read it, or if you forgot the content of the post, it is recommended to go back to it, as it will not be repeated in this post.

Tools to import

In this final blog post, we do not have any new tools to import. As long as you still have FAST-Java-Tools, Carrefour and Motion from the two previous posts, you should have no problem following this blogpost!

A first look on our transformation

At this point, we now have in our tool modified FAST trees of the methods we need to transform, all encapsulated in wrapper objects to save all the necessary information to complete the transformation, and apply it where needed. Hovewer, before that, you may feel the need to check the transformation produced by your tool.

To do so, we can use the TransformationEditor tool, a dialog window able to display the code before and after our code transformation, along with some nice and practical features to fulfill this task.

First, let’s add a new method to our transformation tool; displayTransformationOfMethod: aMethodWrapper. We will call this new method in the already existing method transformMethod: aJavaMethodWrapper, as shown here :

transformMethod: aJavaMethodWrapper

    | methodInvocationNode codeEditor |
    methodInvocationNode := self motionQueryForFastMethod:
        aJavaMethodWrapper transformedFastMethod.

    methodInvocationNode receiver: self createNewReceiverNode.

    aJavaMethodWrapper addTransformedFastNode:
        methodInvocationNode receiver.

    codeEditor := self displayTransformationOfMethod: aJavaMethodWrapper.

displayTransformationOfMethod: aMethodWrapper

    | exportVisitor transformedCode codeEditor |
    exportVisitor := FASTJavaExporttVisitor new.

    transformedCode := exportVisitor export:
        aMethodWrapper transformedFastMethod.

    codeEditor := TransformationEditor
        openForOriginalText:
        aMethodWrapper originalFastMethod sourceText
        transformed: transformedCode
        originalEntity: aMethodWrapper famixMethod.

    ^ codeEditor

As you can notice in this new method, we also call another tool, the FASTJavaExportVisitor. This visitor will allow us to convert a FAST Java tree to a string giving us the source code of our method after transformation.

Coupling the result of using this tool to this class method of the TransformationEditor class will display such results :

In this new window, you can find on the left the original source code of the entity, before any transformation. And on the right, you can find the source code of the same entity, after applying the transformation.

The first thing we can notice, is how radically different the two methods look at a glance. Hovewer, everything is the same (to the exception of the new ArgoLogger class added with our transformation). They look different because the transformed code is generated from the transformed FAST tree, and the export visitor exports that code with its own formating.

The source code after transformation (on the right pane) is also editable, meaning that you can manually add code to the entity if it is necessary for any reason. Do be careful that the code is syntaxically correct however, as there is no checks (syntax, compilation) on this code in Moose.

Adding highlights

We now have a way to view our transformation, but as it is, it is still tough to view properly where are the changes and what they are. And in this example, the transformation is very basic! If the transformation was spread throughout a class rather than a method, or if the transformation was simply more complex, then the tool would fail as a means to review the changes.

But of course, we can fix that. 😉

To do so, we will modify the new method displayTransformationOfMethod: and use a subclass of the export visitor previously used; FASTJavaExportAndHiglightVisitor.

displayTransformationOfMethod: aMethodWrapper

    | exportVisitor transformedCode codeEditor |
    exportVisitor := FASTJavaExportAndHighlightVisitor new.
    exportVisitor entitiesToHighlight:
        aMethodWrapper transformedFastNodes.

    transformedCode := exportVisitor export:
        aMethodWrapper transformedFastMethod.

    codeEditor := TransformationEditor
        openForOriginalText:
        aMethodWrapper originalFastMethod sourceText
        transformed: transformedCode
        removedHighlights: OrderedCollection new
        addedHighlights: exportVisitor highlights
        originalEntity: aMethodWrapper famixMethod.

    ^ codeEditor

Our first edit simply calls the method wrapper to add a “transformed node”, a FAST node added during the transformation. In a similar fashion, you can add an “original node”, a FAST node removed during the transformation (no such node exists in our example, so we don’t use that feature).

This new transformed node is used in our second edit, where we give the transformed nodes collection to the new visitor tool that we are going to use.

This tool behaves just like the original exporter tool, except when it encounters a node present in the transformed nodes list. In this case, the tool save an interval corresponding to the startPos and endPos of the node, to know precisely its position in the generated string.

This allows our editing window to apply an highlight on this interval, therefore showing only the nodes added in the transformation for the user to easily review the work of their transformation tool.

To use this feature in the editing window, simply do as done in the edit, giving the tool a collection of interval for the removedHighlights and / or addedHighlights slots.

Applying our changes on the source files

Now that we reviewed and accepted the transformation on our entities, only one step remains. To apply those code changes on the actual source code of the modeled software, thereby completing the transformation task.

Once again, we have a tool ready for this need, in the form of the TransformationApplyerOnSourceAnchor class. Let’s do a final edit on our method transformSourceCodeOfMethod to add it:

transformSourceCodeOfMethod: aMethodWrapper

    | exportVisitor transformedCode codeEditor |
    exportVisitor := FASTJavaExportAndHighlightVisitor new.
    exportVisitor entitiesToHighlight:
        aMethodWrapper transformedFastNodes.

    transformedCode := exportVisitor export:
        aMethodWrapper transformedFastMethod.

    codeEditor := TransformationEditor
        openForOriginalText:
        aMethodWrapper originalFastMethod sourceText
        transformed: transformedCode
        removedHighlights: OrderedCollection new
        addedHighlights: exportVisitor highlights
        originalEntity: aMethodWrapper famixMethod.

    codeEditor isOk ifTrue: [
    | tool |
    tool := TransformationApplyerOnSourceAnchor new.
    tool
        transformMethod: aMethodWrapper famixMethod
        withNewSource: codeEditor fetchEditedCode ]

As you can see, all this tool needs in our context is a method and the new source code as a String to complete the operation. You can also see that we called the method fetchEditedCode on our code editor window, which returns the String present in the right window of said editor. Using this method ensures that you get the right data, even if the code was manually edited during its review.

Once this tool has made its job and transformed the source code of your software, do keep in mind that the model of that software is no longer up to date with the source. This means that any analysis that you could run on this model could be flawed, so it is recommended to regularly replace that model with a new one, based from the source after it has been transformed.

Hovewer, the model is slightly updated by the tool after each transformation applyed. The values of startPos and endPos on the source anchor of entities within the file which received a transformation. This means that their source code will still be properly fetched by Moose whenever you need it.

For example, if your transformation tool relies mostly on the FAST models, this allows you to apply some transformations on several entities even if those are located in the same file without having to worry too much about the order in which you make those transformations.

Try it yourself!

This concludes our blog post series about code transformation!

We now have a tool managing every step of the way to complete a specific transformation case, from finding the necessary informations needed for the transformation, transforming a FAST model and using those transformed FAST models to view, edit and apply the transformations to the source code.

Just like with the two other posts, feel free to now try the tool and methods that we created here in a Playground and experiment with the results, or the methods themselves!

t := LoggerTransformationTool onModel: MooseModel root first.
t findAndTransformAllLogErrorInvocations

The whole source code that was written on the three blog posts is still available on that repository.

Conclusion

Through the use of tools to query and search through FAST models, transform said models, before generating, reviewing and applying the new code on the source files, we managed to build a fully functionnal and semi-automatic transformation tool for our use case!

This use case was a rather small example. A real transformation in a software would need more queries and FAST edits. But the only real difference would be the scale. Using the methodology and tools in the same way that they were used in this blog post would be enough to apply to many use cases out there. 😄

Parametrics next generation

Wed, 07 May 2025 00:00:00 GMT

How do we represent the relation between a generic entity, its type parameters and the entities that concretize it? The Famix metamodel has evolved over the years to improve the way we represent these relations. The last increment is described in a previous blogpost. We present here a new implementation that eases the management of parametric entities in Moose.

The major change between this previous version and the new implementation presented in this post is this: We do not represent the parameterized entities anymore.

A parametric entity is an entity that declares type parameters or arguments. It can be generic or parameterized, according to the following definitions.
A generic entity is an entity that defines 1 or several type parameters. To be instantiated, inherited, implemented or invoked, their type parameters must be replaced by existing types (type arguments). ArrayList<E> is a generic class. In Famix Java, we represent generic classes, interfaces and methods.
A type parameter is a parameter that must be replaced by an existing type in order to use the generic entity that defines it. In ArrayList<E>, E is a type parameter.
A parameterized entity is a parametric entity for which the type parameters have been replaced by existing types. ArrayList<String> is a parameterized class. Parameterized entities are not represented as entities in the new Famix implementation.
A type argument is the type used to replace a type parameter. It can be any type, except primitive types. String is a type argument of ArrayList<String>.
A concretization is a Famix association that represents the fact that a type parameter has been replaced by a type argument.

What’s wrong with the previous parametrics implementation?

Difference between parametric and non-parametric entities

The major issue with the previous implementation was the difference between parametric and non-parametric entities in practice, particularly when trying to trace the inheritance tree. Here is a concrete example: getting the superclass of the superclass of a class.

For a non-parametric class, the sequence is straightforward: ask the inheritance for the superclass, repeat.

For a parametric class (see the little code snippet below), there was an additional step, navigating through the concretization:

import java.util.ArrayList; "public class ArrayList<E> { /* ... */ }"

public MySpecializedList extends ArrayList<String> {}

This has caused many headaches to developers who wanted to browse a hierarchy: how do we keep track of the full hierarchy when it includes parametric classes? How to manage both situations without knowing if the classes will be parametric or not? The same problem occurred to browse the implementations of parametric interfaces and the invocations of generic methods.

Naming

The previous implementation naming choices were a little complex to grasp and did not match the standard vocabulary, especially in Java:

A type parameter was named a ParameterType
A type argument was named a ConcreteParameterType

Entities duplication

Each time there was a concretization, a parametric entity was created. This created duplicates of virtually the same entity: one for the generic entity and one for each parameterized entity. Let’s see an example:

public MyClass implements List<Float> {

 public List<Integer> getANumber() {
  List<Number> listA;
  List<Integer> listB;
  }
}

For the interface List<E>, we had 6 parametric interfaces:

One was the generic one: #isGeneric >>> true
3 were the parameterized interfaces implemented by ArrayList<E>, its superclass AbstractList<E> and MyClass. They were different because the concrete types were different: E from ArrayList<E>, E from AbstractList<E>and Float.
2 were declared types: List<Number> and List<Integer>.

The new implementation

Parametric associations

When deciding of a new implementation, our main goal was to create a situation in which the dependencies would work in the same way for all entities, parametric or not. That’s where we introduce parametric associations. These associations only differ from standard associations by one property: they trigger a concretization.

Here is the new Famix metamodel traits that represent concretizations:

There is a direct relation between a parametric entity and its type parameters. A concretization is the association between a type parameter and the type argument that replaces it. A parametric association triggers one or several concretizations, according to the number of type parameters the parametric entity has. Example: a parametric association that targets Map<K,V> will trigger 2 concretizations.

The parametric entity is the target of the parametric association. It is always generic. As announced, we do not represent parameterized entities anymore. Coming back to the entities’ duplication example above, we now represent only 1 parametric interface for List<E>and it is the target of the 5 parametric associations.

Entity typing

This metamodel evolution is the occasion of another major change: the replacement of the direct relation between a typed entity and its type. This new association is called Entity typing.

The choice to replace the existing relation by a reified association is made to represent the dependency in coherence with the rest of the metamodel.

With this new association, we can now add parametric entity typings.

In a case like this:

public ArrayList<String> myAttribute;

we have an “entity typing” association between myAttribute and ArrayList. This association is parametric: it triggers the concretization of E in ArrayList<E> by String.

The method #declaredType is still part of the API of TTypedEntity. However, it’s not a getter for the #declaredType attribute anymore. It now returns the type that is target of the association.

The setter #declaredType: is deprecated and cannot be automatically transformed. If you need to replace it, you should create an instance of the EntityTyping class in your metamodel and set the declared type as its target. You will have something like:

myVariable := myModel newLocalVariableNamed: #myVar.
myType := myModel newTypeNamed: #MyType.
myVariable typing: (myModel newEntityTyping
 declaredType: myType;
 yourself).

Type parameters bounds

Type parameters can be bounded:

public class MyParametricClass<T extends Number> {}

In the previous implementation, the bounds of type parameters were implemented as inheritances: in the example above, Number would be the superclass of T. Since this change, bounds were introduced for wildcards. We have now the occasion to also apply them to type parameters. In the new implementation, Number is the upper bound of T.

Summary

This diagram sums up the new parametrics implementation in Famix traits and Java metamodel. Please note that this is not the full Java metamodel but only a relevant part.

What’s next?

Should Concretization really be an association?

The representation of parametric entities is a challenge that will most likely continue as Famix evolves. The next question will probably be this one: should Concretization really be an association? An association is the reification of a dependency. Yet, there is no dependency between a type argument and the type parameter it replaces. Each can exist without the other. The dependency is in fact between the source of the parametric association and the type parameter.

With one of our previous examples:

public MySpecializedList extends ArrayList<String> {}

MySpecializedList has a superclass (ArrayList<E>) and also depends on String, as a type argument. However, String does not depend on E neither E on String.

The next iteration of the representation of parametric entities will probably cover this issue. Stay tuned!

Creating an importer for an alien grammar

Sat, 29 Mar 2025 00:00:00 GMT

In this blog-post, we see some tricks to create a visitor for an alien AST. This visitor can allow, for example, to generate a Famix model from an external AST.

In a previous blog-post, we saw how to create a parser from a tree-sitter grammar. This parser gives us an AST (Abstract Syntax Tree) which is a tree of nodes representing any given program that the parser can understand. But the structure is decided by the external tool and might not be what we want. For example it will not be a Famix model.

Let see some tricks to help convert this alien grammar into something that better fits our needs.

The Visitor design pattern

Let’s first look at what a “Visitor” is. If you already know, you can skip this part.

When dealing with ASTs or Famix models, visitors are very convenient tools to walk through the entire tree/model and perform some actions.

The Visitor is a design pattern that allows to perform some actions on a set of interconnected objects, presumably all from a family of classes. Typically, the classes all belong to the same inheritance hierarchy. In our case, the objects will all be nodes in an AST. For Famix, the objects would be entities from a Famix meta-model.

In the Visitor pattern, all the classes have an #accept: method. Each #accept: in each class will call a visiting method of the visitor that is specific to it. For example the classes NodeA and NodeB will respectively define:

NodeA >> accept: aVisitor
  aVisitor visitNodeA: self.

NodeB >> accept: aVisitor
  aVisitor visitNodeB: self.

Each visiting method in the visitor will with the element it receives, knowing what is its class: in #visitNodeA: the visitor knows how to deal with a NodeA instance and similarly for #visitNodeB:.

The visitor pattern is a kind of ping-pong between the visiting and #accept: methods:

Typically, all the node are interconnected in a tree or a graph. To walk through the entire structure, it is expected that each visiting method take care of visiting the sub-objects of the current object. For example we could say that NodeA has a property child containing another node:

NodeVisitor >> visitNodeA: aNodeA
  "do some stuff"
  aNodeA child accept: self

It is easy to see that if child contains a NodeB, this will trigger the visiting method visitNodeB: on it. If it’s a instance of some other class, similarly it will trigger the appropriate visiting method. To visit the entire structure one simply calls accept: on the root of the tree/graph passing it the visitor.

Visitors are very useful with ASTs or graphs because once all the accept: methods are implemented, we can define very different visitors that will "do some stuff" (see above) on all the object in the tree/graph.

Several of the “Famix-tools” blog-posts are based on visitors.

An external AST

In a preceding blog-post we saw how to create an AST from a Perl program using the Tree-Sitter Perl grammar.

We will use this as an example to see how to create a visitor on this external AST. Here “external” means it was created by an external tool and we don’t have control on the structure of the AST. If we want to create a Famix-Perl model from a Tree-Sitter AST, we will need to convert the nodes in the Tree-Sitter AST into Famix entities.

We will use a simple Perl program as example:

package Person;
sub new {
  my $class = shift;
  my $self = {
    _firstName => shift,
    _lastName  => shift,
    _ssn       => shift,
  };
  # Print all the values just for clarification.
  print "First Name is $self->{_firstName}\n";
  print "Last Name is $self->{_lastName}\n";
  print "SSN is $self->{_ssn}\n";
  bless $self, $class;
  return $self;
}
sub setFirstName {
  my ( $self, $firstName ) = @_;
  $self->{_firstName} = $firstName if defined($firstName);
  return $self->{_firstName};
}
sub getFirstName {
  return $self->{_firstName};
}

(Note: In Perl, “package” is used to create classes. Therefore in our example, “new”, “setFirstName”, and “getFirstName” are some kind of Perl methods.)

Following the instructions in the previous post, you should be able to get a Tree-Sitter AST like this one:

A Visitor on a Tree-Sitter AST

To have a visitor for this AST, we first need to have an accept: method in all the classes of the AST’s nodes. Fortunately this is all taken care of by the Pharo Tree-Sitter project. In TSNode one finds:

accept: aTSVisitor
  ^ aTSVisitor visitNode: self

And a class TSVisitor defines:

visitNode: aTSNode
  aTSNode collectNamedChild do: [ :child |
    child accept: self ]

Which is a method ensuring that all children of a TSNode will be visited. Thanks guys!

But less fortunately, there are very few different nodes in a Tree-Sitter AST. Actually, all the nodes are instances of TSNode. So the “subroutine_declaration_statement”, “block”, “expression_statement”, “return_expression”,… of our example are all of the same class, which is not very useful for a visitor.

This happens quite often. For example a parser dumping an AST in XML format will contain mostly XMLElements. If it is in JSON, they are all “objects” without any native class specification in the format. 😒

Fortunately, people building ASTs usually put inside a property with an indication of the type of each node. For Tree-Sitter, this is the “type” property. Every TSnode has a type which is what is displayed in the screenshot above.

How can we use this to help visiting the AST in a meaningfull way (from a visitor point a view)? We have no control on the accept: method in TSNode, it will always call visitNode:. But we can add an extra indirection to call different visiting methods according to the type of the node.

So, our visitor will inherit from TSVisitor but it will override the visitNode: method. The new method will take the type of the node, build a visiting method name from it, and call the method on the node.

Let’s decide that all our visiting methods will be called “visitPerl<some-type>”. For example for a “block”, the method will be visitPerlBlock:, for a “return_expression” it will be `visitPerlReturn_expression:”.

This is very easily done in Pharo with this method:

visitNode: aTSNode
  | selector |
  selector := 'visitPerl' , aTSNode type capitalized , ':'.
  ^self perform: selector asSymbol with: aTSNode

This method builds the new method name in a temporary variable selector and then calls it using perform:with:.

Note that the type name is capitalized to match the Pharo convention for method names. We could have removed all the underscores (_) but it would have required a little bit of extra work. This is not difficult with string manipulation methods. You could try it… (or you can continue reading and find the solution further down.)

With this simple extra indirection in #visitNode:, we can now define separate visiting method for each type of TSNode. For example to convert the AST to a Famix model, visitPerlPackage: would create a FamixPerlClass, and visitPerlSubroutine_declaration_statement: will create a FamixPerlMethod. (Of course it is a bit more complex than that, but you got the idea, right?)

Creating the visiting methods

Our visitor is progressing but not done yet. If we call astRootNode accept: TreeSitterPerlVisitor new with the root node of the Tree-Sitter AST, it will immediately halt on a DoesNotUnderstand error because the method visitPerlSource_file: does not exist in the visitor.

We can create it that way:

visitPerlSource_file: aTSNode
  ^self visitPerlAbstractNode: aTSNode.


visitPerlAbstractNode: aTSNode
  ^super visitNode: aTSNode

Here we introduce a visitPerlAbstractNode: that is meant to be called by all visiting methods. From the point of view of the visitor, we are kind of creating a virtual inheritance hierarchy where each specific TSNode will “inherit” from that “PerlAbstractNode”. This will be useful in the future when we create sub-classes of our visitor.

By calling super visitNode:, in visitPerlAbstractNode: we ensure that the children of the “source_file” will be visited. And… we instantly get a new halt with DoesNotUnderstand: visitPerlPackage_statement:. Again we define it:

visitPerlPackage_statement: aTSNode
  ^self visitPerlAbstractNode: aTSNode

This is rapidly becoming repetitive and tedious. There are a lot of methods to define (25 for our example) and they are all the same.

Let’s improve that. We will use the Pharo DoesNotUnderstand mechanism to automate everything. When a message is sent that an object that does not understand it, then the message doesNotUnderstand: is sent to this object with the original message (not understood) as parameter. The default behavior is to raise an exception, but we can change that. We will change doesNotUnderstand: so that it creates the required message automatically for us. This is easy all we need to do is create a string:

visitPerl<some-name>: aTSNode
  ^self visitPerlAbstractNode: aTSNode

We will then ask Pharo to compile this method in the Visitor class and to execute it. et voila!

Building the string is simple because the selector is the one that was not understood originally by the visitor. We can get it from the argument of doesNotUnderstand:.

So we define the method like that:

doesNotUnderstand: aMessage
  | code |
  code := aMessage selector , ' aTSNode
  ^super visitNode: aTSNode'.

  self class compile: code classified: #visiting.

  self perform: aMessage selector with: aMessage arguments first

First we generate the source code of the method in the code variable. Then we compile it in the visitor’s class. Last we call the new method that was just created. Here to call it, we use perform:with: again, knowing that our method has only one argument (so only one “with:” in the call).

For more security, it can be useful to add the following guard statement at the beginning of our doesNotUnderstand: method:

  (aMessage selector beginsWith: 'visitPerl')
    ifFalse: [ super doesNotUnderstand: aMessage ].

This ensures that we only create methods that begins with “visitPerl”, if for any reason, some other message is not understood, it will raise an exception as usual.

Now visiting the AST from our example creates all the visiting methods automatically:

Of course this visitor does not do anything but walking through the entire AST. Let’s say it is already a good start and we can create specific visitors from it.

For example we see in the screen shot above that there is a TreeSitterPerlDumpVisitor. It just dumps on the Transcript the list of node visited. For this, it only needs to define:

visitPerlAbstractNode: aTSNode
  ('visiting a ', aTSNode type) traceCr.
  super visitPerlAbstractNode: aTSNode.

Et voila! (number 2)

Note: Redefining doesNotUnderstand: is a nice trick to quickly create all the visiting methods, but it is recommended that you remove it once the visitor is stable, to make sure you catch all unexpected errors in the future.

Better visiting methods

This is all well and good, but the visiting methods have one drawback: They visit the children of a node in an unspecified order. For example, an “assignment_expression” has two children, the variable assigned and the expression assigned to it. We must rely on Tree-Sitter to visit them in the right order so that the first child is always the variable assigned and the second child is always the right-hand-side expression.

It would be better to have a name for these children so as to make sure that we know what we are visiting at any time.

In this case, Tree-Sitter helps us with the collectFieldNameOfNamedChild method of TSNode. This method returns an OrderedDictionary where the children are associated to a (usually) meaningful key. In the case of “assignment_expression” the dictionary has two keys: “left” and “right” each associated to the correct child. It would be better to call them instead of blindly visit all the children.

So we will change our visitor for this. The visitNode: method will now call the visiting method with the dictionnary of keys/children as second parameter, the dictionnary of fields. This departs a bit from the traditional visitor pattern where the visiting methods usually have only one argument, the node being visited. But the extra information will help make the visiting methods simpler:

visitNode: aTSNode
  | selector |
  selector := String streamContents: [ :st |
      st << 'visitPerl'.
    ($_ split: aTSNode type) do: [ :word | st << word capitalized ].
    st << ':withFields:'
  ].

  ^self
    perform: selector asSymbol
    with: aTSNode
    with: aTSNode collectFieldNameOfNamedChild

It looks significantly more complex, but we also removed the underscores (_) in the visiting method selector (first part of the #visitNode: method). So for “assignment_expression”, the visiting method will now be: visitPerleAssignmentExpression:withFields:.

From this, we could have the following template for our visiting methods:

visitPerlAssignmentExpression: aTSNode withFields: fields
  ^{
    self visitKey: 'left' inDictionnary: fields.
    self visitKey: 'right' inDictionnary: fields.
  }

Where visitKey: inDictionnary: takes care of the fact that several nodes may be associated to the same key. Here it is:

visitKey: aKey inDictionnary: childrenDictionnary
  | child |
  child := childrenDictionnary at: aKey ifAbsent: [^nil].

  ^child isCollection
    ifTrue: [ child collect: [ :c | c accept: self ] ]
    ifFalse: [ child accept: self ]

The doesNotUnderstand: method to generate all this is also more complex because there is more to generate. Here it is:

doesNotUnderstand: aMessage
  (aMessage selector beginsWith: 'visitPerl')
    ifFalse: [ super doesNotUnderstand: aMessage ].

  self class
    compile: (self createVisitMethod: aMessage)
    classified: #visiting.

  self
    perform: aMessage selector
    with: aMessage arguments first
    with: aMessage arguments second

The code generation has been extracted in a separate method for the sake of readability:

createVisitMethod: aMessage

  | fields aTSNode |
  aTSNode := aMessage arguments first.
  fields := aMessage arguments second.

  ^String streamContents: [ :str |
    str
      << 'visitPerl'.
      ($_ split: aTSNode type) do: [ :word | str << word capitalized ].
      str
        << ': aTSNode withFields: fields
  ^{
'.

    fields keysDo: [ :key |
      str
        << '    self visitKey: ''' ;
        << key ;
        << ''' inDictionnary: fields.' ;
        cr
    ].
    str
      << '  }' ;
      cr
  ]

Again, it may look a bit complex, but this is only building a string with the needed source code. Go back to the listing of #visitPerlAssignmentExpression: above to see that:

we first build the selector of the new visiting method with its parameter;
then we put a return and start a dynamic array;
after that we create a call to #visitKey:inDictionnary for each field;
and finally, we close the dynamic array.

Et voila! (number 3).

This is it. If we call again this visitor on an AST from Tree-Sitter, it will generate all the new visiting methods with explicit field visiting. For example:

The implementation of all this can be found in the https://github.com/moosetechnology/Famix-Perl repository on github. All that’s left to do is create a sub-class of this visitor and override the visiting methods to do something useful with each node type.

That’s all for today folks.

Creating a Parser based on Tree-Sitter grammar

Tue, 25 Mar 2025 00:00:00 GMT

Moose is a huge consumer of language parsers. Relying on external tools help us with this.

We are always looking into integrating new programming languages into the platform. There are two main requirements for this:

create a parser of the language, to “understand” the source code
create a meta-model for the language, to be able to represent and manipulate the source code

Creating the meta-model has already been covered in an other blogpost: /blog/2021-02-04-coasters

In this post, we will be looking at how to use a Tree-Sitter grammar to help build a parser for a language. We will use the Perl language example for this.

Note: Creating a parser for a language is a large endehavour that can easily take 3 to 6 months of work. Tree-Sitter, or any other grammar tool, will help in that, but it remains a long task.

Getting the Tree-Sitter grammar

We do not explain in detail here how to install tree-sitter or a new Tree-Sitter grammar. I found this page (https://dcreager.net/2021/06/getting-started-with-tree-sitter/) useful in this sense.

For this blog post, we will use the Perl grammar in https://github.com/tree-sitter-perl/tree-sitter-perl.

Do the following:

clone the repository on your disk
go in the directory
do make (note: it gave me some error, but the library file was generated all the same)
(on Linux) it creates a libtree-sitter-perl.so dynamic library file. This must be moved in some standard library path (I chose /usr/lib/x86_64-linux-gnu/ because this is where the libtree-sitter.so file was).

Pharo uses FFI to link to the grammar library, that’s why it’s a good idea to put it in a standard directory. You can also put this library file in the same directory as your Pharo image, or in the directory where the Pharo launcher puts the virtual machines.

The subclasses of FFILibraryFinder can tell you what are the standard directories on your installation. For example on Linux, FFIUnix64LibraryFinder new paths returns a list of paths that includes '/usr/lib/x86_64-linux-gnu/' where we did put our grammar.so file.

Binding tree-sitter in Pharo

We use the Pharo-Tree-Sitter project (https://github.com/Evref-BL/Pharo-Tree-Sitter) of Berger-Levrault, created by Benoit Verhaeghe, a regular contributor to Moose and this blog. You can import this project in a Moose image following the README instructions.

Metacello new
  baseline: 'TreeSitter';
  repository: 'github://Evref-BL/Pharo-Tree-Sitter:main/src';
  load.

The README file of Pharo-Tree-Sitter gives an example of how to use it for Python:

parser := TSParser new.
tsLanguage := TSLanguage python.
parser language: tsLanguage.
[...]

We want to have the same thing for Perl, so we will need to define a TSLanguage class >> #perl method. Let’s take a look at how it’s done in Python:

TSLanguage class >> #python
  ^ TSPythonLibrary uniqueInstance tree_sitter_python

It’s easy to do something similar for perl:

TSLanguage class >> #perl
  ^ TSPerlLibrary uniqueInstance tree_sitter_perl

But we need to define the TSPerlLibrary class. Again let’s look at how it’s done for Python and copy that:

create a TreeSitter-Perl package
create a TSPerlLibrary class in it inheriting from FFILibrary

define the class method:

tree_sitter_perl
  ^ self ffiCall: 'TSLanguage * tree_sitter_perl ()'

and define the class methods for FFI (here for Linux):

unix64LibraryName
  ^ FFIUnix64LibraryFinder findAnyLibrary: #( 'libtree-sitter-perl.so' )

Notice that we gave the name of the dynamic library file created above (libtree-sitter-perl.so). If this file is in a standard library directory, FFI will find it.

A first Pharo AST

We can now experiment “our” parser on a small example:

parser := TSParser new.
tsLanguage := TSLanguage perl.
parser language: tsLanguage.

string := '# this is a comment

my $var = 5;
'.

tree := parser parseString: string.

tree rootNode

This gives you the following window:

That looks like a very good start!

But we are still a long way from home. Let’s look at a node of the tree for fun.

node := tree rootNode firstNamedChild will give you the first node in the AST (the comment). If we inspect it, we see that it is a TSNode

we can get its type: node type returns the string 'comment'
node nextSibling returns the next TSNode, the “expression-statement”
node startPoint and node endPoint tell you where in the source code this node is located. It returns instances of TSPoint:
- node startPoint row = 0 (0 indexed)
- node startPoint column = 0
- node endPoint row = 0
- node endPoint column = 19 That is to say the node is on the first row, extending from column 0 to 19. With this, one could get the text associated to the node from the original source code.

That’s it for today. In a following post we will look at doing something with this AST using the Visitor design pattern.

See you latter

First look at GitProjectHealth

Mon, 24 Mar 2025 00:00:00 GMT

When it comes to understand a software system, we are often focusing on the software artifact itself. What are the classes? How they are connected with each other?

In addition to this analysis of the system, it can be interesting to explore how the system evolves through time. To do so, we can exploit its git history. In Moose, we developed the project GitProjectHealth that enables the analysis of git history for projects hosted by GitHub, GitLab, or BitBucket. The project also comes with a set of metrics one could use directly.

First use of GitProjectHealth

GitProjectHealth is available in the last version of Moose, it can be easily installed using a Metacello script in a playground.

Metacello new
  repository: 'github://moosetechnology/GitProjectHealth:main/src';
  baseline: 'GitLabHealth';
  onConflict: [ :ex | ex useIncoming ];
  onUpgrade: [ :ex | ex useIncoming ];
  onDowngrade: [ :ex | ex useLoaded ];
  load

Example on Famix

For this first blog post, we will experiment GitProjectHealth on the Famix project. Since this project is a GitHub project, we first create a GitHub token that will give GitProjectHealth the necessary authorization.

Then, we import the moosetechnology group (that hosts the Famix project).

glhModel := GLHModel new.

githubImporter := GithubModelImporter new
  glhModel: glhModel;
  privateToken: '<private token>';
  yourself.

githubImporter withCommitsSince: (Date today - 100 days).
group := githubImporter importGroup: 'moosetechnology'.

This first step allows us to get first information on projects. For instance, by inspecting the group, we can select the “Group quality” view and see the group projects and the last status of their pipelines.

Then, by navigating to the Famix project and its repository, you can view the Commits History.

It is also possible to explore the recent commit distribution by date and author

In this visualization, we discover that the most recent contributors are “Clotilde Toullec” and “CyrilFerlicot”. The “nil” refers to a commit authors that did not fill GitHub with their email. It is anquetil (probably the same person as “Nicolas Anquetil”). The square without name is probably someone that did not fill correctly the local git config for username.

Who modified my code? - code churn

A popular metric when looking at git history is the code churn. Code churn refer to edit of code introduced in the past. It corresponds to the percentage of code introduced in a commit and then modified in other comments during a time period (e.g in the next week). However many code churn definitions exit.

The first step is thus to discover what commits modified my code. To do so, we implemented in GitProjectHealth information about diff in commit.

To extract this information, we first ask GitProjectHealth to extract more information for the commits of the famix project.

famix := group projects detect: [ :project | project name = 'Famix' ].

"I want to go deeper in analysis for famix repository, so I complete commit import of this project"
githubImporter withCommitDiffs: true.
famix repository commits do: [ :commit | githubImporter completeImportedCommit: commit ].

Then, when inspecting a commit, it is possible to switch to the “Commits tree” view.

Here how to read to above example

The orange square “Remove TClassWithVisibility…” is the inspected commit.
The gray square is the parent commit of the selected ones.
The red squares are subsequent commits that modify at least one file in common with the inspected commit
The green squares are commits that modifies other part of the code

Based on this example, we see that Clotilde Toullec modifies code introduced in selected commits in three next commits. Two are Merged Pull Request. This can represent linked work or at least actions on the same module of the application.

Can we go deeper in the analysis?

Connect GitProjectHealth

It is possible to go even deeper in the analysis by connecting GitProjectHealth with other analysis. This is possible by connecting metamodels. For instance, it is possible to link GitProjectHealth with Jira system, of Famix models. You can look at the first general documentation, or stay tune for the next blog post about GitProjectHealth!

Control Flow Graph for FAST Fortran

Sat, 08 Mar 2025 00:00:00 GMT

A Control Flow Graph analysis for FAST Fortran

Control Flow Graphs (CFG) are a common tool for static analyzis of a computation unit (eg. a method) and find some errors (unreachable code, infinite loops)

It is based on the concept of Basic Block: a sequence of consecutive statements in which flow of control can only enter at the beginning and leave at the end. Only the last statement of a basic block can be a branch statement and only the first statement of a basic block can be a target of a branch.

There are two distinctive basic blocks:

Start Block: The entry block allows the control to enter into the control flow graph. There should be only one start block.
Final Block: Control flow leaves through the exit block. There may be several final blocks.

The package FAST-Fortran-Analyses in https://github.com/moosetechnology/FAST-Fortran contains classes to build a CFG of a Fortran program unit (a main program, a function, or a subroutine).

Creating the FAST Model

We must first create a FAST model of a Fortran program. For this we need an external parser. We currently use fortran-src-extras from https://github.com/camfort/fortran-src-extras.

To run it on a fortran file you do:

fortran-src-extras serialize -t json -v77l encode <fortran-file.f>

This will produce a json AST of the program that we can turn into a FAST-Fortran AST.

If you have fortran-src-extras installed on your computer, all this is automated in FAST-Fortran

 <fortran-file.f> asFileReference
  readStreamDo: [ :st |
    FortranProjectImporter new getFASTFor: st contents ]

This script will create an array of ASTs from the <fortran-file.f> given fortran file. If there are several program units in the file, there will be several FAST models in this array. In the example below, there is only one program, so the list contains only the AST for this program.

We will use the following Fortran-77 code:

      PROGRAM EUCLID
*     Find greatest common divisor using the Euclidean algorithm
        PRINT *, 'A?'
        READ *, NA
        IF (NA.LE.0) THEN
          PRINT *, 'A must be a positive integer.'
          STOP
        END IF
        PRINT *, 'B?'
        READ *, NB
        IF (NB.LE.0) THEN
          PRINT *, 'B must be a positive integer.'
          STOP
        END IF
        IA = NA
        IB = NB
    1   IF (IB.NE.0) THEN
          ITEMP = IA
          IA = IB
          IB = MOD(ITEMP, IB)
          GOTO 1
        END IF
        PRINT *, 'The GCD of', NA, ' and', NB, ' is', IA, '.'
        STOP
        END

Creating the CFG

From the FAST model above, we will now create a Control-Flow-Graph:

 <FAST-model> accept: FASTFortranCFGVisitor new

The class FASTFortranCFGVisitor implements an algorithm to compute basic blocks from https://en.wikipedia.org/wiki/Basic_block.

This visitor goes throught the FAST model and creates a list of basic blocks that can be inspected with the #basicBlocks method.

There is a small hierarchy of basic block classes:

FASTFortranAbstractBasicBlock, the root of the hierarchy. It contains #statements (which are FAST statement nodes). It has methods to test its nature: isStart, isFinal, isConditional. It defines an abstract method #nextBlocks that returns a list of basic blocks that this one can directly reach. Typically there are 1 or 2 next blocks, but Fortran can have more due to “arithmetic IF”, “computed GOTO” and “assigned GOTO” statements.
FASTFortranBasicBlock, a common basic block with no branch statement. If it is final, its #nextBlocks is empty, otherwise it’s a list of 1 block.
FASTFortranConditionalBasicBlock, a conditional basic block. It may reach several #nextBlocks, each one associated with a value, for example true and false. The method #nextBlockForValue: returns the next block associated to a given value. In our version of CFG, a conditional block may only have one statement (a conditional statement).

You may have noticed that our blocks are a bit different from the definition given at the beginning of the blog-post:

our “common” blocs cannot have several next, they never end with a conditional statement;
our conditional blocks can have only one statement.

For the program above, the CFG has 10 blocks.

the first block is a common block and contains 2 statements, the PRINT and the READ;
its next bloc is a conditional block for the IF. It has 2 next blocs:
- true leads to a common block with 2 statements, the PRINT and the STOP. This is a final block (STOP ends the program);
- false leads to the common block after the IF
…

Visualize the CFG

As a first analysis tool, we can visualize the CFG. Inspecting the result of the next script will open a Roassal visualization on the CFG contained in the FASTFortranCFGVisitor.

FASTFortranCFGVisualization on: <aFASTFortranCFGVisitor>

For the program above, this gives the visualization below.

the dark dot is the starting block (note that it is a block and contains statements);
the hollow dots are final blocks;
it’s not the case here, but a block may also be start and final (if there are no conditional blocks in the program) and this would be represented by a “target”, a circle with a dot inside;
a grey square is a comon block;
a blue square is a conditional block;
hovering the mouse on a block will bring a pop up with the list of its statements (this relies on the FASTFortranExporterVisitor)

One can see that:

the start block has 2 associated statements (PRINT and READ);
there are several final blocks, due to the STOP statements;
there is a loop at the bottom left of the graph where the last blue conditional block is “IF (IB.NE.0)” and the last statement of the grey block (true value of the IF), is a GOTO.

Other analyses

There are little analyses for now on the CFG, but FASTFortranCFGChecker will compute a list of unreachableBlocks that would represent dead code.

Control flow graphs may also be used to do more advanced analyses and possibly refactor code. For example, we mentioned the loop at the end of our program implemented with a IF statement and a GOTO. This could be refactored into a real WHILE loop that would be easier to read.

This is left as an exercise for the interested people 😉

Adapting to other languages

Building a control flow graph is language dependant to identify the conditional statements, where they lead, and the final statements.

But much could be done in FAST core based on FASTTReturnStatement and a (not yet existing at the time of writing) FASTTConditionalStatement.

Inspiration could be taken from FASTFortranCFGVisitor and the process is not overly complicated. It would probably be even easier for modern languages that do not have the various GOTO statements of Fortran.

Once the CFG is computed, the other tools (eg. the visualization) should be completely independant of the language.

All hands on deck!

Modular Moose | Blog

Building and sharing Moose models with MooseNexus

What MooseNexus does

Build a model with the API

1. Load MooseNexus

2. Identify the source project

3. Describe the build

4. Execute the build

5. Load the model in Moose

Build from the command line

What an image artifact contains

Reuse an image artifact

Share artifacts through a registry

API or CLI?

Setup MooseCI on your repository with GitHub Actions

Setup MooseCI on your repository with GitHub Actions

What is MooseCI?

Run MooseCI with Docker

Run MooseCI in a CI with GitHub Actions

Example with VerveineJ

Use MooseCI in another CI

Conclusion

Implementing a Static Single Assigment (SSA) in FAST-Python

Implementing the SSA of Python in FAST-Python

What is SSA?

Architecture: one visitor on top of the CFG

The state

The three SSA value classes

The links carried on the model

The pipeline inside resolve:

Producing the SSA versions

The core: visitFASTTCanBeVariable:

Creating a version

Reordering the visits: the order matters for SSA

Assignments: the right side comes first

Function definitions: parameters before the body

Definitions rebuild their own CFG

Building the phi versions

Step 1 — snapshot before the branches

Step 2 — record the new versions of each branch

Step 3 — build the phis and merge

Traced example

Querying the result

Known limits

Quick start

Advice for implementing SSA in your own FAST project

Moving this implementation to FAST

Implementing the Local Resolver of Python in FAST-Python

Implementing the Local Resolver of Python in FAST-Python

What is local resolution?

The API

Implementation: a Moose visitor

The Python problem: there is no declaration of variables

A scope stack

The core operations

What happens when no declaration is found

The heart: visitFASTTCanBeVariable:

Reordering the visit, or dealing with Python scoping

Block scoping is inconsistent: if and while

for loops leak their variable

Function and method definitions: parameters are declarations

Comprehensions: a scope that leaks itself, but not its body

Other constructs that declare names

global and nonlocal statements

Shadowing in Python

The rule we implemented

The shadowing API

A caution about “same kind”

Known limits and what is left to do

Quick start

Advice for implementing it in your own FAST project

Mixing up analysis of Project Architecture (Famix) and its history (Mining Software Repository)

Creating the Famix model

Retrieving the history of the project

Connecting the models

Building an ownership cartography of a system - Part 2

One color for each developer

Propagate to Distribution Map

Building an ownership cartography of a system

Context

The pipeline inside `resolve:`

The core: `visitFASTTCanBeVariable:`

The heart: `visitFASTTCanBeVariable:`

Block scoping is inconsistent: `if` and `while`

`for` loops leak their variable

`global` and `nonlocal` statements

Speed up models creation: application to JSON/MSE parsing

The `FamixCimporter` class

1. `treeSitterLanguage` method

2. `visitorClass` method

3. `importFileReference:` method

The `FamixCVisitor` class

`modelClass` method