Announcing Yaegi

Yet Another Go Interpreter

In this post, we present Yaegi, Yet Another Go Interpreter, with the E standing for Elegant, Embedded, Easy, or whatever you prefer.

Yaegi is an open source project developed by Containous, (the company behind Traefik and TraefikEE), to bring executable Go scripts, embedded plugins, interactive shells, and instant prototyping on top of the Go runtime. Yaegi project is hosted on GitHub.

Motivation

Despite being static and strongly typed, Go feels like a dynamic language. The standard library even provides the Go parser used by the compiler and the reflection system to interact dynamically with the runtime. So why not just take the last logical step and finally build a complete Go interpreter?

Programming languages for high level scripting and for low level implementation are usually different. This time, with Go, we have an opportunity to unify both. Imagine all the C/C++/Java fast libraries for Python being written in Python instead. That’s what Yaegi is for Go (or, the reverse). No burden due to syntax switch, no need to rewrite or modify slow code to make it fast, and full access to goroutines, channels, type safety, etc. at script level.

Goals and Priorities

• Simplicity: New(), Eval() and Use() is the only API. No external dependencies besides the standard Go library.
• Standard: the interpreter supports 100% of the Go Specification.
• Robustness: preserve type safety and runtime integrity: unsafe and syscallpackages are not used nor exposed by default. Security over performance.
• Interoperability: scripts can “import” pre-compiled runtime packages, but also compiled code can “import” script packages during execution. See usage examples below for more details.

Using Yaegi

As a Command-Line Interpreter

The Yaegi executable can interpret Go files or run an interactive Read-Eval-Print-Loop:

$ yaegi
> 1 + 2
3
> import "fmt"
> fmt.Println("Hello World")
Hello World
>

Yaegi works like go run (but faster). It also enables executable Go scripts (starting with #!). Yaegi provides a full Go environment with a complete standard library in a single standalone executable.

As an Embedded Interpreter

In the following example, an interpreter is created with New() and then it evaluates Go code using Eval(). Nothing surprising here.

package main

import (
    "github.com/containous/yaegi/interp"
    "github.com/containous/yaegi/stdlib"
)

func main() {
    i := interp.New(interp.Options{})
    i.Use(stdlib.Symbols)
    i.Eval(`import "fmt"`)
    i.Eval(`fmt.Println("hello")`)
}

This example demonstrates the ability to use executable pre-compiled symbols in the interpreter. Thanks to the statement i.Use(stdlib.Symbols), the interpreted import "fmt"will load the fmt package from the executable itself (wrapped in reflect.Values) instead of trying to parse source files.

Yaegi also provides the goexports command to build the binary wrapper of any package from its source. This is the command we used to generate all stdlib wrappers provided by default.

As a Dynamic Extension Framework

The program is compiled ahead of time, except for the function bar() that is interpreted from a script. The import process involves the following steps:

1. use of i.Eval(src) to internally compile the script package in the context of the interpreter
2. use of v,_ := i.Eval("foo.Bar") to get the symbol that we want to use as a reflect.Value (please forgive me for the missing error handling)
3. application of Interface()method and type assertion to convert the reflect.Value into a usable Go typed function

package main

import "github.com/containous/yaegi/interp"

const src = `package foo
func Bar(s string) string { return s + "-Foo" }`

func main() {
    i := interp.New(interp.Options{})
    i.Eval(src)
    v, _ := i.Eval("foo.Bar")
    bar := v.Interface().(func(string) string)
    
    r := bar("Kung")
    println(r)
    // Output:
    // Kung-Foo
}

What about Performance?

Interpreters are quite common in domains like gaming or science. But could Yaegi also be usable as a plugin engine in the context of distributed systems?

To answer that, we have benchmarked the use of gziphandler, a middleware for compressing HTTP responses. Gziphandler provides interface methods implementing compression for the HTTP server in the standard library. This demonstrates an interpreted dynamic processing inserted into a statically compiled server program, with a direct impact on latency and throughput.

In one case, gziphandler is compiled and used directly as a callback passed to http.Handle(). In the other, the interpreted version of gziphandler is used.

Benchmark      old ns/op   new ns/op    delta
Compress-8     769088      817632       +6.31%

We measured less than 10% of overhead for the interpreted version of gziphandler compared to the compiled version, which is to us an acceptable cost for to the benefits of dynamically extending foour server.

One important factor is that the gzip compression itself is provided by the standard lib, already compiled, and exposed to the interpreter by Use().

Interpreters work best at providing the glue language and offloading CPU intensive computations to optimized compiled libraries. With Go and Yaegi, this can be achieved simply and almost transparently.

Conclusion

We are announcing Yaegi, a Go interpreter. It’s a young project and as such, far from perfect. Yaegi is open source and needs your feedback and contributions. We hope to make it as useful as possible for the Go community and beyond. This is only the beginning. At Containous, we are focused on solving practical problems through innovative solutions. Stay tuned for upcoming announcements about both Yaegi and Containous.

Join us at github.com/containous/yaegi.

I want to thank Containous, Emile Vauge, Ludovic Fernandez, Mathieu Lonjaret who coined the name, Peka for the awesome logo, and many other colleagues for their support.