Any plans to get Julia to be fully self-hosting?

[Dzhelil Rufat]

As the maturity of a language is measured by whether it is implemented in
itself, I think it is important to get the Julia language to be
self-hosting.

My understanding is that the tokenizer and parser for julia are implemented
in a dialect of scheme called femtolisp? Are there any plans to migrate the
frontend wholly to Julia?

Additionally, for the backend, can one create julia bindings to llvm, and
then use those to write the backend completely in julia as well? What are
the obstacles to getting Julia to be fully self-hosting that right now? It
would be really great if we could one day have a julia language implemented
completely in julia.

[Stefan Karpinski]

This would definitely be good, but it's a lot of work with no user-facing
benefit, so it's a pretty low priority. If you're interested in working on
it, however, it would certainly be nice. I kind of suspect that being
self-hosting is mainly a measure of language maturity because only in
fairly mature languages have all the things more important than being
self-hosting already been taken care of.

Post by Dzhelil Rufat
As the maturity of a language is measured by whether it is implemented in
itself, I think it is important to get the Julia language to be
self-hosting.
My understanding is that the tokenizer and parser for julia are
implemented in a dialect of scheme called femtolisp? Are there any plans to
migrate the frontend wholly to Julia?
Additionally, for the backend, can one create julia bindings to llvm, and
then use those to write the backend completely in julia as well? What are
the obstacles to getting Julia to be fully self-hosting that right now? It
would be really great if we could one day have a julia language implemented
completely in julia.

[Jeff Bezanson]

I agree this would be cool, but it can't be the only measure of
language maturity --- not every language is designed for writing
compilers.

The main problem is that this involves lots of tedious work, after
which you end up with the same thing you started with, just
implemented differently. I'm not even sure I want to use LLVM in any
language but C++, since that's the language it's documented in and C++
APIs tend to be highly coupled to the language.

But, as our static compilation emerges and improves (thanks to
Jameson), and C++ interop emerges (thanks to Keno and Isaiah), this
may gradually become a reasonable thing to do. But we can't really
afford to prioritize being self-hosting for its own sake; the real
value is in these steps along the way (static compiler, C++ interop,
etc.)

Post by Dzhelil Rufat
As the maturity of a language is measured by whether it is implemented in
itself, I think it is important to get the Julia language to be
self-hosting.
My understanding is that the tokenizer and parser for julia are implemented
in a dialect of scheme called femtolisp? Are there any plans to migrate the
frontend wholly to Julia?
Additionally, for the backend, can one create julia bindings to llvm, and
then use those to write the backend completely in julia as well? What are
the obstacles to getting Julia to be fully self-hosting that right now? It
would be really great if we could one day have a julia language implemented
completely in julia.

[Ivar Nesje]

Can someone explain to me what the benefit of self-hosting a language is?

Links would also be helpful.

[Stefan Karpinski]

The biggest benefit for us would be that there would be a one-to-one correspondence between data types in the parser language and the program being parsed. That means you can parse literals as their values and represent expressions with expression objects. For example the effort to support 128-bit integer literals is somewhat complicated by the lack of 128-bit integer types in flisp. If the parser was written in Julia, you would have a natural representation for every possible kind of literal.

Post by Ivar Nesje
Can someone explain to me what the benefit of self-hosting a language is?
Links would also be helpful.

[Kevin Squire]

It would also make hacking on the parser easier. I actually like lisp, but
I don't make any claims about being good at it, and at this time, julia is
much easier to deal with.

Although, at this stage in language development, it might actually be
better that only a few people are hacking on the parser. ;-)

Kevin


On Sat, Nov 23, 2013 at 3:16 PM, Stefan Karpinski <

Post by Stefan Karpinski
The biggest benefit for us would be that there would be a one-to-one
correspondence between data types in the parser language and the program
being parsed. That means you can parse literals as their values and
represent expressions with expression objects. For example the effort to
support 128-bit integer literals is somewhat complicated by the lack of
128-bit integer types in flisp. If the parser was written in Julia, you
would have a natural representation for every possible kind of literal.

Post by Ivar Nesje
Can someone explain to me what the benefit of self-hosting a language is?
Links would also be helpful.

[Gour]

On Sat, 23 Nov 2013 17:03:38 -0500

Post by Jeff Bezanson
But, as our static compilation emerges and improves (thanks to
Jameson), and C++ interop emerges (thanks to Keno and Isaiah), this
may gradually become a reasonable thing to do.

What is the current status and/or estimation when Julia might provide
statically compiled executables?



Sincerely,
Gour

[Tobi]

Post by Gour
What is the current status and/or estimation when Julia might provide
statically compiled executables?

What is the main motivation to have binary executables? Using the C-API it
should be quite simple to write a small C program that links to libjulia
and executes some code.

[Gour]

On Sun, 16 Feb 2014 13:45:47 -0800 (PST)

Post by Tobi
What is the main motivation to have binary executables? Using the
C-API it should be quite simple to write a small C program that links
to libjulia and executes some code.

I am thinking about statically compiled and linked application -
convenience is certainly one aspecti when one wants to deliver e.g.
complete GUI app, but I guess it by using type-annotations, one should
achieve even better performance?


Sincerely,
Gour

[Stefan Karpinski]

This is not going to happen any time soon – there are a lot of fairly large
steps to get to fully static compilation. It will, however, happen.

Post by Gour
On Sun, 16 Feb 2014 13:45:47 -0800 (PST)

Post by Tobi
What is the main motivation to have binary executables? Using the
C-API it should be quite simple to write a small C program that links
to libjulia and executes some code.

I am thinking about statically compiled and linked application -
convenience is certainly one aspecti when one wants to deliver e.g.
complete GUI app, but I guess it by using type-annotations, one should
achieve even better performance?
Sincerely,
Gour
--
A self-realized man has no purpose to fulfill in the discharge
of his prescribed duties, nor has he any reason not to perform
such work. Nor has he any need to depend on any other living being.
http://www.atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810

[Gour]

On Sun, 16 Feb 2014 18:23:25 -0500

This is not going to happen any time soon – there are a lot of fairly
large steps to get to fully static compilation.

Do you mean it's post-1.0 target?

It will, however, happen.

Very nice.


Sincerely,
Gour

[Tobi]

Post by Gour
I am thinking about statically compiled and linked application -
convenience is certainly one aspecti when one wants to deliver e.g.
complete GUI app, but I guess it by using type-annotations, one should
achieve even better performance?

No, it won't be faster in general. How can it be as Julia is almost as fast
as C?
Furthermore I have doubts that a statically compiled binary application is
more convenient.
We certainly need an easy way to distribute Julia applications but static
compilation is IMHO not the crutial thing to get there.
Any application I can think of will use some packages and the question is
more how to get all these things together and install
it on a user PC.

That beeing said there currently is one performance bottleneck which is
package loading time. For the base module this has been
solved by static compilation of the system image (sys.dll/dylib/so). For
fast package loading time one can extend the system
image by including packages in it (This is very recent work!). But this is
only a stopgap solution. At some point we can hopefully
precompile packages.

[Tobi]

P.S.: Would be also interesting what you are planing to do. Should your
application be cross platform?

Post by Tobi

Post by Gour
I am thinking about statically compiled and linked application -
convenience is certainly one aspecti when one wants to deliver e.g.
complete GUI app, but I guess it by using type-annotations, one should
achieve even better performance?

No, it won't be faster in general. How can it be as Julia is almost as
fast as C?
Furthermore I have doubts that a statically compiled binary application is
more convenient.
We certainly need an easy way to distribute Julia applications but static
compilation is IMHO not the crutial thing to get there.
Any application I can think of will use some packages and the question is
more how to get all these things together and install
it on a user PC.
That beeing said there currently is one performance bottleneck which is
package loading time. For the base module this has been
solved by static compilation of the system image (sys.dll/dylib/so). For
fast package loading time one can extend the system
image by including packages in it (This is very recent work!). But this is
only a stopgap solution. At some point we can hopefully
precompile packages.

[Gour]

On Mon, 17 Feb 2014 00:54:58 -0800 (PST)

Post by Tobi
P.S.: Would be also interesting what you are planing to do. Should
your application be cross platform?

Yes, I'll develop on Free/PC-BSD, but would like to be able to provide
application for both Mac OS & Windows users...however, here there is a
question of GUI bindings, but that was already addressed here.


Moreover, yesterday I've found out that Qt bindings is one of the ideas
for Nimrod's GSoC projects, so I'll keep an eye on that language as
well.


Sincerely,
Gour

[Tobi]

If you want to make a distributable cross-platform app, you will have to
make custom installers for Windows and
OSX. This is some effort but justified if the project is not to small. But
the installer will have to handle not only
Julia (or Nimrod) but also the Qt libraries if you want to use that toolkit.

As your most important thing seems to be that you are using Qt, you could
also use C++ for your GUI and use
embedded Julia for the computations. In the current state this seems to be
the most robust solution.

Post by Gour
On Mon, 17 Feb 2014 00:54:58 -0800 (PST)

Post by Tobi
P.S.: Would be also interesting what you are planing to do. Should
your application be cross platform?

Yes, I'll develop on Free/PC-BSD, but would like to be able to provide
application for both Mac OS & Windows users...however, here there is a
question of GUI bindings, but that was already addressed here.
Moreover, yesterday I've found out that Qt bindings is one of the ideas
for Nimrod's GSoC projects, so I'll keep an eye on that language as
well.
Sincerely,
Gour
--
An intelligent person does not take part in the sources of misery,
which are due to contact with the material senses. O son of Kuntī,
such pleasures have a beginning and an end, and so the wise man does
not delight in them.
http://www.atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810

[Gour]

On Mon, 17 Feb 2014 02:11:36 -0800 (PST)

Post by Tobi
If you want to make a distributable cross-platform app, you will have
to make custom installers for Windows and OSX. This is some effort but
justified if the project is not to small.

I'm aware of it considering that app might be interesting for not so
tech savvy users and Mac/Windows users are accustomed to install and not
to build their apps.

Post by Tobi
But the installer will have to handle not only Julia (or Nimrod) but
also the Qt libraries if you want to use that toolkit.

Yes, but it's the same no matter which GUI toolkit one uses.

Post by Tobi
As your most important thing seems to be that you are using Qt, you
could also use C++ for your GUI and use embedded Julia for the
computations. In the current state this seems to be the most robust
solution.

I'd simply like to avoide using C(++) and focus on my problem using
higher-level language which provides enough speed.

As far as Qt is concerned, wx is also viable alternative, but Qt seems
to be the best option today for multi-platform app due to having much
more developing activity than wx and, as I wrote earlier, wherever one
goes, nobody recommends using GTK for anything except Linux...actually I
just hear the word 'pain' mentioned often in regard.


Sincerely,
Gour

[Tobi]

Post by Gour
I'd simply like to avoide using C(++) and focus on my problem using
higher-level language which provides enough speed.

You will always have to make compromises. The holy grail is not there yet.
If your project starts now, you have to make a decision. And if you want to
use
Julia the options are:

- Qt(C++) with embedded Julia
- Write Qt bindings for Julia and make all in Julia
- Use Qt with Python and the Python Julia bindings
- Use Gtk.jl
- Use Tk.jl

Each of these has another downside. You have to decide yourself, what
compromises you want to make.

Post by Gour
As far as Qt is concerned, wx is also viable alternative, but Qt seems
to be the best option today for multi-platform app due to having much
more developing activity than wx and, as I wrote earlier, wherever one
goes, nobody recommends using GTK for anything except Linux...actually I
just hear the word 'pain' mentioned often in regard.

Gour, we have understand that you are a Qt fan, and actually I like it too.
But please avoid saying insulting things about Gtk here. One of the core
Julia
programmer (Jameson) has done awesome work to implement Gtk bindings and
I have done quite some work to fill missing pieces. julia-dev is not the
place to
bash others technology.

[Gour]

On Mon, 17 Feb 2014 03:13:02 -0800 (PST)

Post by Tobi
Gour, we have understand that you are a Qt fan, and actually I like
it too. But please avoid saying insulting things about Gtk here. O

I'm not Qt fan, but just (trying to be) pragmatic here.

Most of my Linux days I spent with GTK technology and would be happy if
it would be in better shape multi-platform-wise.

Post by Tobi
of the core Julia programmer (Jameson) has done awesome work to
implement Gtk bindings and I have done quite some work to fill missing
pieces.

It was certainly not meant to offend anyone personally and it's
glorious to work & provide *any* GUI bindings for Julia making it
attractive as 'general purpose' language'.

Post by Tobi
julia-dev is not the place to bash others technology.

I apologize for it. It was never my intention.


Sincerely,
Gour

[Gour]

On Mon, 17 Feb 2014 00:53:04 -0800 (PST)

Post by Tobi
No, it won't be faster in general. How can it be as Julia is almost
as fast as C?

Well, afaik, it's still in the range of 2X.

Post by Tobi
Furthermore I have doubts that a statically compiled binary
application is more convenient.

I did not express clearly - I was not thinking about statically compiled
and linked application, but about the speed difference between using JIT
compiler and native code generation as it is e.g. between Julia and
Nimrod.

For other external libs, I do not mind using share libs avaialble on my
native OS.


Sincerely,
Gour
--
As the embodied soul continuously passes, in this body,
from boyhood to youth to old age, the soul similarly passes
into another body at death. A sober person is not bewildered
by such a change.

http://www.atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810

[Tobi]

Post by Gour
Well, afaik, it's still in the range of 2X.

My experience is that it is most of the time a factor of about 1.2. One
often has to do some manual tweeking though (i.e. us @inbounds).
When the autovectorizer of gcc/llvm/intel kicks in the factor might be
larger. But autovectorization will also land in Julia (there is already a
pull request for that). Static compilation will not change the execution
speed.

Post by Gour
I did not express clearly - I was not thinking about statically compiled
and linked application, but about the speed difference between using JIT
compiler and native code generation as it is e.g. between Julia and
Nimrod.

I don't want to start any comparison with Nimrod here (which looks like a
really cool language by the way). But before thinking about whether
a statically compiled application is faster than a JIT compiled, one has to
benchmark it. gcc can statically compile Java and it is not faster
than the regular JIT compiled Java.

The nice thing about Julia is that it uses the llvm JIT compiler which
allows to dump generated functions into a shared library. This is done
in the 0.3 release and leads to a blazing fast startup. In the future we
will see more of this for modules and maybe in the end even for Julia
scripts.
This is a non-trivial process but I am totally impressed with the
developement speed of Julia. Having statically compiled system image is a
major achievment.

[Gour]

On Mon, 17 Feb 2014 05:23:56 -0800 (PST)

Post by Tobi
My experience is that it is most of the time a factor of about 1.2.

That's really great and much more than I could expect.

Post by Tobi
Static compilation will not change the execution speed.

Yeah, I got some explanation in #nimrod why it may be so for Julia.


Sincerely,
Gour

[Francesco Bonazzi]

Post by Tobi
What is the main motivation to have binary executables? Using the C-API it
should be quite simple to write a small C program that links to libjulia
and executes some code.

Well, there is a good reason indeed. Companies wishing to produce non-free
closed-source software would prefer to distribute binary-compiled programs,
in order to avoid their code to be stolen.

There are lots of companies developing in Java who choose to use compilers
to machine code instead of the JVM for their final product, for the same
reason (Java bytecode is easily decompilable).

I agree that having compilation to binary executable would be of small
advantage for open source projects (maybe there's some minor advantage on
resource-limited devices?).

[Stefan Karpinski]

On Mon, Feb 17, 2014 at 1:01 PM, Francesco Bonazzi

Post by Francesco Bonazzi

Post by Tobi
What is the main motivation to have binary executables? Using the C-API
it should be quite simple to write a small C program that links to libjulia
and executes some code.

Well, there is a good reason indeed. Companies wishing to produce non-free
closed-source software would prefer to distribute binary-compiled programs,
in order to avoid their code to be stolen.
There are lots of companies developing in Java who choose to use compilers
to machine code instead of the JVM for their final product, for the same
reason (Java bytecode is easily decompilable).
I agree that having compilation to binary executable would be of small
advantage for open source projects (maybe there's some minor advantage on
resource-limited devices?).

This is certainly a use case. I'm sure you can see, however, why this isn't
a very compelling use case for people donating their efforts to an open
source project. There's not a lot of motivation to implement very tricky
features for free so that other people can make more money.

[Tobi]

Post by Stefan Karpinski
This is certainly a use case. I'm sure you can see, however, why this
isn't a very compelling use case for people donating their efforts to an
open source project. There's not a lot of motivation to implement very
tricky features for free so that other people can make more money.

Stefan, you are absolutely right that this does not seem to be of interest
for the core Julia community on a first sight. My feeling as someone
working in a company and still contributing to Julia core and Gtk.jl is
that hiding the code is yet another big feature that Julia will have. And I
am further quite confident that this feature will actually help Julia a
lot. When I would distribute a closed source application I would want to
hide a) my secret algorithm and b) the boring but development intensiv
application logic. However, there is also c) which is the standard
libraries (i.e. packages). And for these I would always contribute upstream.

When module precompilation lands this is IMHO a real selling point for
Julia and this will make it very attractive for many companies that
currently avoid languages like Python and Java.

[Stefan Karpinski]

That may well be. Personally, I think fully static compilation is primarily
attractive because it drastically simplifies application deployment: put a
single pre-compiled binary on a server and run it – end of story.

Post by Stefan Karpinski
This is certainly a use case. I'm sure you can see, however, why this

Post by Stefan Karpinski
isn't a very compelling use case for people donating their efforts to an
open source project. There's not a lot of motivation to implement very
tricky features for free so that other people can make more money.

Stefan, you are absolutely right that this does not seem to be of interest
for the core Julia community on a first sight. My feeling as someone
working in a company and still contributing to Julia core and Gtk.jl is
that hiding the code is yet another big feature that Julia will have. And I
am further quite confident that this feature will actually help Julia a
lot. When I would distribute a closed source application I would want to
hide a) my secret algorithm and b) the boring but development intensiv
application logic. However, there is also c) which is the standard
libraries (i.e. packages). And for these I would always contribute upstream.
When module precompilation lands this is IMHO a real selling point for
Julia and this will make it very attractive for many companies that
currently avoid languages like Python and Java.

[Gour]

On Mon, 17 Feb 2014 16:52:33 -0500

Post by Stefan Karpinski
That may well be. Personally, I think fully static compilation is
primarily attractive because it drastically simplifies application
deployment: put a single pre-compiled binary on a server and run it –
end of story.

I agree, especially for end-users on the OS-es where that is a
common practice for deploying.


Sincerely,
Gour

[Fil Mackay]

Post by Stefan Karpinski
That may well be. Personally, I think fully static compilation is
primarily attractive because it drastically simplifies application
deployment: put a single pre-compiled binary on a server and run it – end
of story.

Would the static executable be purely for deployment convenience, ie. would
JIT at runtime identical to the current model by just embedding the AST (or
text?) of the code in the executable file?

[Stefan Karpinski]

In that case, you'd probably want to pre-compile everything. There are
other situations where you'd want to keep the JIT capability around.

Post by Fil Mackay

Post by Stefan Karpinski
That may well be. Personally, I think fully static compilation is
primarily attractive because it drastically simplifies application
deployment: put a single pre-compiled binary on a server and run it – end
of story.

Would the static executable be purely for deployment convenience, ie.
would JIT at runtime identical to the current model by just embedding the
AST (or text?) of the code in the executable file?

[Fil Mackay]

Post by Stefan Karpinski
In that case, you'd probably want to pre-compile everything. There are
other situations where you'd want to keep the JIT capability around.

I presume you could pre-JIT what you can, but still be able to JIT at
runtime if required. The stuff I work on is all metaprogramming so relies
on runtime eval. Deferring this to runtime JIT also ensures you can
optimise to the actual CPU.

[Steven G. Johnson]

Post by Stefan Karpinski
In that case, you'd probably want to pre-compile everything. There are
other situations where you'd want to keep the JIT capability around.

That's what I was thinking of as static compilation: Precompile as much
as possible, launch a small executable that loads this and goes, but still
link to libjulia (including LLVM). I don't think shipping libjulia is too
much of a burden (unless you're on a tiny embedded system), as long as
things are mostly precompiled so that you don't have a big startup latency.

Analyzing code to determine whether it requires runtime JIT is a much
harder problem, and I don't see that being solved for a while.

[Tobi]

Post by Steven G. Johnson
That's what I was thinking of as static compilation: Precompile as much
as possible, launch a small executable that loads this and goes, but still
link to libjulia (including LLVM). I don't think shipping libjulia is too
much of a burden (unless you're on a tiny embedded system), as long as
things are mostly precompiled so that you don't have a big startup latency.

I second that. One has to ship some shared libraries (BLAS/LAPACK) anyway.
Furthermore, when module precompilation lands one presumely haa to ship one
more shared library for each package one is using.

[Jameson Nash]

How does this differ from what we do now with sys.so?

Post by Tobi

Post by Steven G. Johnson
That's what I was thinking of as static compilation: Precompile as much
as possible, launch a small executable that loads this and goes, but still
link to libjulia (including LLVM). I don't think shipping libjulia is too
much of a burden (unless you're on a tiny embedded system), as long as
things are mostly precompiled so that you don't have a big startup latency.

I second that. One has to ship some shared libraries (BLAS/LAPACK) anyway.
Furthermore, when module precompilation lands one presumely haa to ship one
more shared library for each package one is using.

[Tobi]

Post by Jameson Nash
How does this differ from what we do now with sys.so?

Not at all. I know that it is now possible to extend the sys.so with
packages. But it seems to be cleaner when we have one .so per
package/module.

This is what I understand under static compilation. The last step is to
compile Julia scripts to binary applications. But IMHO this is not the most
important thing. One can put everything in modules and only provide a
trivial loader script.

[Steven G. Johnson]

Post by Jameson Nash
How does this differ from what we do now with sys.so?

It might be nice to avoid the julia executable entirely, and provide an
easy way to generate a little loader executable. But you're right that a
trivial little shell script which calls /usr/bin/julia -J with a custom
sysimage file should be sufficient.

[Jiahao Chen]

Post by Dzhelil Rufat
As the maturity of a language is measured by whether it is implemented in
itself, I think it is important to get the Julia language to be
self-hosting.

Counterexample: Fortran.

[Stefan Karpinski]

While being self-hosting is not essential, I do think it would be really convenient, so I don't want to discourage any work in that direction. Homoiconicity makes it particularly handy since the parser can literally just generate Expr objects and lowering passes can then work with those, much like our type inference code does already.

Post by Jiahao Chen

As the maturity of a language is measured by whether it is implemented in itself, I think it is important to get the Julia language to be self-hosting.

Counterexample: Fortran.

[Jeff Bezanson]

The parser already does generate Expr objects, via the C shim in
ast.c. And if we wanted to support more types of literals,
julia-parser.scm could return expressions like (int128 lo-bits
hi-bits), which would also be translated by this shim. Nobody ever
sees this intermediate stage except the shim, so while this is
slightly clumsy it's not a serious limitation.

On Mon, Nov 25, 2013 at 8:43 AM, Stefan Karpinski

Post by Stefan Karpinski
While being self-hosting is not essential, I do think it would be really
convenient, so I don't want to discourage any work in that direction.
Homoiconicity makes it particularly handy since the parser can literally
just generate Expr objects and lowering passes can then work with those,
much like our type inference code does already.

Post by Dzhelil Rufat
As the maturity of a language is measured by whether it is implemented in
itself, I think it is important to get the Julia language to be
self-hosting.

Counterexample: Fortran.

[Stefan Karpinski]

Well, yes, if you include ast.c in "the parser", but I was considering only
the Scheme code to be "the parser" proper.

Post by Jeff Bezanson
The parser already does generate Expr objects, via the C shim in
ast.c. And if we wanted to support more types of literals,
julia-parser.scm could return expressions like (int128 lo-bits
hi-bits), which would also be translated by this shim. Nobody ever
sees this intermediate stage except the shim, so while this is
slightly clumsy it's not a serious limitation.
On Mon, Nov 25, 2013 at 8:43 AM, Stefan Karpinski

Post by Stefan Karpinski
While being self-hosting is not essential, I do think it would be really
convenient, so I don't want to discourage any work in that direction.
Homoiconicity makes it particularly handy since the parser can literally
just generate Expr objects and lowering passes can then work with those,
much like our type inference code does already.

Post by Dzhelil Rufat
As the maturity of a language is measured by whether it is implemented

in

Post by Stefan Karpinski

Post by Dzhelil Rufat
itself, I think it is important to get the Julia language to be
self-hosting.

Counterexample: Fortran.