This is a talk we gave at the "Compilers for Machine Learning" workshop at CGO this past Sunday. Obviously the slides are missing some of the context of the presentation, but the core argument is that one shouldn't build compilers for machine learning, but rather build general purpose compilers that are flexible and extensible. From there, you can easily get all the same benefits as dedicated machine learning compilers without building a huge monolith. The rest of the slides highlight some of the work done in the Julia community to work in this direction.
Reproducing it here:
The Compilers for Machine Learning workshop was recently held at CGO 2019. Since compiler techniques affect a large part of the machine learning stack, this workshop aimed to highlight research that incorporates compiler techniques and algorithms in optimizing machine learning workloads. The workshop included talks from various projects - Julia (Julia Computing), TVM (UW), Glow (Facebook), XLA (Google), nGraph (Intel), TensorRT (Nvidia), and the soon to release MLIR (Google).
Our talk introduced the abstractions in the Julia language and the kind of compiler transforms involved in implementing them. We then had a deep dive into dynamic semantics + static analysis - our JAOT (Just-Ahead-Of-Time) analysis. Building on these capabilities, the Zygote system implements automatic differentiation, effectively treating it as a compiler problem, giving us differentiable programming for free. Finally, compiler backends for GPUs and TPUs give us high performance execution. All this comes together beautifully in Neural ODEs, which we had to show off as our first slide!
Of course, our view is that Julia is one such language that people should consider seriously. The talk linked here is a peek under the hood and shows that differentiable programming in Julia is not a special add-on, but something that fits naturally within the language.
A long time ago, Yann LeCun wrote Lush, which was a numerically-focused Lisp dialect, with a focus on C interop. It might be one of Julia's closest sibling.
> Lush is an object-oriented programming language designed for researchers, experimenters, and engineers interested in large-scale numerical and graphic applications. Lush is designed to be used in situations where one would want to combine the flexibility of a high-level, weakly-typed interpreted language, with the efficiency of a strongly-typed, natively-compiled language, and with the easy integration of code written in C, C++, or other languages.
A Lisp with expressive macro support is at least what is needed. Nothing I know of nothing else that has the affordances to support internal DSLs and creative control flow.
Julia as hygienic macros. From their documentation [1]: "The strongest legacy of Lisp in the Julia language is its metaprogramming support. Like Lisp, Julia represents its own code as a data structure of the language itself."
Parts of Julia are even implemented in Lisp [2], although they tend to be ported to Julia now IIUC. But it's clear the Julia developers are well aware of Lisp and its strong points.
Is there a video of the talk? Would love to watch it. Congrats on the work, it looks great! I'm just missing more detail, since having just the slides is a bit on the dry side.
> How ... whatever was the reason for thinking that you needed a ML-specific compiler?
As the slides say, you need the program to be differentiable. Do general purpose compilers make it easy to automatically differentiate a program? No, not until very recent work. So people thought a better idea until we figured out how to do that was ML-specific compilers and frameworks.
> Do people also think you need a timecard-tracking-specific compiler?
No, nobody thinks this, because timecard-tracking does not need any special properties such as being differentiable.
The answer isn't as crazy as your extremely snarky question makes it out to be, is it?
Well, I do think there are some very real shortcomings that current general purpose compilers have when applied to machine learning (aggrevated by the fact that lots of machine learning code is written in languages with poor compiler support), so lots people looked at that, wrote small optimizers and got good performance. But then it turned out that researchers wanted to do more and more things in their ML models and these small optimizers are turning into full-blown compilers, with not a lot of thought about whether that is truly the correct thing to do.
do you know how modern ml works? do you know that every functional unit in a net needs to be differentiable and so needs to carry around either dual numbers (forward mode) or adjoints (reverse mode)? it's not as simple as just writing a math library.
Please give the parent a benefit of the doubt. Do you even comments are alienating. It could have been written as
Every functional unit in a net needs to be differentiable and so needs to carry around either dual numbers (forward mode) or adjoints (reverse mode). These requirements necessitate new languages and compilers that current imperative representations don’t currently support.
There's an extensive discussion of AD in the slides (the topic of this HN post), and how it is done in Julia. Precisely because it is not as simple as writing a math library is why you need language and compiler support.
