The most well known is a 100% Julia neural network library called Flux.jl [1], which aims to become what Swift for Tensorflow wants as well (to make the entire Julia language a fully differentiable language) through Zygote.jl [2], and even without it has already great integration with the ecosystem, for example with the differentiable equations library through DiffEqFlux.jl [3]. Plus the source code is very high level (while being high performance, including easy GPU support), so you can easily see what each component does and implement any extension directly on your code without worrying about performance.
There is also another feature complete native library that allows some very concise code, Knet.jl [4], and the Tensorflow bindings [5].
Neural differential equations are something that's worked out in Julia. Neural ordinary differential equations (ODEs), stochastic differential equations (SDEs), and delay differential equations (DDEs) can be found in this blog post (https://julialang.org/blog/2019/01/fluxdiffeq). Neural jump diffusions (jump SDEs) and neural partial differential equations (PDEs) are described here: http://www.stochasticlifestyle.com/. All of this is built on the Flux.jl machine learning library which is extremely flexible.
