Unity (and its GC) is not representative of the performance you get with CoreCLR.
The article discusses ref lifetime analysis that does have relationship with GC, but it does not force you into using one. Byrefs are very special - they can hold references to stack, to GC-owned memory and to unmanaged memory. You can get a pointer to device mapped memory and wrap it with a Span<T> and it will "just work".
Well, when I worked in Unity I used to compile C# code with the LLVM backend. It was as fast as C++ code would be. So Unity is perhaps an example in favor of C#.
The article discusses ref lifetime analysis that does have relationship with GC, but it does not force you into using one. Byrefs are very special - they can hold references to stack, to GC-owned memory and to unmanaged memory. You can get a pointer to device mapped memory and wrap it with a Span<T> and it will "just work".