My biggest challenge with immer has been that my code aesthetic taste leans functional, but immer makes code look imperative even if it is ultimately functional. Always looks a little wrong to my eye!
Writing old-style pure code to do changes in a deeply nested structure is cumbersome, both to write and read. Eventually every language will find nicer ways to allow programmers to express these mutations.
JavaScript doesn't provide a nice way to extend the language syntax, unlike some other programming languages. So most DSLs end up looking like imperative code but do pure mutations underneath. I understand, for the reader it's confusing, because unless you know that it's a DSL you never know whether the mutations are pure or not.
Compare that with Haskell, with its support for custom operators, allows for some quite succinct code to express deep mutations (cf. lens library)