When he says "object serialization" is broken, does he mean Java's built-in object serialization (JOS), or the concept of serialization, independent of any specific implementation? He later says "default serialization", which is what JOS's serialization of an object is called if its serialization hasn't been customized - but maybe he means JOS itself?
Anyway: for JOS, you don't need to provide no-arg constructors and you don't need to un-final fields, because JOS extralinguistically both creates without constructors and sets final fields. JOS also provides hooks for you to initialize objects. It has many other other hooks that few people use.
It's true that some aspects of JOS are ugly in how it implements what it does, but much of what it sets out to do is necessary for a full-featured serialization, that can work over the network. Serialization in other languages hasn't shown a "right" way to do it that I'm aware of.
I'm interested to hear more about the problems Charles found with JOS, and whether he has misunderstandings about this arcane bit of java, or if it's me who's misunderstood his very brief aside on it here.
I mean the built-in serialization. Outside of the classloading and security hacks required to make it work, the fact is that it performs at its worst if you don't do things like provide a no-arg constructor and non-final fields. In those cases, the amount of reflective hackery required behind the scenes is absolutely dreadful, and in benchmarking a simple graph of objects recently I saw that 99% of the time was spent doing reflective access. That's absurd.
Rewriting to use Externalizable was a painful process, but it was orders of magnitude faster than builtin serialization. The default serialization mechanism is basically unusably slow for any high-throughput purposes.
I have not seen the hooks you describe for user-driven initialization of classes, and unfortunately most of the resources I consulted online while trying to write fast deserialization logic recently didn't mention them either. Got a link? I'm certainly willing to learn what I'm doing wrong.
True, reflection is slower than regular access - though JOS does a fair bit of caching to avoid some of the cost (for when you serialize many instances of the same class). I think your experience with rewriting as Externalization shows that JOS is one of those performance vs. ease trade-offs.
BTW: Are you sure that no-arg constructors and un-finaling fields makes a significant performance difference? The only instance I've come across for this showed the (surprising) result that JOS's setting final fields is actually faster than reflective setting of non-final fields - but I haven't profiled that explicitly. I would expect deserialization with a no-arg to be slower, because then it has to call the constructor in addition to actually creating the object (allocating memory etc). But I haven't profiled this either.
You can customize the initialization of a class by creating a readObject() method for it, which is a sort of extra-linguistic constructor. It is confusingly named the same as the method that you call to start deserialization - but here it is a callback method that you write, that JOS itself will call:
private void readObject(ObjectInputStream ois)
throws IOException, ClassNotFoundException {
ois.defaultReadObject(); // "default" deserialization
// ... your initialization here ...
}
You can also read data from the stream explicitly, and set the class fields yourself - if so, you need to write a corresponding writeObject() method that writes the data to the stream. By avoiding reflection, this should be faster than the "default" (but again I haven't profiled this). If you don't provide a readObject, it will call defaultReadObject() for you by, you know, default. Same for writeObject().
There's also hooks for validating the object graph after all of it has been deserialized; and for replacing (resolving) one object with another.
I know something of JOS, but not much of its performance, which seems to be your crucial requirement. So I might not be much help, but let me know if you'd like more info or if I've misunderstood something (I might not reply til tomorrow - it's 3am here).
Ahh, I see where the confusion comes from. I meant initializing in the Ruby way...which is analogous to construction. There's no way for you to specify how to construct the object being deserialized from the stream, and so you need a no-arg version of the constructor (or can allow JOS to generate one for you) and have to do the logic that would be in-constructor in a separate piece of code like readObject. And in my latest experiments with Externalizable, the JOS code being unable to construct objects the way I want it to has become the latest bottleneck.
It also interferes with us serializing objects for which we do want to have final fields initialized on construction. If we want to avoid the reflective construction, we need a no-arg constructor. To have a no-arg constructor, we can't make important fields final. And if we want a particular value to be passed into the constructor, we always need to do it in readObject, without any context provided as to where and when serialization is being called. That's so cumbersome that we simply can't do it.
To check I understand: you need to deserialize objects, and you also want to pass in some arguments to some of those objects, so that their initialization is based on both the serialized data and the arguments? Is that to do with currying (is currying in Ruby?) In a complex object graph, how do you know which arguments should go to which objects?
You could use a global object to hold these arguments (or, subclass Thread, and associate the data with that, and yield while the deserialization runs in that thread). Ugly, yes.
BTW: it is possible to set final fields outside a constructor, using some black hacking (by accessing the same hooks that JOS uses internally). A few serialization tools use this, and I recall a project that consisted entirely of providing nicer access to these hooks (I searched for it, but couldn't find it).
BTW: There's a super-fast approach to serialization involving pointer twiddling: you save the raw memory, and when you load it back, just adjust the pointers for its new location (heard from Andy Hunt).
This doesn't help you if you want to serialize in order to inspect the serialized form; and it throws away Java's lispy symbolic bindings; and it would have to be implemented within the JVM - but it is fast.
That's exactly the sort of thing I was hoping to see out of the JVM's default serialization. As it stands I think I'm going to basically have to define a custom marshaling format and hand-serialize everything for the domain I'm interested in (which, btw, is serializing pre-parsed Ruby code to disk so it can be loaded later without parse time...done to avoid the slowness of our parser during the cold first seconds of the JVM!).
Writing a de/serializer is pretty straightforward, the only slightly tricky bit is handling graphs (you record each object, associated with a label; when you encounter it again, you serialize a ref symbol to its label). It sounds like a whole project, but it's only for your specific use, it wouldn't be that much work, and definitely be worth it.
But hand-serializing everything is a serious grind. Would you use a code generator (or use BCEL or similar) for the default case? I bet the vast majority of cases are automatable, and for the other cases it's a starting point.
For cold start, I personally like the idea of reusing a JVM. It would be nice if it just had a reset or clear button. :-)
BTW: Even though serialization is just a tiny part of your project, you are right on top of it - I'm a little awed.
Thanks for this discussion, you helped inspire me to write my first Ruby program tonight (a regular expression engine) - it felt much lighter-weight than Java (but it's a bit difficult to judge, because it's a problem I'd previously worked out how to implement in Java).
Anyway: for JOS, you don't need to provide no-arg constructors and you don't need to un-final fields, because JOS extralinguistically both creates without constructors and sets final fields. JOS also provides hooks for you to initialize objects. It has many other other hooks that few people use.
It's true that some aspects of JOS are ugly in how it implements what it does, but much of what it sets out to do is necessary for a full-featured serialization, that can work over the network. Serialization in other languages hasn't shown a "right" way to do it that I'm aware of.
I'm interested to hear more about the problems Charles found with JOS, and whether he has misunderstandings about this arcane bit of java, or if it's me who's misunderstood his very brief aside on it here.