this is highly workload-dependent. there are plenty of APIs that are multiple-factor faster and 10x more memory efficient due to native implementation.
for ASCII text, mine finishes in 80ms, while pretext takes 2200ms. i haven't yet checked pretext for accuracy (how closely it matches the browser), but will test tonight - i expect it will do well.
let's see how close pretext can get to 80ms (or better) without adopting the same tricks.
Looks like uWrap only handles latin characters and doesn't deal with things like soft hyphens or emoji correction, plus uWrap only handles white-space: pre-line while Pretext doesn't handle pre-line but does handle both normal and pre-wrap.
correct, it was meant for estimating row height for virtualizing a 100k row table with a latin-ish LTR charset (no emoji handling, etc). its scope is much narrower. still, the difference in perf is significant, which i have found to be true in general of AI-generated geenfield code.
I've worked with text and in my experience all of these things (soft hyphens, emoji correction, non-latin languages, etc) are not exceptions you can easily incorporate later, but rather the rules that end up foundational parts of the codebase.
That is to say, I wouldn't be so quick to call a library that only handles latin characters comparable to one that handles all this breath of things, and I also wouldn't be so quick to blame the performance delta on the assumption of greenfield AI-generated code.
no disagreement. i never claimed uWrap did anything more than it does. it was not meant for typography/text-layout but for line count estimation. it never needed to be perfect, and i did not claim equivalence to pretext. however, for the use case of virtualization of data tables -- a use case pretext is also targeted at -- and in the common case of latin alphabets, right now pretext is significantly slower. i hope that it can become faster despite its much more thorough support for "rest of the owl".
i don't have a mac to test this with currently, so hopefully it's not the price but a matter of adding a Safari-specific adjustement :)
internally it still uses the Canvas measureText() API, so there's nothing fundamentally that should differ unless Safari has broken measureText, which tbh, would not be out of character for that browser.
prepare uses measure text, if it is in a for loop, it won't be fast. This library is meant to do prepare once and then layout many times. layout calls should be sub-1 ms.
it is not clear from the API/docs how i would use prepare() once on one text and then use layout() for completely different text.
i think the intended purpose is that your text is maybe large but static and your layout just changes quickly. this is not the case for figuring out the height of 100k rows of different texts in a table, for example.
I think for that to use pretext is to join each row with hard line break and then do prepare once, then walk each line. At least that will put the single layout performance into the best light.
I am skeptical getting row height of many items only once is the intended behavior though. It is probably the intended behavior to get row height of many items and enables you to resizing width many time later (which is pretty useful on desktop).
There's a handful of perf related PRs open already so maybe it will be faster soon. I'm sure with enough focus on it we could have a hyper optimized version in a few hours.
Hah - that is exactly what I did. Someone asked me this question and after 5 minutes in the weeds of the debounce on the mouse click they said "look all we wanted was to find out if you'd ever heard of DNS, let's move on, that was great".
agreed, heatmaps with logarithmic cell intensity are the way to go for massive datasets in things like 10,000-series line charts and scatter plots. you can generally drill downward from these, as needed.
uPlot maintainer here. this looks interesting, i'll do a deeper dive soon :)
some notes from a very brief look at the 1M demo:
- sampling has a risk of eliminating important peaks, uPlot does not do it, so for apples-to-apples perf comparison you have to turn that off. see https://github.com/leeoniya/uPlot/pull/1025 for more details on the drawbacks of LTTB
- when doing nothing / idle, there is significant cpu being used, while canvas-based solutions will use zero cpu when the chart is not actively being updated (with new data or scale limits). i think this can probably be resolved in the WebGPU case with some additional code that pauses the updates.
- creating multiple charts on the same page with GL (e.g. dashboard) has historically been limited by the fact that Chrome is capped at 16 active GL contexts that can be acquired simultaneously. Plotly finally worked around this by using https://github.com/greggman/virtual-webgl
> data: [[0, 1], [1, 3], [2, 2]]
this data format, unfortunately, necessitates the allocation of millions of tiny arrays. i would suggest switching to a columnar data layout.
Really appreciate you taking the time to look, Leon - uPlot has been a huge inspiration for proving that browser charts don't have to be slow.
Both points are fair:
1. LTTB peak elimination - you're right, and that PR is a great reference. For the 1M demo specifically, sampling is on by default to show the "it doesn't choke" story. Users can set sampling: 'none' for apples-to-apples comparison. I should probably add a toggle in the demo UI to make that clearer.
2. Idle CPU - good catch. Right now the render loop is probably ticking even when static. That's fixable - should be straightforward to only render on data change or interaction. Will look into it.
Would love your deeper dive feedback when you get to it. Always more to learn from someone who's thought about this problem as much as you have.
Blind sampling like this makes it useless for real-world statistics of the kind your users care about.
And column-oriented data is a must. Look at Rlang's data frames, pandas, polars, numpy, sql, and even Fortran's matrix layout.
Also need specialized expicitly targetable support for Float32Array and Float64Array. Both API and ABI are necessary if you want to displace incumbents.
There is huge demand for a good web implementation. This is what it takes.
I once had to deal with many million data points for an application. I ended up mip-mapping them client-side.
But regarding sampling, if it's a line chart, you can sample adaptively by checking whether the next point makes a meaningfully visible difference measured in pixels compared to its neighbours. When you tune it correctly, you can drop most points without the difference being noticeable.
I didn't find any else doing that at the time, and some people seemed to have trouble accepting it as a viable solution, but if you think about it, it doesn't actually make sense to plot say 1 million points in a line chart 1000 pixels wide. On average that would make 1000 points per pixel.
We routinely face this in the audio world when drawing waveforms. You typically have on the order of 10-100k samples per second, durations of 10s-1000s of seconds, and pixel widths of on the order of 1-10k pixels.
Bresenham's is one algorithm historically used to downsample the data, but a lot of contemporary audio software doesn't use that. In Ardour (a cross-platform, libre, open source DAW), we actually compute and store min/max-per-N-samples and use that for plotting (and as the basis for further downsampling.
> In Ardour (a cross-platform, libre, open source DAW), we actually compute and store min/max-per-N-samples and use that for plotting (and as the basis for further downsampling.
I discovered flot during my academic research career circa 2008 and it saved my ass more times than I can count. I just wanted to say thank you for that. I wouldn't be where I am today without your help :)
> But regarding sampling, if it's a line chart, you can sample adaptively by checking whether the next point makes a meaningfully visible difference measured in pixels compared to its neighbours.
uPlot basically does this (see sibling comment), so hopefully that's some validation for you :)
Is there any techniques using wavelet decomposition to decimate the high frequency component while retaining peaks? I feel like that's a more principled approach than sampling but I haven't seen any literature on it describing the specific techniques (unless the idea is fundamentally unsound which is not obvious to me).
Interesting idea - I haven't explored wavelet-based approaches but the intuition makes sense: decompose into frequency bands, keep the low-frequency trend, and selectively preserve high-frequency peaks that exceed some threshold.
My concern would be computational cost for real-time/streaming use cases. LTTB is O(n) and pretty cache-friendly. Wavelet transforms are more expensive, though maybe a GPU compute shader could make it viable.
The other question is whether it's "visually correct" for charting specifically. LTTB optimizes for preserving the visual shape of the line at a given resolution. Wavelet decomposition optimizes for signal reconstruction - not quite the same goal.
That said, I'd be curious to experiment. Do you have any papers or implementations in mind? Would make for an interesting alternative sampling mode.
I don't. I just remember watching a presentation on it and it always struck me that wavelets are an incredibly powerful and underutilized technique for data reduction while preserving quality in a quantifiable and mathematically justifiable way.
I don't have any papers in mind, but I do think that the critique around visual shape vs signal reconstruction may not be accurate given that wavelets are starting to see a lot of adoption in the visual space (at least JPEG2000 is the leading edge in that field). Might also be interesting to use DCT as well. I think these will perform better than LTTB (of course the compute cost is higher but there's also HW acceleration for some of these or will be over time).
No, FFT is perfectly information preserving by definition. Thats why there’s an inverse FFT operation that restores the original signal without any loss (well, modulo accumulated floating point error when working in the discrete instead of symbolic space).
> creating multiple charts on the same page with GL (e.g. dashboard) has historically been limited by the fact that Chrome is capped at 16 active GL contexts that can be acquired simultaneously. Plotly finally worked around this by using https://github.com/greggman/virtual-webgl
Sometimes I like to ponder on the immense amount of engineering effort expended on working around browser limitations.
What I did in a few projects to plot aggregated (resampled) data without loosing peaks was to plot it over an area chart representing the min-max values before aggregating (resampling). It worked pretty well.
Both are useful. With the y-axis staying the same there is a stable point of reference. Then you can see how sub-samples behave relative to your whole sample.
related: i had to jump through the Date hoops recently (again) when rewriting uPlot's DST and timezone handling, but i'm pretty happy with the result that avoided much complexity, performance loss, and adding a dependency: https://github.com/leeoniya/uPlot/pull/1072
> An enduring myth about the Moon is that it doesn't rotate. While it's true that the Moon keeps the same face to us, this only happens because the Moon rotates at the same rate as its orbital motion, a special case of tidal locking called synchronous rotation.
My colleagues once spent a good hour trying to explain this fact to me and I still really struggle to accept it. I can see that the moon is rotating on its own axis from the point of view of a space that is external to the system it forms with the earth. But then isn’t everything on earth rotating about its own axis with respect to that external space? It seems arbitrary to isolate the moon from all this other stuff and make a special case of it…
1. Unlike position and velocity, which are relative (there is no given "origin" for them, no way to say where a thing is or how fast it's moving except relative to other things), rotation is absolute. A thing is either rotating or not, regardless of its relation to other things. Objects that rotate "experience (centrifugal) forces as a result" or "require (centripetal) forces to hold them together" depending on how you choose to describe it. This is detectable: hook two weights together with a newton-meter in space and the newton-meter will read non-zero when the assemblage is rotating, zero when not. The reading tells you how fast it is rotating regardless of any external reference point. (An equivalent device to detect position or velocity is not possible, but it is for acceleration.)
2. Yes, everything "at rest" on earth is in fact rotating at the rate the earth rotates. If you stand on the equator at midday and do not rotate you will be standing on your head at midnight.
>no way to say where a thing is except relative to other things
This is always true. The origin is just a thing that other things are relative to. It's just as possible to define an origin in the real world as it is on a piece of graph paper.
Thanks for this explanation. If I understand correctly then, the moon requires some centripetal force in order not to dissipate due to its rotation whereas e.g. my head or the Eiffel Tower do not because they are not subject to absolute rotation.
Indeed. The Eiffel tower and your head do both have some (extremely small) centripetal force compensating for their rotation along with the earth.
(You can break that down in different ways, i.e. use various choices of generalised coordinates to describe it, so exactly what constitutes "centripetal", "centrifugal", "gravitational", "tidal", etc. forces depends on that. I'm being pretty vague in how I decribe it. Regardless, rotation is absolute, or in other words the equations of physics take a different form in a rotating frame of reference than in a non-rotating one.)
Thanks for the clarification I completely mistook what you were saying. This is the fascinating bit for me then, that what’s happening with the moon’s rotation is also happening with everything else
> But then isn’t everything on earth rotating about its own axis with respect to that external space?
From the point of view of the moon, for the purposes of action due to gravity, anything on Earth is essentially part of the Earth, not an entity that is massive enough to be considered separately. The aggregate centre of mass is what counts. Similar for the Sun looking at the Earth/Moon system: from that PoV Earth+Moon is it a single mass with a centre somewhere between the two major masses that form it.
If the Moon where sufficiently consistent in its shape and density, it could rotate freely in any direction while orbiting the Earth, that fact that it is more dense on one side means that it is more energy efficient for it to spin in step with its orbit such that the dense side keeps facing us. If something massive hit the moon (let's assume this somehow happens without significantly affecting its orbit or causing significant problems for Earth too!) it might push the rotation off for a bit, but it would slowly be pulled back into sync. If something sufficiently massive simply landed on the moon, that would affect the mass distribution and the exact face that points at us would slowly change to reach a new equilibrium.
Pick the sun as reference: the moon rotates. Pick the earth as reference: the moon rotates. Stand on the moon and pick any star as reference: the moon rotates.
Yes but from the point of view of the earth the moon does not appear to be rotating around its own axis since it is tidally locked. In that sense it’s confusing to me to distinguish it from everything else on earth but the comment above about centripetal force clarifies this for me I think
> If I ever felt I was too old to have kids I’d probably have no choice but to kill myself
sperm quantity and quality decreases with age. studies exist that suggest higher risk of autism when father's age >= 45.
you're not too old, but you should probably test & freeze some good sperm before you might actually be too old by the time you find the right person. this way you won't ever feel like you're too old to have kids. then the question becomes more of "how long and in what manner will my remaining health allow me to enjoy them"
this is highly workload-dependent. there are plenty of APIs that are multiple-factor faster and 10x more memory efficient due to native implementation.
reply