PlotSignalConst() - faster rendering for millions of points#70
PlotSignalConst() - faster rendering for millions of points#70swharden merged 9 commits intoScottPlot:masterfrom
Conversation
Work correct only for constant signals(change source after adding not alowed).
Padanian
left a comment
There was a problem hiding this comment.
Searching a big population for its min and max by scrolling through the whole of it, it is efficient neither with a for cycle, nor with a segmented tree.
We need to investigate a parallel.for or a multithreading search.
swharden
left a comment
There was a problem hiding this comment.
This is really great! Signal plots render much faster with this method.
In the comments you noted a size limitation for the input array:
When I try to plot 100 million points I get this exception
Exception thrown: 'System.OutOfMemoryException' in System.Core.dll
An unhandled exception of type 'System.OutOfMemoryException' occurred in System.Core.dll
I understand why I get this exception, but someone less familiar with this plotting library may benefit from a more descriptive failure. What do you think about modifying the code to do one of:
- throw an exception immediately if
PlottableSignalConst()is instantiated with an array too large for the system - if the array is too large for the segmented tree method to process, use the original (non-tree-based) MinMaxRangeQuery so no exception ever happens
looking forward to hearing your thoughts!
@Padanian additional speed improvements could be made with multi-threading, but the |
|
I try parallel.For for a fist time it gives only 2-3 times faster rendering on my 4 core. This is not enough. Anyway can make it for a PlotSignal(). PlotSignalConst works realy fast 20-30 comparasions for each interval. Bottleneck on graphics.DrawLines() not MinMaxSearch. |
|
Memory limitations its c# array limitations. It's trying to allocate in memory continious block for full array. Point of search is using another data types but all algoritms will be more complex and more slow. |
Consider a situation where you create a program and it runs well on your really nice computer with lots of memory. You send your program to a client who has a really bad computer and it crashes on their computer but not yours... this is what I'd like to avoid. What if we made this simpler by not creating a new plot object? We could just add "const" as an argument to PlotSignal():
What do you think? |
|
Its simple to make by call base.MinMaxRangeQuery() inside MinMaxRangeQuerry in different situation. |
autoformat, clarified comments, and moved TreesReady assignment to better spot (in case UpdateTrees() is manually re-run)
Small datasets (e.g., 1k points) were failing to render. I think it's because the way the min/max lookup works for small datasets. This fixes the rendering issue (is there a better solution?)
|
I updated the "plot types" demo so all the signals use PlotSignalConst(). When I did this I noticed that the signal with 1k points didn't display well. This improved it, but I don't think it's a definitive solution: But it made me wonder - what is the lower limit (number of points) this class can display? Should this class be expected to work with an array of 10 data points? |
|
The reason is wrong indexes in calling querry. I thinked it always call (left_index < right_index) |
|
This looks great! Thanks @StendProg! |
|
I added a relevant section to the cookbook: https://github.com/swharden/ScottPlot/tree/master/cookbook#signal PlotSignal() and PlotSignalConst() were showing the same render times initially, then I realized it was because of the multi-threading. I ended-up adding a |
|
Thanks for accepting. Feel free to give better name for interface method |
|
I think the I agree with your suggestion that the argument to turn threading on/off needs a better name (which describes its behavior rather than its implementation). This argument controls whether or not to allow the old low-speed rendering while the trees are still being calculated for the new high-speed method. It will rarely be changed, and probably only in console applications. Perhaps: plt.PlotSignalConst(data, allowEarlyRendering: false); |
PlotSignalConst() - faster rendering for millions of points
Finding MinMax in RenderHighDensity (bottleneck for large signals) using Segmented Tree.
Responsitive rendering in cost of additional memory, preprocessing time and losing ability change source signal after adding.