Skip to content

Add diagnostic mode#553

Merged
swharden merged 20 commits intoScottPlot:masterfrom
StendProg:DiagnosticMode
Oct 1, 2020
Merged

Add diagnostic mode#553
swharden merged 20 commits intoScottPlot:masterfrom
StendProg:DiagnosticMode

Conversation

@StendProg
Copy link
Contributor

@StendProg StendProg commented Sep 19, 2020

Purpose:
Sometimes users provide incorrect data and blame library for incorrect results #546.
This PR allow user to enable Diagnostic mode at init of their project.
As a result, multiple data checks are enabled before each render, and detailed exceptions are thrown if the data is not in the correct format.
With diagnostic mode turned off, there are absolutely no negative performance impacts compared to the existing code.

Added a stub for IValidatableData. If some plottable implements it, instead of attribute checks simple call IValidatableData.ValidateData()

PR contains 3 Diagnostic mode demo.
It needs a good names and descriptions. My bad English won't let me do it myself.
Also feel free to change any namings in this PR.

Expansion of this PR
To enable existing types of check on new plottable fields, just mark it with existing attribute:

public class SomePlottable
{
  [NotNAN]
  double SomeDataField;
}

this field will start to automaticaly be checked for NAN value in Diagnostic mode

To make new Type of check:

  1. Create new Attribute discribing this check. A short name is very preferred because attributes create a lot of visual noise in Plottables.
  2. Create new FieldsChecker derived from FieldsCheckerBase.
  3. Add new check to DiagnosticDataChecker.Checks.
  4. Mark required Plottable fields with new Attribute

New functionality (code):

plt.DiagnosticMode = true;
plt.PlotSignal(new double[]{12, 45, double.NAN, 54});

// Exception wil be thrown on first Render.

@swharden
Copy link
Member

Hi @StendProg, I really like the direction of this PR!

Having decorators like these that we can place on plottable data variables, combined with methods which can check them for validity, is an excellent combination. This will indeed make it easier for users to troubleshoot plots that fail due to bad data.

PR contains 2 Diagnostic mode demo.
It needs a good names and descriptions. My bad English won't let me do it myself.

Actually, it may be best to delete these demos since they throw exceptions which crash the cookbook and demo application.

image

I'm not sure of the best way to communicate that this feature exists though. I wonder if it would be a good idea to put diagnosticMode = false right in the constructor?

https://github.com/swharden/ScottPlot/blob/4d06b49aa8ce85139fbc22a3a4990e311700ff6e/src/ScottPlot/Plot.cs#L25

This would make it easier to see for new users creating console applications, but unfortunately it would not be seen by people using the user control.

I will continue to think about this problem!

@StendProg
Copy link
Contributor Author

Hi Scott (@swharden ),
I'm really glad you liked it.

Actually, it may be best to delete these demos since they throw exceptions which crash the cookbook and demo application.

The sole purpose of these demos is to let the user know about this mode, I think you will find a beautiful solution without them.
We can make custom error notification, say:

  1. Throw
  2. Raise event
  3. Write to log file.

Demo can use second variant, and replace demo description with error detail on event.

But I think most users will be satisfied with getting an exception to diagnose and fix the problem...

@bclehmann
Copy link
Member

To make sure the user knows that diagnostic mode is not intended to be used for non-debugging purposes for performance reasons, it may make sense to log a warning when it is enabled?

@swharden
Copy link
Member

I like the idea...

Maybe something like Debug.WriteLine("WARNING: diagnostic mode is enabled, reducing performance")

Perhaps with some logic so the message is only shown once?

@StendProg
Copy link
Contributor Author

@Benny121221 pointed to the thing I missed. There is a GetPlottables() API method that gives the user full access to the list. In particular, he can delete, add, overwrite Plottable as he wants, and check for the existence of specific types in the list. This breaks the whole concept of decorators. Keeping another list and trying to synchronize them will make it all very difficult. The beauty of decorators is that you don't need to control all the Render calls, which can be scattered throughout the code in a chaotic manner. In reality, we have a clear rendering traversal of the list, and it might be worth really adding checks there without all those wraps.

What can be left of all this is clever universal checks through reflection on attributes. This will greatly reduce code cohesion and eliminate duplication. The only drawback is the visual pollution of the code with attributes.

@swharden
Copy link
Member

swharden commented Sep 22, 2020

What can be left of all this is clever universal checks through reflection on attributes. This will greatly reduce code cohesion and eliminate duplication. The only drawback is the visual pollution of the code with attributes.

I agree this is clever code, but I'm not totally convinced yet that decorators are the best way to solve this problem. A CheckValidation() method still is needed for things like comparing the length of Xs and Ys are equal on scatter plots, or that the number of errorbars is equal to the number of points. Using decorators means some rules would be defined as decorators on properties, and others in CheckValidation().

It might be nicer to put all validation information for a plottable inside a AssertValidity() method (which would be required by IPlottable). We could eliminate duplication by putting assert methods in a static Validate class. Interestingly the validation method would resemble a series of unit tests. Perhaps this is a good thing? If we wanted to resemble nUnit we could create an Assert class and have similarly named methods which also take a message to display if an exception occurs.

public bool AssertValidityBeforeRender = false; // diagnostic mode if true

public void AssertValidity(){
    Assert.NotNull(Xs, "Xs cannot be null");
    Assert.NotNull(Ys, "Ys cannot be null");
    Assert.EqualLength(Xs, Ys, "Xs and Ys have different lengths");
    Assert.AllReal(Xs, "Xs cannot contain infinity or NaN");
    Assert.AllReal(Ys, "Ys cannot contain infinity or NaN");
    Assert.Ascending(Xs, "Xs cannot contain any descending values");
}

I will continue to think about this! These are all interesting ideas.

@StendProg
Copy link
Contributor Author

StendProg commented Sep 23, 2020

CheckValidation() method still is needed for things like comparing the length of Xs and Ys are equal on scatter plots

We can mark xs and ys with [EqualLength] attirbute. This will provide enouth information for check.

Using decorators means some rules would be defined as decorators on properties, and others in CheckValidation().

Decorators, in my terms, are the objects that wrapped the plottable, I have already abandoned them, at least as long as GetPlottable () API exists.

But mark proprtyes with attributes is another concept. The only thing that keeps this PR from closing.
The big advantage of this approach is that it is very easy to add checks to the newly created Plottable object.
No matter how we simplify the creation of checks using helper methods when implementing the IValidate interface, it will still take more lines of code than simply marking up the required properties.
Considering the fact that the number of plotables, even at the moment, is already quite large, any savings have the right to be discussed.
The IValidate interface approach is also very clean, but you need to implement it right now in 20+ classes.

A significant drawback of attribute-based checks is that you will have to request properties through reflection, which is tens of times slower than the usual approach. But since these checks are only for special mode, where performance is not significant, this does not matter much.

@StendProg
Copy link
Contributor Author

StendProg commented Sep 29, 2020

I think it's finished.
Some Plottables like PlottableVectorField, PlottableHeatmap stay without attributes because their behavior not obvious for me.
During tests on demos i found that Finance->SMA demo contains NAN on SMA Scatters. I didn’t touch it since SMA() return NAN for elements outside the window...

@swharden
Copy link
Member

swharden commented Oct 1, 2020

Thank you for your hard work on this @StendProg! As I work toward refactoring the constructors for all plottables (#573) I'll review their public properties and continue to build upon this work.

@swharden swharden merged commit 58a20ad into ScottPlot:master Oct 1, 2020
swharden added a commit that referenced this pull request Oct 1, 2020
instead of settings.plottables.Add(), continuing the theme begun by #553. Relates to #573
@swharden swharden mentioned this pull request Oct 5, 2020
38 tasks
swharden added a commit that referenced this pull request Oct 16, 2020
The new Validation module has static methods that can be used inside plottables so they can validate themselves. #578, relates to #553
swharden added a commit that referenced this pull request Oct 21, 2020
#578

The previous diagnostic system which uses reflection has been disabled (#553) and all plottables but signal now support the new system (#578).

The demo application can be used to test this functionality for scatter (but not signal) plots. #603
swharden added a commit that referenced this pull request Oct 23, 2020
IPlottable now has one method to check for errors and return a string if they exist #578 #553 #603 #605
swharden added a commit that referenced this pull request Oct 23, 2020
excellent code and logic introduced in #553 has now been moved inside individual plottables
@swharden swharden mentioned this pull request Aug 21, 2021
82 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants