Hey @ScrapCodes this is looking good. I think ideally the way this should work is the following. After we run sbt/sbt assembly test in the build script we should run ./spark-class org.apache.spark.tools.GenerateMIMAIgnore and then run the MIMA checker. The generation script should create a file in a well known location (e.g. .mima-excludes) and the MIMA script should read that file and create excludes automatically.

./spark-class org.apache.spark.tools.GenerateMIMAIgnore and then run the MIMA checker. The generation script should create a file in a well known location (e.g. .mima-excludes) and the MIMA script should read that file and create excludes automatically.

I am not very sure, but won't this automatic creation of excludes forfeit the purpose of having mima at all. What would happen if a class was public in version 1 and become private[x] in 1.2. Then we would simply ignore it but it is a violating of binary compatibility.

One or more automated tests failed
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13320/

@ScrapCodes yes you are correct. We'll miss those cases - but without proper MIMA support I don't think we can fix that. Do you see an alternative solution that would catch those types of errors?

@ScrapCodes by the way - I don't think this forfeits all of the functionality. If someone goes and makes a class private that was not previously private it's an obvious change in visibility and something we can easily see during reviews. The much bigger issue are when people change functions and traits in a way that actual breaks compatibility but is hard to detect/notice (for instance, if you change a val to a def in a trait, it is invalid). Those cases are the hardest ones and they will still be caught here.

Hey @ScrapCodes FYI I think I'm gonna take a crack a this tonight using this patch as a starting point. I found a few other issues when doing more review and it's fairly complicated, so I may just submit a second patch. Details fourthcoming... :)

@pwendell @ScrapCodes I think this broken the build for maven yarn - see https://spark-project.atlassian.net/browse/SPARK-1315