java - Assertions on many files with required pre-processing -

i have large set of files (>1000) on want running same set of assertions. assertions organized in logical groups, have broken them down more 30 @test methods in single class.

in order run assertions want, need run before each file expensive pre-processing, , assertions run on results of pre-processing.

as pre-processing required each @test method, want running executing once each file.

i used @dataprovider following way:

@contextconfiguration({ "classpath:/meta-inf/spring/ev-xml-test-context.xml" }) public class fantastictest extends abstracttestngspringcontexttests {      private string[][] files = new string[][] {             { "file1.xml" },             { "file2.xml" },             { "..." },             { "file1000.xml" }     };      private map<string, object> preprocessingresults = new hashmap<>();      @beforeclass     public void setup() throws exception {         for(int i=0; i<files.length; i++) {             preprocessingresults.put(files[i][0], expensivepreprocessing(files[i][0]));         }     }      private object expensivepreprocessing(string file) {         // expensive         return new expensiveobject(file);     }      @dataprovider(name = "mydataprovider")     public string[][] dataprovider() {         return files;     }      @test(dataprovider = "mydataprovider")     public void test1(string filename) {         object preprocessingresultsforfile = preprocessingresults.get(filename);         // assertions     }      @test(dataprovider = "mydataprovider")     public void test2(string filename) {         object preprocessingresultsforfile = preprocessingresults.get(filename);         // assertions     }     // ...     @test(dataprovider = "mydataprovider")     public void test30(string filename) {         object preprocessingresultsforfile = preprocessingresults.get(filename);         // assertions     } } 

what approach in ide (intellij) , in ci (jenkins) can see failed. example, see method test3("file823.xml") failed.

what don't that:

• the pre-processing results held in memory files until assertions begin
• i have wait until preprocessing ran until can see assertions taking place (i prefer if each file pre-processing took place , assertions, , move next file).

i tried approach @factory described in this dzone article. liked each instance constructed in @factory can self-contained, executing pre-processing of file in @beforeclass method.

what didn't that:

• i no longer had name of failing file in failing test method. see method test29() failed. because dataprovider= attribute on @factory, , not @test methods.
• i still haven't managed make part regarding order of test execution work (with testng.xml).

i wondering if have suggestion approach failing tests self-explanatory (like in first approach) pre-processing , assertions evenly spread (like second approach).