xunit example

gistfile1.txt

A profiler can quite easily find the constructor call to default constructor on this code:
[Fact]
public void ModestCreatesParameterWithBlah(){
   var p = new MultiOrderedConstructorType();
   p.Number = ...;
   Assert.AreEqual(0, p.Number);
}

but on this it is created outside the scope of the test (its done internally by xunit)

[Theory, AutoData]
public void ModestCreatesParameterWithBlah([modest] MultiOrderedConstructorType x) {
   Assert.AreEqual(0, x.Number);
}

in the first example the call to the constructor when looking at the current stack is under the context of a test. the second is not.


How would a coverage tool know to associate the constructor called before the test internally to xunit to the Theory?

In that particular example, the constructor isn't invoked by xUnit.net, but rather by the [AutoData] attribute...

The coverage tools I'm familiar with don't do static analysis, but work by either instrumenting or sampling from a running process. Thus, while the [AutoData] attribute invokes the constructor rather than the test itself, all the coverage tools sees and cares about is that the constructor is being invoked.

However, I can see how a static analysis tool like MM would have quite a hard time finding that relationship.

Most code coverage tools also allow you to ask which test covers this code. In that case it basically become a problem without solution as there is no good way to tie back what test is actually covering the code (as you cant easily figure out which test is associated since its not in the context of a test)

btw: this is not for the MM static analysis this is dealing with our dynamic analysis (a profiler). I can know that in running the test suite this code was run but I can't figure out which test its actually involved with.

Right... I don't know whether there's a good answer to your question. It has never been a concern of mine...

But this would make things like dotCover not be able to work appropriately as well no? I can hack in some stuff but I would be highly dependent on the internal (undocumented) workings of xunit. As an example, do those bits of data get called right before the test or earlier and generated for the fixture as a whole. I can't find any decent way of deterministically (and safely) associating the code thats being called in that constructor back to the test that is actually causing it to be called at runtime.

Without knowing any of the details of how MM currently detects that an xUnit test method is about to be executed, it's hard to provide to-the-point suggestions, but one option is to profile 'around' invocations of Xunit.Sdk.ITestCommand.Execute (would cover both Fact & Theory based tests). AFAICT, xUnit does not create the actual test method parameter instances until 'just before' it executes a single test method.

As of now we define a test as the entrance point into your code. I could take a dependency that it will hit just before your code. My worry is that this changes. In this case now we are in a world of pain with multiple versions of xunit.

On Tue, Apr 3, 2012 at 8:13 AM, sgryt < ***@***.*** > wrote: Without knowing any of the details of how MM currently detects that an xUnit test method is about to be executed, it's hard to provide to-the-point suggestions, but one option is to profile 'around' invocations of Xunit.Sdk.ITestCommand.Execute (would cover both Fact & Theory based tests). AFAICT, xUnit does not create the actual test method parameter instances until 'just before' it executes a single test method. --- Reply to this email directly or view it on GitHub: https://gist.github.com/2290767

## Le doute n'est pas une condition agrable, mais la certitude est absurde.

I think that assessment is completely correct. You can't know. It doesn't even depend on xUnit.net itself - it depends on the particular subtype of DataAttribute. As an example, IIRC the built-in [InlineData] attribute creates data 'rows' on a per-method basis, whereas the [ClassData] might create one set of test data per test class.

Is this something best just to leave off IYO? How often is this stuff being used? The only place it sees an error is in the coverage of the default constructor btw.

On Tue, Apr 3, 2012 at 8:20 AM, Mark Seemann < ***@***.*** > wrote: I think that assessment is completely correct. You can't know. It doesn't even depend on xUnit.net itself - it depends on the particular subtype of DataAttribute. As an example, IIRC the built-in [InlineData] attribute creates data 'rows' on a per-method basis, whereas the [ClassData] _might_ create one set of test data per test class. --- Reply to this email directly or view it on GitHub: https://gist.github.com/2290767

## Le doute n'est pas une condition agrable, mais la certitude est absurde.

True. As mentioned on another communication-thread, these details and dependencies could be handled by others (say, me) if you could ship MM with an extensibility point for when profiling a test method starts :) This would keep the xUnit dependencies out of MM, but still let users extend the profile coverage scope as much as possible.
So I guess I'm proposing that as a feature request (nice-to-have).

Possibly except would you want to see the internal to xunit code thats when the test "starts" inside of your sequence diagram MM can generate for you? If not now some magic needs to get applied.

I'm 'only' interested in profiler coverage of my own code - and I'm not entirely sure I understand your question wrt the sequence diagram?

If I were just giving the callchains so your plugin could decide necessarily the other calls there will be in the chain (as of now what is in the chain is what you will see). And no you aren't always only interested only in what happens in your code (I can give at a minimum 5 valid use cases where people want to see other code). So I would also need to give you some way of saying "these aren't important even though we said they are they are only needed for the plugin to decide where start". Then as well we can run into problems with using the profiler on the runner's own code base (it wouldn't profile correctly, though this is admittedly a minor issue). My main worry though is that these seem to not only be method scoped, they are also fixture scoped from this conversation. How many other scopings are there (more importantly how stable are they?). It would be absolute hell if these changed from version to version of the runner. I will go research how some of the other tools are handling these cases. Greg

On Tue, Apr 3, 2012 at 10:29 AM, sgryt < ***@***.*** > wrote: I'm 'only' interested in profiler coverage of my own code - and I'm not entirely sure I understand your question wrt the sequence diagram? --- Reply to this email directly or view it on GitHub: https://gist.github.com/2290767

## Le doute n'est pas une condition agrable, mais la certitude est absurde.

I haven't heard of any other unit testing frameworks that can be customized wrt object construction (but @ploeh can probably say more about that) for test method parameters, but that is probably just due to the fact that I haven't done enough research. I'll be very interested in hearing about your findings, though.

My personal primary interest is to have the profiling test coverage counters be as inclusive as possible - for my code. I'm curious as to why you think I would be interested in test coverage for code that is not under my control, or the intricate details as of how xUnit internals wire stuff together? It doesn't matter that much to me to see a connected call chain containing both the object construction and the test method - I'm just really keen on having the AWESOME in-your-face visibility of test coverage for the largest possible part of my codebase.

I certainly understand your worries - that's why I propose that MM users take that part of the responsibility themselves - with a plugin implemetation of their own, targeted at a specific unit testing framework.