Skip to content

Add OpenClover XML reporter #1080

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 21 commits into from
Closed

Add OpenClover XML reporter #1080

wants to merge 21 commits into from

Conversation

sebastianbergmann
Copy link
Owner

@sebastianbergmann sebastianbergmann commented May 21, 2025

As discussed in #578 and #1079, the Clover XML generated by php-code-coverage does not validate against Clover's XML schema definition.

We should change the generated Clover XML so that it does validate against the Clover XSD.

  • Validate generated Clover XML against Clover XSD in tests (the tests currently fail because of this)
  • Set clover attribute of <coverage> element to version identifier for phpunit/php-code-coverage
  • Make <metrics> element the first child of <project> and <file>
  • Set both name and path attributes on the <file> element
  • Generate <package> element for global namespace
  • Set complexity attribute on the <metrics> element under <project>
  • Set complexity attribute on the <metrics> element under <file>
  • Set attribute name of <class> element to unqualified class name
  • Remove namespace attribute from <class> element
  • Set signature attribute on the <line> element
  • Remove name and crap attributes from <line> element
  • Generate <testproject> element (see Add OpenClover XML reporter #1080 (comment))

Here is an example of a Clover XML file genereated using the changes proposed by this PR: https://gist.github.com/sebastianbergmann/2373d029e8972e038929f717c5ffa868

Copy link

codecov bot commented May 21, 2025

Codecov Report

Attention: Patch coverage is 91.71975% with 13 lines in your changes missing coverage. Please review.

Project coverage is 75.78%. Comparing base (c5d0506) to head (7ea8837).
Report is 22 commits behind head on main.

✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/Node/AbstractNode.php 0.00% 7 Missing ⚠️
src/Report/OpenClover.php 96.00% 6 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #1080      +/-   ##
============================================
+ Coverage     75.25%   75.78%   +0.53%     
- Complexity     1375     1400      +25     
============================================
  Files            96       97       +1     
  Lines          4708     4865     +157     
============================================
+ Hits           3543     3687     +144     
- Misses         1165     1178      +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sebastianbergmann
Copy link
Owner Author

@marek-parfianowicz Sorry for mentioning you out of the blue, but can you help me by pointing me to examples of valid Clover XML file for (small) Java projects so that I may look at them for reference? This is sometimes easier for me than just looking at the XSD. Thank you!

@sebastianbergmann sebastianbergmann changed the title Change generated Clover XML to validate against Clover XSD Make generated Clover XML validate against Clover XSD May 21, 2025
@marek-parfianowicz
Copy link

marek-parfianowicz commented May 21, 2025

Sure!

git clone https://github.com/openclover/clover-examples.git

cd spock-example
mvn clean clover:setup test clover:clover

Here's the content of the clover.xml file generated:

<?xml version="1.0" encoding="UTF-8"?>
<coverage generated="1747839983472" clover="4.4.1">
   <project name="Spock Framework - Example Project 1.0" timestamp="1747839978763">
      <metrics coveredelements="0" complexity="0" loc="3" methods="0" classes="1" statements="0" packages="1" coveredconditionals="0" coveredmethods="0" elements="0" ncloc="0" files="1" conditionals="0" coveredstatements="0"/>
      <package name="default-pkg">
         <metrics coveredelements="0" complexity="0" loc="3" methods="0" classes="1" statements="0" coveredconditionals="0" coveredmethods="0" elements="0" ncloc="0" files="1" conditionals="0" coveredstatements="0"/>
         <file path="/Volumes/DATA/_PROJEKTY/github/clover-examples/spock-example/src/main/groovy/Foo.groovy" name="Foo.groovy">
            <metrics coveredelements="0" coveredconditionals="0" complexity="0" loc="3" coveredmethods="0" methods="0" elements="0" classes="1" ncloc="0" statements="0" conditionals="0" coveredstatements="0"/>
            <class name="Foo">
               <metrics coveredelements="0" coveredconditionals="0" complexity="0" coveredmethods="0" methods="0" elements="0" statements="0" conditionals="0" coveredstatements="0"/>
            </class>
         </file>
      </package>
   </project>
   <testproject name="Spock Framework - Example Project 1.0" timestamp="1747839978763">
      <metrics coveredelements="83" complexity="31" loc="298" methods="29" classes="10" statements="52" packages="1" coveredconditionals="2" coveredmethods="29" elements="83" ncloc="0" files="6" conditionals="2" coveredstatements="52"/>
      <package name="default-pkg">
         <metrics coveredelements="83" complexity="31" loc="298" methods="29" classes="10" statements="52" coveredconditionals="2" coveredmethods="29" elements="83" ncloc="0" files="6" conditionals="2" coveredstatements="52"/>
         <file path="/Volumes/DATA/_PROJEKTY/github/clover-examples/spock-example/src/test/groovy/UnrollWithSeqNumber.groovy" name="UnrollWithSeqNumber.groovy">
            <metrics coveredelements="3" coveredconditionals="0" complexity="1" loc="33" coveredmethods="1" methods="1" elements="3" classes="1" ncloc="0" statements="2" conditionals="0" coveredstatements="2"/>
            <class name="UnrollWithSeqNumber">
               <metrics coveredelements="3" complexity="1" methods="1" testruns="3" statements="2" coveredconditionals="0" coveredmethods="1" elements="3" testfailures="0" testduration="0.002" conditionals="0" coveredstatements="2" testpasses="3"/>
            </class>
            <line complexity="1" visibility="public" signature="$spock_feature_0_0(def,def,def) : void" num="22" count="3" type="method"/>
            <line num="24" count="3" type="stmt"/>
            <line num="25" count="3" type="stmt"/>
         </file>
         <file path="/Volumes/DATA/_PROJEKTY/github/clover-examples/spock-example/src/test/groovy/UnrollWithVarsWithSelectors.groovy" name="UnrollWithVarsWithSelectors.groovy">
            <metrics coveredelements="7" coveredconditionals="2" complexity="4" loc="39" coveredmethods="2" methods="2" elements="7" classes="2" ncloc="0" statements="3" conditionals="2" coveredstatements="3"/>
            <class name="UnrollWithVarsWithSelectors">
               <metrics coveredelements="3" complexity="1" methods="1" testruns="2" statements="2" coveredconditionals="0" coveredmethods="1" elements="3" testfailures="0" testduration="0.003" conditionals="0" coveredstatements="2" testpasses="2"/>
            </class>
            <class name="UnrollWithVarsWithSelectors$Person">
               <metrics coveredelements="4" coveredconditionals="2" complexity="3" coveredmethods="1" methods="1" elements="4" statements="1" conditionals="2" coveredstatements="1"/>
            </class>
            <line complexity="3" visibility="public" signature="getSex() : String" num="24" count="4" type="method"/>
            <line num="25" count="4" type="stmt"/>
            <line falsecount="2" truecount="2" num="25" type="cond"/>
            <line complexity="1" visibility="public" signature="$spock_feature_0_0(def,def) : void" num="29" count="2" type="method"/>
            <line num="31" count="2" type="stmt"/>
            <line num="32" count="2" type="stmt"/>
         </file>
         <file path="/Volumes/DATA/_PROJEKTY/github/clover-examples/spock-example/src/test/groovy/UnrollWithSimpleVar.groovy" name="UnrollWithSimpleVar.groovy">
            <metrics coveredelements="3" coveredconditionals="0" complexity="1" loc="34" coveredmethods="1" methods="1" elements="3" classes="1" ncloc="0" statements="2" conditionals="0" coveredstatements="2"/>
            <class name="UnrollWithSimpleVar">
               <metrics coveredelements="3" complexity="1" methods="1" testruns="3" statements="2" coveredconditionals="0" coveredmethods="1" elements="3" testfailures="0" testduration="0.009" conditionals="0" coveredstatements="2" testpasses="3"/>
            </class>
            <line complexity="1" visibility="public" signature="$spock_feature_0_0(def,def,def) : void" num="22" count="3" type="method"/>
            <line num="24" count="3" type="stmt"/>
            <line num="25" count="3" type="stmt"/>
         </file>
         <file path="/Volumes/DATA/_PROJEKTY/github/clover-examples/spock-example/src/test/groovy/Stacks.groovy" name="Stacks.groovy">
            <metrics coveredelements="47" coveredconditionals="0" complexity="14" loc="120" coveredmethods="14" methods="14" elements="47" classes="3" ncloc="0" statements="33" conditionals="0" coveredstatements="33"/>
            <class name="EmptyStack">
               <metrics coveredelements="12" complexity="4" methods="4" testruns="4" statements="8" coveredconditionals="0" coveredmethods="4" elements="12" testfailures="0" testduration="0.013999999999999999" conditionals="0" coveredstatements="8" testpasses="4"/>
            </class>
            <class name="StackWithOneElement">
               <metrics coveredelements="16" complexity="5" methods="5" testruns="4" statements="11" coveredconditionals="0" coveredmethods="5" elements="16" testfailures="0" testduration="0.001" conditionals="0" coveredstatements="11" testpasses="4"/>
            </class>
            <class name="StackWithThreeElements">
               <metrics coveredelements="19" complexity="5" methods="5" testruns="4" statements="14" coveredconditionals="0" coveredmethods="5" elements="19" testfailures="0" testduration="0.002" conditionals="0" coveredstatements="14" testpasses="4"/>
            </class>
            <line complexity="1" visibility="public" signature="$spock_feature_0_0() : void" num="22" count="1" type="method"/>
            <line num="23" count="1" type="stmt"/>
            <line complexity="1" visibility="public" signature="$spock_feature_0_1() : void" num="26" count="1" type="method"/>
            <line num="27" count="1" type="stmt"/>
            <line num="28" count="1" type="stmt"/>
            <line complexity="1" visibility="public" signature="$spock_feature_0_2() : void" num="31" count="1" type="method"/>
            <line num="32" count="1" type="stmt"/>
            <line num="33" count="1" type="stmt"/>
            <line complexity="1" visibility="public" signature="$spock_feature_0_3() : void" num="36" count="1" type="method"/>
            <line num="38" count="1" type="stmt"/>
            <line num="41" count="1" type="stmt"/>
            <line num="42" count="1" type="stmt"/>
            <line complexity="1" visibility="private" signature="setup() : def" num="49" count="4" type="method"/>
            <line num="50" count="4" type="stmt"/>
            <line complexity="1" visibility="public" signature="$spock_feature_0_0() : void" num="53" count="1" type="method"/>
            <line num="54" count="1" type="stmt"/>
            <line complexity="1" visibility="public" signature="$spock_feature_0_1() : void" num="57" count="1" type="method"/>
            <line num="59" count="1" type="stmt"/>
            <line num="62" count="1" type="stmt"/>
            <line num="63" count="1" type="stmt"/>
            <line complexity="1" visibility="public" signature="$spock_feature_0_2() : void" num="66" count="1" type="method"/>
            <line num="68" count="1" type="stmt"/>
            <line num="71" count="1" type="stmt"/>
            <line num="72" count="1" type="stmt"/>
            <line complexity="1" visibility="public" signature="$spock_feature_0_3() : void" num="75" count="1" type="method"/>
            <line num="77" count="1" type="stmt"/>
            <line num="80" count="1" type="stmt"/>
            <line num="81" count="1" type="stmt"/>
            <line complexity="1" visibility="private" signature="setup() : def" num="88" count="4" type="method"/>
            <line num="89" count="12" type="stmt"/>
            <line num="89" count="4" type="stmt"/>
            <line complexity="1" visibility="public" signature="$spock_feature_0_0() : void" num="92" count="1" type="method"/>
            <line num="93" count="1" type="stmt"/>
            <line complexity="1" visibility="public" signature="$spock_feature_0_1() : void" num="96" count="1" type="method"/>
            <line num="98" count="1" type="stmt"/>
            <line num="99" count="1" type="stmt"/>
            <line num="100" count="1" type="stmt"/>
            <line num="101" count="1" type="stmt"/>
            <line complexity="1" visibility="public" signature="$spock_feature_0_2() : void" num="104" count="1" type="method"/>
            <line num="106" count="1" type="stmt"/>
            <line num="107" count="1" type="stmt"/>
            <line num="108" count="1" type="stmt"/>
            <line num="109" count="1" type="stmt"/>
            <line complexity="1" visibility="public" signature="$spock_feature_0_3() : void" num="112" count="1" type="method"/>
            <line num="114" count="1" type="stmt"/>
            <line num="117" count="1" type="stmt"/>
            <line num="118" count="1" type="stmt"/>
         </file>
         <file path="/Volumes/DATA/_PROJEKTY/github/clover-examples/spock-example/src/test/groovy/HelloSpock.groovy" name="HelloSpock.groovy">
            <metrics coveredelements="3" coveredconditionals="0" complexity="1" loc="31" coveredmethods="1" methods="1" elements="3" classes="1" ncloc="0" statements="2" conditionals="0" coveredstatements="2"/>
            <class name="HelloSpock">
               <metrics coveredelements="3" complexity="1" methods="1" testruns="3" statements="2" coveredconditionals="0" coveredmethods="1" elements="3" testfailures="0" testduration="0.019" conditionals="0" coveredstatements="2" testpasses="3"/>
            </class>
            <line complexity="1" visibility="public" signature="$spock_feature_0_0(def,def) : void" num="20" count="3" type="method"/>
            <line num="22" count="3" type="stmt"/>
            <line num="23" count="3" type="stmt"/>
         </file>
         <file path="/Volumes/DATA/_PROJEKTY/github/clover-examples/spock-example/src/test/groovy/SpecInheritance.groovy" name="SpecInheritance.groovy">
            <metrics coveredelements="20" coveredconditionals="0" complexity="10" loc="41" coveredmethods="10" methods="10" elements="20" classes="2" ncloc="0" statements="10" conditionals="0" coveredstatements="10"/>
            <class name="BaseSpec">
               <metrics coveredelements="10" coveredconditionals="0" complexity="5" coveredmethods="5" methods="5" elements="10" statements="5" conditionals="0" coveredstatements="5"/>
            </class>
            <class name="DerivedSpec">
               <metrics coveredelements="10" complexity="5" methods="5" testruns="2" statements="5" coveredconditionals="0" coveredmethods="5" elements="10" testfailures="0" testduration="0.002" conditionals="0" coveredstatements="5" testpasses="2"/>
            </class>
            <line complexity="1" visibility="private" signature="setupSpec() : def" num="22" count="1" type="method"/>
            <line num="22" count="1" type="stmt"/>
            <line complexity="1" visibility="private" signature="cleanupSpec() : def" num="23" count="1" type="method"/>
            <line num="23" count="1" type="stmt"/>
            <line complexity="1" visibility="private" signature="setup() : def" num="25" count="2" type="method"/>
            <line num="25" count="2" type="stmt"/>
            <line complexity="1" visibility="private" signature="cleanup() : def" num="26" count="2" type="method"/>
            <line num="26" count="2" type="stmt"/>
            <line complexity="1" visibility="public" signature="$spock_feature_0_0() : void" num="28" count="1" type="method"/>
            <line num="28" count="1" type="stmt"/>
            <line complexity="1" visibility="private" signature="setupSpec() : def" num="34" count="1" type="method"/>
            <line num="34" count="1" type="stmt"/>
            <line complexity="1" visibility="private" signature="cleanupSpec() : def" num="35" count="1" type="method"/>
            <line num="35" count="1" type="stmt"/>
            <line complexity="1" visibility="private" signature="setup() : def" num="37" count="2" type="method"/>
            <line num="37" count="2" type="stmt"/>
            <line complexity="1" visibility="private" signature="cleanup() : def" num="38" count="2" type="method"/>
            <line num="38" count="2" type="stmt"/>
            <line complexity="1" visibility="public" signature="$spock_feature_1_0() : void" num="40" count="1" type="method"/>
            <line num="40" count="1" type="stmt"/>
         </file>
      </package>
   </testproject>
</coverage>

@sebastianbergmann
Copy link
Owner Author

Sure!

Thank you so much!

@sebastianbergmann
Copy link
Owner Author

@marek-parfianowicz I do not understand the purpose of the <testproject> element. Its documentation reads "Project metrics relating to test source". Is this for code coverage information of test code? We do not collect that information, though.

@marek-parfianowicz
Copy link

Hi Sebastian! Yes, in OpenClover you can separate code into application and test. The tag contains coverage information for application code, the tag for test. Collecting coverage information for test code is very useful in OpenClover, because then it instruments tests allowing to measure per-test code coverage as well as record test execution time and results. Which is later being used in reports, especially the HTML one.

@sebastianbergmann sebastianbergmann marked this pull request as ready for review May 22, 2025 13:09
@sebastianbergmann
Copy link
Owner Author

sebastianbergmann commented May 22, 2025

With the current state of the clover branch (which this PR proposed to merge into main) the test cases we have for the Clover XML reporter generate Clover XML documents that 1) match our expectations and 2) validate against the Clover XSD if we ignore the requirement of the <testproject> element. As far as compliance with the official Clover XML format is concerned, this is a much better state than we had before. And in the ~ 20 years of support for Clover XML in PHPUnit and this library, there was never a problem reported about the <testproject> element missing.

@RykHawthorn @jimdelois @vnrmc @amottier @ravage84 Sorry to bother you with a mass-ping, but you commented on #578 and/or #1079 and I would like to ask for your help and advice.

Can you please have a look at the Clover XML that is generated on the clover branch? I think my biggest concern right now is that while the changes proposed by this PR correct mistakes of the past to make the Clover XML generated by this library compliant with Clover XSD these changes are significant and could be considered a break of backward compatibility. It would therefore be helpful to check whether the new Clover XML can be consumed by common consumers without problems.

@marek-parfianowicz
Copy link

It would therefore be helpful to check whether the new Clover XML can be consumed by common consumers without problems.

Hint: Atlassian Bamboo offers integration with Clover/OpenClover. If you select the manual clover integration option, it should be possible to trick Bamboo to show a report generated by CloverPHP on the build results tab. This would be a good test, in my opinion.

Also, the Jenkins has CloverPHP plugin as far as I recall. It could be worth checking it too.

https://openclover.org/doc/manual/latest/hacking--using-openclover-for-php.html

https://confluence.atlassian.com/bamboo/enabling-clover-for-bamboo-289277268.html

It should be possible to see something like this (in Bamboo):
https://confluence.atlassian.com/bamboo/viewing-the-clover-code-coverage-for-a-plan-289276963.html

@sebastianbergmann
Copy link
Owner Author

Here's an idea: we could keep the existing Report\Clover class as-is and introduce a new Report\CloverV4 (V4 because the generated XML validates against the Clover XSD von Clover 4.5). In the context of this library, we could then deprecate Report\Clover and later remove it.

In the context of PHPUnit, we could then introduce a configuration option for opting into using Report\CloverV4. If/when Report\Clover is removed then PHPUnit would 1) only support Report\CloverV4 and 2) remove this configuration option.

@marek-parfianowicz
Copy link

Clover's schema didn't change for years. I doubt that there's anyone using older version of Clover than 4.5.

@sebastianbergmann
Copy link
Owner Author

Clover's schema didn't change for years.

The question is: how many years? :)

However, the code that generates code coverage information in Clover XML in the php-code-coverage dependency of PHPUnit is almost 20 years old. Over the years, it has only been adapted to the changing internals of PHPUnit / php-code-coverage. Maybe it never generated valid Clover XML, maybe the schema evolved, I do not know.

I am happy, though, that we are (hopefully) getting out of this messy situation now.

@ravage84
Copy link

@ravage84
Copy link

However, the code that generates code coverage information in Clover XML in the php-code-coverage dependency of PHPUnit is almost 20 years old. Over the years, it has only been adapted to the changing internals of PHPUnit / php-code-coverage. Maybe it never generated valid Clover XML, maybe the schema evolved, I do not know.

Yeah, that code goes back to 14.11.2009. The implementation back then hints at that Clover back then didn't have any official schema documentation.

Generates an XML logfile with code coverage information using the
Clover format "documented" at
http://svn.atlassian.com/svn/public/contrib/bamboo/bamboo-coverage-plugin/trunk/src/test/resources/test-clover-report.xml

0c8db36#diff-0e105740559ef4553f3ff8c3e5e6dc0db0eb9df4e5b8142a36b5aa958f372b81R49-R51

Here's an idea: we could keep the existing Report\Clover class as-is and introduce a new Report\CloverV4 (V4 because the generated XML validates against the Clover XSD von Clover 4.5). In the context of this library, we could then deprecate Report\Clover and later remove it.

Since it isn't Clover but OpenClover, anymore and because it seems the current schema goes back to the inception of OpenClover, it could even be named OpenClover or OpenClover4.

@ravage84
Copy link

Also, the Jenkins has CloverPHP plugin as far as I recall. It could be worth checking it too.

I'm in touch with the maintainer of the Jenkins plugin and use that plugin myself.
I'll try to test-drive the updated report code.

@sebastianbergmann
Copy link
Owner Author

sebastianbergmann commented May 23, 2025

Since it isn't Clover but OpenClover, anymore and because it seems the current schema goes back to the inception of OpenClover, it could even be named OpenClover or OpenClover4.

Even better, thank you for this idea. I have updated this PR to keep the old Report\Clover class as-is and to introduce a new Report\OpenClover class with the changes discussed in this PR. As of right now, there is no option in PHPUnit yet to select Report\OpenClover instead of Report\Clover. This will hopefully change over the weekend.

@sebastianbergmann sebastianbergmann deleted the clover branch May 23, 2025 15:37
@sebastianbergmann sebastianbergmann changed the title Make generated Clover XML validate against Clover XSD Add OpenClover XML reporter May 23, 2025
@sebastianbergmann sebastianbergmann self-assigned this May 23, 2025
@sebastianbergmann
Copy link
Owner Author

This has been released in phpunit/php-code-coverage 12.3.0.

Support for this has been implemented for PHPUnit 12.2 which will be released on June 6, 2025.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants