Implement a "delta-analysis" of only the changed dependencies in a given project

[porsche-rishisaxena]

Is there any road-map designed for enabling "delta-scan" feature for Scan-Code from ORT standpoint?
If yes, please share some information.

Edit by @sschuberth: Please see this comment which clarifies that actually no delta-scan but a delta-analysis is desired. The above mention of "Scan-Code" (sic) is misleading.

[sschuberth]

No, a delta-scan feature for use together with ScanCode is not currently planned.

[nikpete]

@sschuberth We do have several use cases where delta scan capabilities will make our life much easier as we have to tranfer all results to a subsequent legal workflow tool.
Is this something that was not required / requested by the community so far and / or how do you currently handle delta scans? I just want to get a proper understanding on this topic and if we can pick it up...not sure how difficult it might be to introduce a delta-scan feature...you are the expetert :-)

[tsteenbe]

@fviernau and myself had some private discussions a few weeks ago on adding a file hash cache to enable delta scanning once scan-by-repository has been merged. Looking at ways to speed up ORT scans in CI/CD, believe cache to be cheaper, simpler and cross scanner solution compared to using DeltaCode

[sschuberth]

Is this something that was not required / requested by the community so far

Indeed, this has not been requested so far. I believe we had some internal discussions about this a really long time ago, but since it's not trivial to implement we first wanted to get the "basic" mode running.

and / or how do you currently handle delta scans?

We simple don't, and always do full scans if something changes.

[fviernau]

A very simple and probably good enough implementation could be to use a map which maps
sha1(file) -> (copyright findings[], license findings[]). That map could become quite large so a LRU strategy would need to be used. Such map could be really used as transparent cache (not a persistent storage) just for speeding things up.

The scanner implementation would need to be changed by first computing the SHA1 of all files and looking up the cache entries, only then scan the remaining files. IIRC ScanCode internally has some ignore patterns, so some effort may be required to handle these properly.

[sschuberth]

For delta-scanning, I believe the tricky bit to be to determine the list of files that do need to be scanned, not to determine / cache the results for known files.

[fviernau]

For delta-scanning, I believe the tricky bit to be to determine the list of files that do need to be scanned, not to determine / cache the results for known files.

Above I proposed "by first computing the SHA1 of all files and looking up the cache entries" as a way to "determine the list of files that do need to be scanned." Do you consider this as the tricky bit or did you think of some other solution you found tricky?

[rbieniek]

First of all, I would like to clarify what we at Porsche understand as a delta scan. Given our legal workflow, we want to build a full report for the first iteration of a project. For any subsequent release / revision, we want the lawyers to revisit only those dependencies which have changed. And by changed, we mean a dependency which has either been freshly introduced or updated with a more recent version.

To break it down into use cases, there are three different cases to be considered for a generic delta scan capability, given that you want to build a delta between two iterations (v1 and v2) of the same project:

1. some new dependencies are introduced in the transition from v1 to v2
2. the set of dependencies stays the same but updated versions of dependencies have been introduced
3. dependencies are removed in the transition between v1 and v2

The third use case is, of course, trivial and IMHO there is nothing to be done for it.

For other use cases it is not that trivial. The afformentioned hash calculation can be a quick way to detect the need for a delta scan. But it doesn't solve the problem.

We built a simple test case to demonstrate our requirements and the expected result:

• The base project is a simple spring boot project with a simple set of dependencies:

<dependencies>
        <dependency>
	        <groupId>org.springframework.boot</groupId>
	        <artifactId>spring-boot-starter-validation</artifactId>
        </dependency>
        <dependency>
	        <groupId>org.springframework.boot</groupId>
	        <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
	        <groupId>org.springframework.boot</groupId>
	        <artifactId>spring-boot-starter-test</artifactId>
	        <scope>test</scope>
        </dependency>
</dependencies>

This set of depenencies models our baseline project andn would be passed over to the lawyer for review

• in the second iteration, we introduce just one additional dependency (spring data mongodb), thus leading to this set of dependencies:

<dependencies>
        <dependency>
	        <groupId>org.springframework.boot</groupId>
	        <artifactId>spring-boot-starter-validation</artifactId>
        </dependency>
        <dependency>
	        <groupId>org.springframework.boot</groupId>
	        <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-data-mongodb</artifactId>
        </dependency>
        <dependency>
	        <groupId>org.springframework.boot</groupId>
	        <artifactId>spring-boot-starter-test</artifactId>
	        <scope>test</scope>
        </dependency>
</dependencies>

• Our expectation is that a delta scan results consists only of the spring boot data mongodb dependency and all the additional dependencies the spring data mongodb dependency relies on.

We have given it a try by using the orth subtract scan result function to build a difference, thus delta, between the two revisions but this didn't deliver a minimal set of add dependecies that we expected.

Implementationwise, I think ORT needs to be enhanced with a persistence capability to store the overall scan result per project and revision / timestamp of the scan, so that the difference between project iterations can be determined in the first place.
This persistence capabilty should support filesystem-based persistence as well as database- oder ReST-based persistence. And it should support an efficient way to determine the changeset between two project iterations.

[fviernau]

@rbieniek the terminology "delta scanning" (for ScanCode) was so far being used (hope it wasn't just me) for a non-functional transparent performance optimization, e.g. to be achieved by using (finer granularity) caching. So Porsche's delta scanning IMO is a different unrelated topic.

[nikpete]

@fviernau Understood, however, is there a need / interest to implement a functional delta scanning capability ( as described by Rainer) or is this just related to Porsche?

[sschuberth]

Let's phrase it like this: Before Porsche, no one else has publicly expressed this feature request 😉

Anyway, from reading through the past comments, my understanding is that you don't really want "a delta-scan feature for ScanCode", but actually a scanner-agnostic way of reporting only findings from changed dependencies. Correct?

If so, one could actually already handle this on the analyzer level, by only looking at the difference between a baseline analysis and the analysis of a newer version of the project, constructing a "fake project" from this which only contains the changes dependencies in the analyzer result, and using that to proceed with the scan.

[nikpete]

We want to report findings from any newly added, changed and deleted dependencies.

Just to give you a simple example:

Version 1.0:
100 dependencies identified, curated and uploaded in the legal workflow tool => all good

now version 1.1 is coming around which might consist of 110 dependencies with the following structure:

• 100 dependencies are the same as in version 1.0
• 15 dependencies newly added
• 5 dependencies have been deleted from version 1.0
• 10 dependencies updated

Conclusion: we need to idetify the "delta", meaning we need to be able to report the 30 dependencies that have been either deleted, added or updated.

Do we have here the same udnerstanding?

I would be very happy if this can be already handled on the analyzer level?

[sschuberth]

Do we have here the same udnerstanding?

I believe so, yes. Most important take-away for me: Your feature request is not related to ScanCode at all.

I would be very happy if this can be already handled on the analyzer level?

I currently believe it should be handled there, but I'm not saying it's already possible. Maybe yet another helper CLI sub-command to produce the delta between analyzer runs of the same project could become handy here.