Functional testing redesign

[MichaelClerx]

With the new database back-end, and Ben's stand-alone analysis in R, it might be good to rethink the code for functional testing.

I imagine we could have something like this:

1. External tool (Azure) schedules tests, updates things etc.
2. Current pfunk module runs tests and manages pints submodule, writes results to DB file
3. New R code reads DB file, runs analysis and generates plots
4. New module generates report, based on R code output
5. Script regenerates and updates website

So 3, 4 and 5 would be split off from pfunk, making it a test running tool.

Does that sound sensible @martinjrobins @fcooper8472 @iamleeg ?

[ben18785]

A somewhat related issue is, can we now add further test problems? I think there are a couple more problems that would be good to include (the toy problems are already implemented so it's just a case of inserting them into functional testing, I think).

…

On Fri, May 17, 2019 at 12:49 PM Michael Clerx ***@***.***> wrote: With the new database back-end, and Ben's stand-alone analysis in R, it might be good to rethink the code for functional testing. I imagine we could have something like this: 1. External tool (Azure) schedules tests, updates things etc. 2. Current pfunk module runs tests and manages pints submodule, writes results to DB file 3. New R code reads DB file, runs analysis and generates plots 4. New module generates report, based on R code output 5. Script regenerates and updates website So 3, 4 and 5 would be split off from pfunk, making it a test *running* tool. Does that sound sensible @martinjrobins <https://github.com/martinjrobins> @fcooper8472 <https://github.com/fcooper8472> @iamleeg <https://github.com/iamleeg> ? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#832?email_source=notifications&email_token=ABCILKASSX6RWZDHM7TWGPLPV2LVVA5CNFSM4HNUSVH2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GUMKMAA>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABCILKFAAUENVVN2XZKRDILPV2LVVANCNFSM4HNUSVHQ> .

[MichaelClerx]

We sure can! There are already some tickets open for that, but feel free to add more!

[MichaelClerx]

See https://github.com/pints-team/functional-testing#adding-tests for mini how-to

[iamleeg]

I think that's a great separation of concerns. If all of this stuff ends up in one tool it becomes tightly coupled, and workflows become constrained (e.g. you can only do analysis when Azure triggers a CI build)

[martinjrobins]

Yup, think this sounds good, that pfunk becomes a test running tool only. I was also thinking of running the comparison matrix (optimisers/samplers versus toy models) through pfunk as well.

[fcooper8472]

Sounds good to me!

[MichaelClerx]

Not sure anymore, now that we have changepoint detection in python too!

[MichaelClerx]

Happy to keep it as it is