Parameterize a script.Pipe with a user data struct ?

[fbaube]

There's many pipeline packages "out there" but script seems to be one that gets it conceptually correct.

Question: Should it be possible to attach a data structure to a pipeline using generics ? Something like

type ParamPipe[T any] struct {
        UserData T 
        Pipe
}

Then a process pipeline for an instance of a user-defined struct could easily be constructed in a one-liner, and new functions could specifically process the user data.

I wish to write processing pipelines for chunks of content, and script's bash-style primitives provide a lot of helpful functionality.

[bitfield]

Thanks for the suggestion, @fbaube! Can you come up with an example of the kind of program you'd like to write using this idea? That'll help me get a clearer picture of how it might work.

[fbaube]

Can you come up with an example of the kind of program you'd like to write using this idea? That'll help me get a clearer picture of how it might work.

My goal is something like a DSL for processing Lightweight DITA. LwDITA is a DITA with a greatly reduced tag set, plus support for HTML5 and Markdown. This is where script would be used, and I would create new ParamPipe functions.

(As an aside, I figure that when I have M pipelines for M files, each with N processing stages, then there is a number of ways that this load could be distributed across multiple processors.)

So in the CLI program, the processing for a file looks (or will look) something like this:

• Gather CLI references to files and directories
• Expand directories into file lists
• Process in-file metadata (e.g. HTML )
• Read file content
• Analyze file content (MIME type? Is XML? Has DOCTYPE? Is valid XML? etc.)
• Parse file into an AST (e.g. using goldmark for Markdown, stdlib for HTML5 and XML)
• Extract "interesting" links (cross-references, ToC entries, etc.)
• (Note that up until this point, each file can be processed in isolation)
• Resolve and check validity of inter-file links
• Prepare file set for XSLT processing

fred

[bitfield]

That sounds great! So what would the script code look like to do this?

[fbaube]

Good question! I already have code that looks like this:

return p.
                st1a_ProcessMetadata().
                st1b_GetCPR().         // Concrete Parse Results 
                st1c_MakeAFLfromCFL(). // Abstract Flat List from Concrete Flat List 
                st1d_PostMeta_notmkdn()

A pure DSL tho would need to deal with how a list of N files fans out into N separate pipelines. I'm not sure whether script can do this, and it's not a typical task for a shell script either. I don't know whether there is a best practice for DSLs to do this.

To [Param]Pipe I would also add a debug io.Writer and a DB connection.

[mahadzaryab1]

@bitfield what're your thoughts on this issue?

[bitfield]

A generic pipe type is an intriguing idea—I don't think it would fit into script, but it could make an interesting package of its own.