What extension thing is not clear from the docs?

[teunbrand]

I'm currently reviewing the docs ggplot2 has on the ggproto base classes (tidyverse/ggplot2#6404) and was wondering if there are some things you personally had trouble with absorbing from the docs. Did you find particular things hard to find? Are some processes less clear than you'd like? I'm trying to fill any holes that might be in the docs.

[yjunechoe]

So you're talking about reference/ggplot2-ggproto.html (vs. articles/extending-ggplot2.html), right?

A small thought echoing what you already said in the linked issue is that it'd help if the data type of input and output are clearly stated for each ggproto method.

And one high-level comment is that I think that documentation should more heavily commit to the fact that the audience is folks who are looking to extend/subclass existing ggprotos (vs. creating them from scratch). That could help introduce a hierarchy of information that prioritize some fields/methods over others -- much needed here because there's just so much info. So ideally, more commentaries like this from the Stat doc would be great:

It's usually best to start by overriding compute_group: if you find substantial performance optimisations, override higher up. You'll need to read the source code of the default methods to see what else you should be doing.

It doesn't have to take the form of injecting a whole commentary, of course. But basically, I would like to see more division between fields/methods which are "often necessary to think about" vs. those that "can be ignored until you have to think about it". So kind of like the "required" vs. "optional" arguments distinction, except I get that it's awkward for the doc because everything is required to be present for the ggproto to function. But in the shoes of users who aren't creating them from scratch, re-drawing that distinction, even if very loosely, could maybe help digest all the info that's there in the docs

[yjunechoe]

There was a period of time when I cheekily experimented with a bunch of extensions via just Stat$finish_layer() and later realized that actually no one extends this method and for good reason (because whatever you want to do with the stat should probably be implemented elsewhere)

[teunbrand]

no one extends this method

I'm not disputing the vibe of the comment, but I know of only one place where this is used:

https://github.com/thomasp85/ggforce/blob/4f73e82a24304035fe459d30d0dab2f8a7832cee/R/sina.R#L289

I think one of the reasons you could in theory use this is to interpolate hex colour or somesuch as a stat, but it feels more like a geom task anyway.

[yjunechoe]

I think StatSina is exactly where I learned the method from, only to never see it in the wild again 😆

https://x.com/yjunechoe/status/1534597466590351361

[teunbrand]

I secretly think this is one of those 'developer privilages' in that you can implement something that makes your life easier in a niche capacity with only limited outside use. Lord knows I've put a thing here and there in the new guide system 👼(e.g. tidyverse/ggplot2#6090)

[yjunechoe]

a well-earned privilege! 😁

[EvaMaeRey]

This sounds great. I'd love to be able to follow some data through the whole plot build (so I'm imagining a pipeline/cascade). I think both schematics and some pipeline examples (maybe with intercepted dataframes along the way), would be very helpful. @pepijn-devries might have some ideas on this.

[willgearty]

You can embed a shiny app into R Markdown: https://bookdown.org/yihui/rmarkdown/shiny-embedded.html

[yjunechoe]

You can embed a shiny app into R Markdown: https://bookdown.org/yihui/rmarkdown/shiny-embedded.html

Just to be clear so as to not give Teun a heart attack -- if I make a shiny app I will not be trying to PR it into ggplot 😆

[yjunechoe]

Experimenting... https://fosstodon.org/@yjunechoe/114314676260404152

Untitled.video.-.Made.with.Clipchamp.4.mp4

[teunbrand]

Oh wow, impressive! 🎉

[yjunechoe]

Thanks! If I get this going more seriously imma start a separate thread for suggestions/feedback :)

[yjunechoe]

Oh and as for specifics of "what thing is unclear", I have a very hard time knowing when a grob-generating method should return a grob vs. grobTree vs. gtable, etc. Also, when and why (if ever) we should use internal functions like ggplot2:::ggname().

I think there are many cases like that where the doc says I should look at a method, I go look at the method, and I see a bunch of internal helper functions which aren't documented (and idk if I should read the source code to understand what they do and implement it for my own extension, or just ignore it as a quirk of "vanilla" ggplot2 ggprotos)

[teunbrand]

To be honest I also don't really know why ggplot2:::ggname() is a thing. I should dig a little bit 😅

I have a very hard time knowing when a grob-generating method should return a grob vs. grobTree vs. gtable, etc

Yeah I think just every method should have the return type documented.

[teunbrand]

I'm thinking of putting the following ASCII diagram in the docs. Does this make sense?

#' # Inside `ggplot_build()`
#'  |
#' layer(data)
#'  |
#'  |
#'  | # Inherit plot data
#'  |
#' Layer$layer_data()
#'  |
#'  |
#'  | # Finalise mapping
#'  |
#' Layer$setup_layer()
#'  |
#'  |
#'  | # Append PANEL variable for facets
#'  |
#'  |<- Layout$setup()
#'  |    |
#'  |    +-> Facet$setup_data()
#'  |    |
#'  |    +-> Coord$setup_data()
#'  |
#'  |
#'  | # Evaluate mappings to new data and infer group
#'  |
#' Layer$compute_aesthetics()
#'  |
#'  |
#'  | # Scale-transform all aesthetics
#'  |
#'  |<- ScalesList$transform_df()
#'  |    |
#'  |    +-> Scale$transform_df()
#'  |
#'  |
#'  | # Map x/y aesthetics with initial scale
#'  |
#'  |<- Layout$map_position()
#'  |    |
#'  |    +-> Scale$map()
#'  |
#'  |
#'  | # Compute stat part of layer
#'  |
#' Layer$compute_statistic()
#'  | |
#'  | +-> Stat$setup_data()
#'  | |
#'  | +-> Stat$compute_layer()
#'  |
#'  |
#'  | # Add `after_stat()` stage
#'  | # Scale transform computed variables
#'  |
#' Layer$map_statistic()
#'  |
#'  |
#'  | # Setup geom part of layer
#'  |
#' Layer$compute_geom_1()
#'  | |
#'  | +-> Geom$setup_data()
#'  |
#'  |
#'  | # Apply position adjustments
#'  |
#' Layer$compute_position()
#'  | |
#'  | +-> Position$use_defaults()
#'  | |
#'  | +-> Position$setup_data()
#'  | |
#'  | +-> Position$compute_layer()
#'  |
#'  |
#'  | # Map x/y aesthetics with final scales
#'  |
#'  |<- Layout$map_position()
#'  |    |
#'  |    +-> Scale$map()
#'  |
#'  | # Map non-position aesthetics
#'  |
#'  |<- ScalesList$map_df()
#'  |    |
#'  |    +-> Scale$map()
#'  |
#'  |
#'  | # Fill in defaults and fixed aesthetics
#'  |
#' Layer$compute_geom_2()
#'  | |
#'  | +-> Geom$use_defaults()
#'  |
#'  |
#'  | # Apply final Stat hook
#'  |
#' Layer$finish_statistics()
#'  | |
#'  | +-> Stat$finish_layer()
#'  |
#'  |
#'  | # Apply final Facet hook
#'  |
#'  |<- Layout$finish_data()
#'  |    |
#'  |    +-> Facet$finish_data()
#'  |
#'  V
#' # `ggplot_build()` is finished
#' # Hand off to `ggplot_gtable()`
#'  |
#'  |
#'  | # Draw the geom part
#'  |
#' Layer$draw_geom()
#'  |
#'  +-> Geom$handle_na()
#'  |
#'  +-> Geom$draw_layer()

[yjunechoe]

Hacky mermaid diagram conversion with claude 😆

See diagram

flowchart TD
title["ggplot_build()"]

layer["layer(data)"]
layer_data["Layer$layer_data()"]
setup_layer["Layer$setup_layer()"]
compute_aesthetics["Layer$compute_aesthetics()"]
compute_statistic["Layer$compute_statistic()"]
map_statistic["Layer$map_statistic()"]
compute_geom_1["Layer$compute_geom_1()"]
compute_position["Layer$compute_position()"]
compute_geom_2["Layer$compute_geom_2()"]
finish_statistics["Layer$finish_statistics()"]
draw_geom["Layer$draw_geom()"]

layout_setup["Layout$setup()"]
facet_setup_data["Facet$setup_data()"]
coord_setup_data["Coord$setup_data()"]

scaleslist_transform["ScalesList$transform_df()"]
scale_transform["Scale$transform_df()"]

layout_map_1["Layout$map_position()"]
layout_map_2["Layout$map_position()"]
scale_map_1["Scale$map()"]
scale_map_2["Scale$map()"]

stat_setup_data["Stat$setup_data()"]
stat_compute_layer["Stat$compute_layer()"]

geom_setup_data["Geom$setup_data()"]

position_use_defaults["Position$use_defaults()"]
position_setup_data["Position$setup_data()"]
position_compute_layer["Position$compute_layer()"]

scaleslist_map["ScalesList$map_df()"]
scale_map_3["Scale$map()"]

geom_use_defaults["Geom$use_defaults()"]

stat_finish_layer["Stat$finish_layer()"]

layout_finish_data["Layout$finish_data()"]
facet_finish_data["Facet$finish_data()"]

geom_handle_na["Geom$handle_na()"]
geom_draw_layer["Geom$draw_layer()"]

finished["ggplot_gtable()"]

title --> layer
layer -->|"Inherit plot data"| layer_data
layer_data -->|"Finalise mapping"| setup_layer
setup_layer -->|"Append PANEL variable for facets"| layout_setup

layout_setup --> facet_setup_data
layout_setup --> coord_setup_data

facet_setup_data --> compute_aesthetics
coord_setup_data --> compute_aesthetics

compute_aesthetics -->|"Evaluate mappings to new data and infer group"| scaleslist_transform
scaleslist_transform -->|"Scale-transform all aesthetics"| scale_transform

scale_transform --> layout_map_1
layout_map_1 -->|"Map x/y aesthetics with initial scale"| scale_map_1

scale_map_1 --> compute_statistic
compute_statistic --> stat_setup_data
compute_statistic --> stat_compute_layer

stat_setup_data --> map_statistic
stat_compute_layer --> map_statistic

map_statistic -->|"Add 'after_stat()' stage\nScale transform computed variables"| compute_geom_1
compute_geom_1 -->|"Setup geom part of layer"| geom_setup_data

geom_setup_data --> compute_position
compute_position -->|"Apply position adjustments"| position_use_defaults
compute_position --> position_setup_data
compute_position --> position_compute_layer

position_use_defaults --> layout_map_2
position_setup_data --> layout_map_2
position_compute_layer --> layout_map_2

layout_map_2 -->|"Map x/y aesthetics with final scales"| scale_map_2
scale_map_2 --> scaleslist_map
scaleslist_map -->|"Map non-position aesthetics"| scale_map_3

scale_map_3 --> compute_geom_2
compute_geom_2 -->|"Fill in defaults and fixed aesthetics"| geom_use_defaults

geom_use_defaults --> finish_statistics
finish_statistics -->|"Apply final Stat hook"| stat_finish_layer

stat_finish_layer --> layout_finish_data
layout_finish_data -->|"Apply final Facet hook"| facet_finish_data

facet_finish_data -->|"ggplot_build() is finished. Hand off to ggplot_gtable()"| finished
finished --> draw_geom

draw_geom -->|"Draw the geom part"| geom_handle_na
draw_geom --> geom_draw_layer

Loading

[teunbrand]

That is pretty good! A few submodules interpreted as parallel instead of sequential, but not too bad :D

[pepijn-devries]

Qmd documents natively support Mermaid diagrams.

[teunbrand]

I'm planning to render it to .Rd, so it probably won't be any fancier than ASCII 😅

[pepijn-devries]

You could include a pre-rendered diagram in an Rd file. See for instance https://github.com/pepijn-devries/ProTrackR/blob/17d1ceed77b871e9bc446a254095241e576e79c1/R/06PTModule.r#L43-L44

[pepijn-devries]

I have just browsed through https://ggplot2.tidyverse.org/articles/extending-ggplot2.html, most of it is pretty clear and makes sense. The only suggestion that I have is to change the structure a tad bit. To me the Creating a new stat-section comes a bit out of the blue. I think it would logical to first explain what 'stats' are and then tell how to create one. Most of the explanation is already there in that particular section, but I would make that into a separate introduction to stats (or perhaps layers in general).

[teunbrand]

Right, so would you suggest that every topic is introduced in a sentence or two, and perhaps links to further documentation like the book or docpages?

[pepijn-devries]

Yes, that's what I mean...

[EvaMaeRey]

If there is openness to changes to the vignette, I think showing that you can write a user-facing function that has a geom_* prefix even if you only do the work of writing a Stat could be illuminating to newcomers to the ggplot2 user community just about how layer are specified -- stat_ and geom_ functions can be made up of the same ingredients. Such geom_* functions falsely advertise the existence of a Geom with the same name, but they are not uncommon and are a pragmatic approach: 'ggplot2 users are accustomed to adding geoms, not stats, when building up a plot' (for better or worse!). I think a lot of newcomers want to write a geom_* for this reason, but might wrongly think they need to write a fresh Geom to do this.

Showing how to write a geom_* constructor with the same ingredients as the stat would mirror what is demonstrated in https://ggplot2-book.org/ext-springs.html#constructors (it even starts with defining a geom_* (!), and then following up with a stat_*)

geom_chull demonstrates this too, but a little bit less straightforwardly, GeomPolygon -> GeomPolygonHallow, and theres so much later in the vignette that I think the connection gets lost.