after_stat() history rabbit hole

[yjunechoe]

IDK what propelled me to do this but last night I went down a rabbit hole of how we got to after_stat() in ggplot.

The timeline as I understand it:

1. Pre-2007, not a concept that existed in ggplot1 (the geom/stat distinction also didn't exist then)
2. In 2007, the ..x.. syntax (ggplot2 initial commit)
3. In 2017, the short-lived calc(x) syntax that briefly existed between the v2-v3 transition
4. In 2018, the stat(x) syntax which marked v3.0 (and thus wiping any mention of calc(x) in NEWS for v3.0)
5. In 2020, the after_stat(x) syntax in v3.4, which also introduced stage() and after_scale()

And my own 5.5 contribution: the stage(after_stat = x) syntax whose status as off-label usage had kinda been up in the air but now codified into the docs in 2024 😛

From a quick and dirty code search on GitHub (with some false positives), I see:

• ..x.. has 15.9k hits (CRAN only: 328 hits)
• calc(x) has just 79 hits (CRAN only: a single hit from {ggformula})
• stat(x) has 2k hits (CRAN only: 53 hits)
• after_stat(x) has 6k hits (CRAN only: 578 hits)
• stage(after_stat = x) has a single hit from {ggpmisc} by @aphalo !! (note that stage() itself only has around 500 hits)

---

Some other fun stuff:

1. A footnote hot take from @mjskay's ggdist paper😆:

2. Oh the irony of calc(x)! https://github.com/tidyverse/ggplot2/pull/2319/files

[teunbrand]

I think the concept of delayed evaluation is enormously convenient and helps to propel you from a proficient ggplot2 user to an advanced user if you wield it properly. I don't really care too much about whether the verb is declarative or procedural as these terms mean very little to me (sorry June!). Behind the scenes some complicated data masking and selection of mapping variables is going on, which is a bit of a pain to deal with from a maintenance perspective. You may have noticed that after_stat is just another identity function whose special action is only obeyed because some evaluation shenannigans with plenty of edge cases is going on. If one were to design the system from scratch without worrying about backwards compatibility, what would a better design look like?

I'm not saying the following is ideal, but to get the conversation rolling, it might be something like:

aes(x = foo, y = bar, colour = qux, after_stat = list(x = x + 1, y = y ^ 2), after_scale = list(fill = alpha(colour, 0.5), y = sqrt(y)))

[mjskay]

I'm biased, but I think in an ideal world these shouldn't all be stuffed into a single aes(). As an alternative, see the ggbuilder syntax:

df |>
  ggplot(aes(x = condition, y = response, color = condition)) +
  geom_boxplot() +
  stat_("boxplot", aes(y = response)) |>
    geom_("label", aes(y = ymax, label = ymax)) |>
    remap(aes(fill = colorspace::lighten(color, 0.9)))

Stuffing it all into one aes() means the specifications are not attached syntactically to the thing they actually modify. Plus on the implementation side it seems to lead to the need for hiding/unhiding stuff from different stages, creating unexpected behavior in corner cases for users.

[teunbrand]

If this is translated under the hood to the corresponding stage()'d expressions, the same internal struggle takes place naturally 😁 I like the ggbuilder syntax but it is perhaps my ggplot2-addled brain that expected that stat_() would have the after-stat part of the mapping, but that is now a part of geom_() it seems like. In my mental model the after-stat part is a re-map, as you map computed variables to graphical properties. The after-scale part feels more like a re-use/re-target/re-assignment rather than a re-mapping as mapping the variable to a scale has already occurred at that stage. You can't map a numerical column to a colour/fill aesthetic anymore, for example (or maybe the first few integers that correspond to the base palette).

[mjskay]

I like the ggbuilder syntax but it is perhaps my ggplot2-addled brain that expected that stat_() would have the after-stat part of the mapping, but that is now a part of geom_() it seems like.

Yeah I feel that too. Perhaps it's that the aes() steps need to feel more like they are "in between" the data and stat or the stat and geom rather than tied to a stat / geom.

The after-scale part feels more like a re-use/re-target/re-assignment rather than a re-mapping as mapping the variable to a scale has already occurred at that stage.

Agreed, going back to this I think I picked the wrong name for remap(). I should change it when I next revise the package.

[yjunechoe]

"in between" the data and stat or the stat and geom rather than tied to a stat / geom

That's also the framing that made the whole thing click for me! So I also share the feeling that it'd be more helpful and transparent for users to reflect that semantics in the syntax.

Relevant gif from my rstudioconf slides 😄

[mjskay]

That would make it more generic, something like:

df |>
  ggplot(aes(x = condition, y = response, color = condition)) +
  geom_boxplot() +
  map_aes(y = response) |>
    stat_("boxplot") |>
    map_aes(y = ymax, label = ymax) |>
    geom_("label") |>
    map_aes(fill = colorspace::lighten(color, 0.9))

Getting kinda verbose though. I think that having aes() as an argument to the stat or geom is still in keeping with typical semantics of arguments to function calls. It's just the typical use of aes() with functions prefixed with geom_ --- which looks like you're passing things "to the geom" even if you're actually passing things to stat_identity() --- that makes it feel strange.