Skip to content

Recycler API #1401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 49 commits into from
Nov 14, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
b69adc2
Apply spotless automatically
jvz Jan 9, 2023
b46bc30
Migrate Recycler API to log4j-api
jvz Jan 9, 2023
156e747
Move Recycler API to spi and fix related queue factory classes
jvz Jan 14, 2023
b001831
Update ReusableMessageFactory to use Recycler API
jvz Jan 14, 2023
22d2db5
Merge branch 'master' into dev-master-recycler-api
jvz Jan 14, 2023
ade3184
Extract ReusableLogEvent interface and use Recycler API
jvz Jan 15, 2023
de66f20
Update RecyclerFactory API and use in more scenarios
jvz Jan 16, 2023
f2f43d7
Use a bounded queue for recursion
jvz Jan 16, 2023
633938f
Update StatusLogger to use MPMC queue
jvz Jan 17, 2023
ab7b640
Update log4j-api/src/main/java/org/apache/logging/log4j/spi/RecyclerF…
jvz Jan 22, 2023
0b96146
Minimize visibility of inner classes
jvz Jan 22, 2023
160670f
Fix RecyclerFactories
jvz Jan 22, 2023
13d498f
Rename defaultCleaner() to getDefaultCleaner()
jvz Jan 22, 2023
44744c5
Add more tests for ThreadLocalRecyclerFactory
jvz Jan 22, 2023
815c22c
Use constant for reusable messages
jvz Jan 22, 2023
583bf6b
Simplify LogEventFactory lookup
jvz Jan 22, 2023
b10e118
Merge branch 'master' into dev-master-recycler-api
jvz Jan 22, 2023
bcc1ee1
Add missing license header
jvz Jan 22, 2023
86c6ec4
Use Recycler API for LogBuilder
jvz Jan 22, 2023
994c0f6
Merge changes from upstream
vy Mar 20, 2023
d96b370
[LOG4J2-3647] Add global filter support to `LogBuilder`
ppkarwasz Jan 13, 2023
3ab7a69
Move `RecyclerFactory` type converter to `log4j-plugins`
vy Mar 20, 2023
6d2a5a2
Fix code typo in `AbstractLogger`
vy Mar 20, 2023
4834eaa
Make `HierarchicalCollections` thread safe
vy Mar 20, 2023
21abf71
Use a single cleaner in recyclers
vy Mar 20, 2023
3b1608c
Merge changes from `main`
vy Mar 20, 2023
baf4a9b
Merge changes from `main`
vy Mar 23, 2023
7cdd075
More structural fixes
vy Mar 24, 2023
dbf31c0
Merge remote-tracking branch 'origin/main' into dev-master-recycler-api
vy Mar 24, 2023
cd3cac5
Merge upstream changes for `SLF4JLogBuilder`
vy Mar 25, 2023
7299ef4
More changes
vy Mar 25, 2023
623c47c
More `ThreadLocal`-to-recycler refactoring
vy Mar 28, 2023
1ec22cb
More refactoring
vy Mar 28, 2023
3005574
Fix tests
vy Mar 29, 2023
b163e8f
Merge from `main`
vy Mar 29, 2023
13b507b
Remove `ThreadLocal`s in `Strings`
vy Mar 29, 2023
a9114d8
Remove `ThreadLocal`s in `JsonUtils`
vy Mar 29, 2023
6faaf20
Remove `ThreadLocal`s in `ParametrizedMessage`
vy Mar 29, 2023
cefdb05
Fix minor issues
vy Mar 29, 2023
777d76c
Test fixes
vy Mar 29, 2023
2b12ba2
More test fixes
vy Mar 29, 2023
c72f4c3
Make sure `LogBuilder` recycles `DefaultLogBuilder`
vy Apr 21, 2023
7b69544
Merge remote-tracking branch 'origin/main' into recycler-api-3.x
vy Apr 21, 2023
d82722e
Merge remote-tracking branch 'origin/main' into recycler-api-3.x
vy Apr 24, 2023
4c9a39a
Add RecyclerAware interface
ppkarwasz Apr 22, 2023
6c0fba8
Simplify `AbstractRecycler`
vy Apr 24, 2023
0392734
Make queue-size in `ThreadLocalRecyclerFactory` configurable
vy Apr 24, 2023
49db130
Merge remote-tracking branch 'origin/main' into recycler-api-3.x
vy Nov 14, 2023
1d44878
Merge remote-tracking branch 'origin/main' into recycler-api-3.x
vy Nov 14, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,7 @@ private Appender createAppender(final String name, final Log4j1Configuration con
resolveSyslogHost(syslogHost, host, port);
final org.apache.logging.log4j.core.Layout messageLayout = LayoutAdapter.adapt(layout);
final Log4j1SyslogLayout appenderLayout = Log4j1SyslogLayout.newBuilder()
.setConfiguration(configuration)
.setHeader(header)
.setFacility(Facility.toFacility(facility))
.setFacilityPrinting(facilityPrinting)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import org.apache.log4j.builders.AbstractBuilder;
import org.apache.log4j.config.PropertiesConfiguration;
import org.apache.log4j.xml.XmlConfiguration;
import org.apache.logging.log4j.core.config.DefaultConfiguration;
import org.apache.logging.log4j.core.layout.HtmlLayout;
import org.apache.logging.log4j.plugins.Namespace;
import org.apache.logging.log4j.plugins.Plugin;
Expand Down Expand Up @@ -78,6 +79,7 @@ public Layout parse(final PropertiesConfiguration config) {

private Layout createLayout(final String title, final boolean locationInfo) {
return LayoutWrapper.adapt(HtmlLayout.newBuilder()
.setConfiguration(new DefaultConfiguration())
.setTitle(title)
.setLocationInfo(locationInfo)
.build());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import org.apache.log4j.config.PropertiesConfiguration;
import org.apache.log4j.layout.Log4j1XmlLayout;
import org.apache.log4j.xml.XmlConfiguration;
import org.apache.logging.log4j.core.config.DefaultConfiguration;
import org.apache.logging.log4j.plugins.Namespace;
import org.apache.logging.log4j.plugins.Plugin;
import org.w3c.dom.Element;
Expand Down Expand Up @@ -73,6 +74,6 @@ public Layout parse(final PropertiesConfiguration config) {
}

private Layout createLayout(final boolean properties, final boolean locationInfo) {
return LayoutWrapper.adapt(Log4j1XmlLayout.createLayout(locationInfo, properties));
return LayoutWrapper.adapt(Log4j1XmlLayout.createLayout(new DefaultConfiguration(), locationInfo, properties));
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import org.apache.logging.log4j.core.Layout;
import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.core.StringLayout;
import org.apache.logging.log4j.core.config.Configuration;
import org.apache.logging.log4j.core.config.plugins.PluginBuilderFactory;
import org.apache.logging.log4j.core.layout.AbstractStringLayout;
import org.apache.logging.log4j.core.net.Facility;
Expand Down Expand Up @@ -85,7 +86,7 @@ public Log4j1SyslogLayout build() {
LOGGER.error("Log4j1SyslogLayout: the message layout must be a StringLayout.");
return null;
}
return new Log4j1SyslogLayout(facility, facilityPrinting, header, (StringLayout) messageLayout, getCharset());
return new Log4j1SyslogLayout(getConfiguration(), facility, facilityPrinting, header, (StringLayout) messageLayout, getCharset());
}

public Facility getFacility() {
Expand Down Expand Up @@ -147,9 +148,14 @@ public static <B extends Builder<B>> B newBuilder() {
private final LogEventPatternConverter dateConverter = DatePatternConverter.newInstance(dateFormatOptions);


private Log4j1SyslogLayout(final Facility facility, final boolean facilityPrinting, final boolean header,
final StringLayout messageLayout, final Charset charset) {
super(charset);
private Log4j1SyslogLayout(
final Configuration config,
final Facility facility,
final boolean facilityPrinting,
final boolean header,
final StringLayout messageLayout,
final Charset charset) {
super(config, charset);
this.facility = facility;
this.facilityPrinting = facilityPrinting;
this.header = header;
Expand All @@ -168,32 +174,36 @@ public String toSerializable(final LogEvent event) {
// so we generate the message first
final String message = messageLayout != null ? messageLayout.toSerializable(event)
: event.getMessage().getFormattedMessage();
final StringBuilder buf = getStringBuilder();

buf.append('<');
buf.append(Priority.getPriority(facility, event.getLevel()));
buf.append('>');

if (header) {
final int index = buf.length() + 4;
dateConverter.format(event, buf);
// RFC 3164 says leading space, not leading zero on days 1-9
if (buf.charAt(index) == '0') {
buf.setCharAt(index, Chars.SPACE);
final StringBuilder buf = stringBuilderRecycler.acquire();

try {
buf.append('<');
buf.append(Priority.getPriority(facility, event.getLevel()));
buf.append('>');

if (header) {
final int index = buf.length() + 4;
dateConverter.format(event, buf);
// RFC 3164 says leading space, not leading zero on days 1-9
if (buf.charAt(index) == '0') {
buf.setCharAt(index, Chars.SPACE);
}

buf.append(Chars.SPACE);
buf.append(localHostname);
buf.append(Chars.SPACE);
}

buf.append(Chars.SPACE);
buf.append(localHostname);
buf.append(Chars.SPACE);
}

if (facilityPrinting) {
buf.append(facility != null ? toRootLowerCase(facility.name()) : "user").append(':');
}

buf.append(message);
// TODO: splitting message into 1024 byte chunks?
return buf.toString();
buf.append(message);
// TODO: splitting message into 1024 byte chunks?
return buf.toString();
} finally {
stringBuilderRecycler.release(buf);
}
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,11 @@
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import org.apache.logging.log4j.core.Layout;
import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.core.config.Configuration;
import org.apache.logging.log4j.core.config.plugins.PluginConfiguration;
import org.apache.logging.log4j.core.layout.AbstractStringLayout;
import org.apache.logging.log4j.core.layout.ByteBufferDestination;
import org.apache.logging.log4j.core.layout.Encoder;
import org.apache.logging.log4j.core.util.Transform;
import org.apache.logging.log4j.plugins.Configurable;
import org.apache.logging.log4j.plugins.Plugin;
Expand All @@ -53,15 +56,16 @@ public final class Log4j1XmlLayout extends AbstractStringLayout {
@PluginFactory
public static Log4j1XmlLayout createLayout(
// @formatter:off
@PluginConfiguration Configuration configuration,
@PluginAttribute(value = "locationInfo") final boolean locationInfo,
@PluginAttribute(value = "properties") final boolean properties
// @formatter:on
) {
return new Log4j1XmlLayout(locationInfo, properties);
return new Log4j1XmlLayout(configuration, locationInfo, properties);
}

private Log4j1XmlLayout(final boolean locationInfo, final boolean properties) {
super(StandardCharsets.UTF_8);
private Log4j1XmlLayout(final Configuration configuration, final boolean locationInfo, final boolean properties) {
super(configuration, StandardCharsets.UTF_8);
this.locationInfo = locationInfo;
this.properties = properties;
}
Expand All @@ -76,16 +80,29 @@ public boolean isProperties() {

@Override
public void encode(final LogEvent event, final ByteBufferDestination destination) {
final StringBuilder text = getStringBuilder();
formatTo(event, text);
getStringBuilderEncoder().encode(text, destination);
final StringBuilder text = stringBuilderRecycler.acquire();
try {
formatTo(event, text);
final Encoder<StringBuilder> stringBuilderEncoder = stringBuilderEncoderRecycler.acquire();
try {
stringBuilderEncoder.encode(text, destination);
} finally {
stringBuilderEncoderRecycler.release(stringBuilderEncoder);
}
} finally {
stringBuilderRecycler.release(text);
}
}

@Override
public String toSerializable(final LogEvent event) {
final StringBuilder text = getStringBuilder();
formatTo(event, text);
return text.toString();
final StringBuilder text = stringBuilderRecycler.acquire();
try {
formatTo(event, text);
return text.toString();
} finally {
stringBuilderRecycler.release(text);
}
}

@SuppressFBWarnings(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.core.StringLayout;
import org.apache.logging.log4j.core.config.DefaultConfiguration;
import org.apache.logging.log4j.core.layout.PatternLayout;
import org.apache.logging.log4j.core.net.Facility;
import org.apache.logging.log4j.core.time.MutableInstant;
Expand Down Expand Up @@ -71,15 +72,18 @@ static Stream<Arguments> configurations() {
public void testSimpleLayout(final String expected, final Facility facility, final boolean header, final boolean facilityPrinting) {
final LogEvent logEvent = createLogEvent();
StringLayout appenderLayout = Log4j1SyslogLayout.newBuilder()
.setConfiguration(new DefaultConfiguration())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit-pick, though it shouldn't affect the test: all these layouts can use the same Configuration instance.

.setFacility(facility)
.setHeader(header)
.setFacilityPrinting(facilityPrinting)
.build();
assertEquals(expected, appenderLayout.toSerializable(logEvent));
final StringLayout messageLayout = PatternLayout.newBuilder()
.setConfiguration(new DefaultConfiguration())
.setPattern("%m")
.build();
appenderLayout = Log4j1SyslogLayout.newBuilder()
.setConfiguration(new DefaultConfiguration())
.setFacility(facility)
.setHeader(header)
.setFacilityPrinting(facilityPrinting)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
package org.apache.log4j.layout;

import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.core.config.DefaultConfiguration;
import org.apache.logging.log4j.core.impl.ContextDataFactory;
import org.apache.logging.log4j.core.impl.Log4jLogEvent;
import org.apache.logging.log4j.message.SimpleMessage;
Expand All @@ -34,7 +35,7 @@ public class Log4j1XmlLayoutTest {

@Test
public void testWithoutThrown() {
final Log4j1XmlLayout layout = Log4j1XmlLayout.createLayout(false, true);
final Log4j1XmlLayout layout = Log4j1XmlLayout.createLayout(new DefaultConfiguration(), false, true);

final Log4jLogEvent event = Log4jLogEvent.newBuilder()
.setLoggerName("a.B")
Expand All @@ -55,7 +56,7 @@ public void testWithoutThrown() {

@Test
public void testWithPropertiesAndLocationInfo() {
final Log4j1XmlLayout layout = Log4j1XmlLayout.createLayout(true, true);
final Log4j1XmlLayout layout = Log4j1XmlLayout.createLayout(new DefaultConfiguration(), true, true);

final StringMap contextMap = ContextDataFactory.createContextData(2);
contextMap.putValue("key1", "value1");
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@
*/
package org.apache.logging.log4j.message;

import org.apache.logging.log4j.spi.ThreadLocalRecyclerFactory;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;
Expand All @@ -25,35 +27,39 @@
*/
public class ReusableMessageFactoryTest {

private ReusableMessageFactory factory;

@BeforeEach
void setUp() {
factory = new ReusableMessageFactory(new ThreadLocalRecyclerFactory(8));
}

@Test
public void testCreateEventReturnsDifferentInstanceIfNotReleased() throws Exception {
final ReusableMessageFactory factory = new ReusableMessageFactory();
final Message message1 = factory.newMessage("text, p0={} p1={} p2={} p3={}", 1, 2, 3, 4);
final Message message2 = factory.newMessage("text, p0={} p1={} p2={} p3={}", 9, 8, 7, 6);
assertNotSame(message1, message2);
ReusableMessageFactory.release(message1);
ReusableMessageFactory.release(message2);
factory.recycle(message1);
factory.recycle(message2);
}

@Test
public void testCreateEventReturnsSameInstance() throws Exception {
final ReusableMessageFactory factory = new ReusableMessageFactory();
final Message message1 = factory.newMessage("text, p0={} p1={} p2={} p3={}", 1, 2, 3, 4);

ReusableMessageFactory.release(message1);
factory.recycle(message1);
final Message message2 = factory.newMessage("text, p0={} p1={} p2={} p3={}", 9, 8, 7, 6);
assertSame(message1, message2);

ReusableMessageFactory.release(message2);
factory.recycle(message2);
final Message message3 = factory.newMessage("text, AAA={} BBB={} p2={} p3={}", 9, 8, 7, 6);
assertSame(message2, message3);
ReusableMessageFactory.release(message3);
factory.recycle(message3);
}

private void assertReusableParameterizeMessage(final Message message, final String txt, final Object[] params) {
assertTrue(message instanceof ReusableParameterizedMessage);
final ReusableParameterizedMessage msg = (ReusableParameterizedMessage) message;
assertTrue(msg.reserved, "reserved");

assertEquals(txt, msg.getFormat());
assertEquals(msg.getParameterCount(), params.length, "count");
Expand All @@ -65,7 +71,6 @@ private void assertReusableParameterizeMessage(final Message message, final Stri

@Test
public void testCreateEventOverwritesFields() throws Exception {
final ReusableMessageFactory factory = new ReusableMessageFactory();
final Message message1 = factory.newMessage("text, p0={} p1={} p2={} p3={}", 1, 2, 3, 4);
assertReusableParameterizeMessage(message1, "text, p0={} p1={} p2={} p3={}", new Object[]{
new Integer(1), //
Expand All @@ -74,7 +79,7 @@ public void testCreateEventOverwritesFields() throws Exception {
new Integer(4), //
});

ReusableMessageFactory.release(message1);
factory.recycle(message1);
final Message message2 = factory.newMessage("other, A={} B={} C={} D={}", 1, 2, 3, 4);
assertReusableParameterizeMessage(message1, "other, A={} B={} C={} D={}", new Object[]{
new Integer(1), //
Expand All @@ -83,12 +88,11 @@ public void testCreateEventOverwritesFields() throws Exception {
new Integer(4), //
});
assertSame(message1, message2);
ReusableMessageFactory.release(message2);
factory.recycle(message2);
}

@Test
public void testCreateEventReturnsThreadLocalInstance() throws Exception {
final ReusableMessageFactory factory = new ReusableMessageFactory();
final Message[] message1 = new Message[1];
final Message[] message2 = new Message[1];
final Thread t1 = new Thread("THREAD 1") {
Expand Down Expand Up @@ -123,8 +127,8 @@ public void run() {
new Integer(3), //
new Integer(4), //
});
ReusableMessageFactory.release(message1[0]);
ReusableMessageFactory.release(message2[0]);
factory.recycle(message1[0]);
factory.recycle(message2[0]);
}

}
Loading