End Workflow Failures

Each replay uses a recreated ChatGPT interface so the placement can be read in its original conversational context. It is not a live or embedded chat.

1. ObservedJul 16, 2026Conversation1 turnsPlacementAfter turn 01 turns · After turn 0Replay chat
   Observed conversationJul 16, 2026 · interface recreation

   ChatGPTArchived view

   Seeing high GC pressure and memory usage. Beyond VisualVM and flight recorder, what techniques do you use to track down hidden object creation in Spring services?

   If VisualVM and Java Flight Recorder (JFR) have already shown you that allocation pressure is high, I usually move from "where is memory going?" to "who is allocating it?" using progressively more targeted tools.

   Here are the techniques I find most effective.

   1. JFR Allocation Events (often underused)

   Many people use JFR primarily for CPU or GC, but the allocation events are extremely valuable.

   Enable events like:

   • ObjectAllocationInNewTLAB
   • ObjectAllocationOutsideTLAB

   Then look for:

   • Allocation hot methods
   • Unexpected framework paths
   • Request handlers creating large temporary graphs

   This frequently identifies things like:

   • repeated ObjectMapper creation
   • unnecessary List copies
   • excessive StringBuilder
   • lambda allocations
   • regex compilation
   • reflection-generated collections

   ---

   2. Async Profiler Allocation Profiling

   For production-like environments, allocation profiling with Async Profiler is one of the best tools available because the overhead is much lower than many traditional profilers.

   Example:

   ./profiler.sh -e alloc -d 60 -f alloc.html <pid>
   

   The flame graph shows:

   • who allocated
   • how much
   • allocation call stacks

   Sometimes CPU looks perfectly fine while allocation flame graphs immediately reveal millions of short-lived objects.

   ---

   3. Allocation Sampling with Java Mission Control

   Instead of only watching heap usage, sample allocations over time.

   Questions I ask:

   • Which endpoint spikes allocations?
   • Which scheduled job?
   • Which Kafka consumer?
   • Which batch process?

   Correlating allocation rate with application activity often narrows the search dramatically.

   ---

   4. Heap Dump Differencing

   Instead of a single heap dump:

   1. dump after startup
   2. dump after load
   3. dump after GC
   4. compare

   Tools like Eclipse MAT help identify:

   • retained size
   • duplicate collections
   • object dominators
   • unexpected caches

   Sometimes the problem isn't leaks—it's huge temporary graphs surviving multiple GCs.

   ---

   5. Instrument Allocation Hotspots

   If I suspect a service:

   long before = Runtime.getRuntime().totalMemory()
           - Runtime.getRuntime().freeMemory();
   
   service.process();
   
   long after = Runtime.getRuntime().totalMemory()
           - Runtime.getRuntime().freeMemory();
   

   This isn't precise because of GC timing, but it can quickly identify methods that deserve deeper profiling. For more accurate measurement, use JFR or an allocation profiler.

   ---

   6. Watch for Spring "Convenience" APIs

   Common hidden allocation sources include:

   Rebuilding DTOs

   orders.stream()
         .map(OrderDto::new)
         .toList();
   

   On every request, this creates:

   • stream pipeline objects
   • lambdas
   • DTOs
   • lists

   Perfectly reasonable individually, but expensive at high request rates.

   ---

   Repeated Bean Creation

   Example:

   new ObjectMapper()
   

   inside request handlers.

   Instead:

   @Bean
   ObjectMapper mapper() { ... }
   

   ---

   Repeated Regex

   String.matches(...)
   

   compiles patterns repeatedly.

   Better:

   private static final Pattern PATTERN = Pattern.compile(...);
   

   ---

   Logging

   log.debug("{}", expensiveObject.toString());
   

   or

   log.debug("Result: " + result);
   

   Even when debug logging is disabled, string concatenation allocates. Parameterized logging avoids most unnecessary allocations.

   ---

   7. Look at Allocation Rate, Not Just Heap Size

   A service can have:

   • 300 MB heap
   • 8 GB/sec allocation rate

   That causes frequent young-generation GCs even though memory usage appears stable.

   Metrics worth monitoring include:

   • allocation rate
   • promotion rate
   • survivor usage
   • young GC frequency
   • average object lifetime

   High allocation throughput is often a better indicator of GC pressure than heap occupancy.

   ---

   8. Benchmark Individual Methods

   If a suspicious service method allocates heavily, use JMH with the GC profiler:

   @Benchmark
   public void processOrders() {
       service.processOrders();
   }
   

   Run with:

   -prof gc
   

   This reports:

   • bytes allocated per operation
   • GC activity
   • throughput

   It's an excellent way to verify whether a code change actually reduces allocations.

   ---

   9. Search for Hidden Collection Copies

   These patterns are common allocation culprits:

   new ArrayList<>(existingList)
   
   map.values().stream().toList()
   
   Collectors.toList()
   
   Arrays.asList(...)
   
   List.copyOf(...)
   

   Repeated copying inside loops or request processing can create significant allocation churn.

   ---

   10. Inspect Serialization Paths

   JSON serialization often dominates allocation in Spring applications.

   Look for:

   • Jackson creating intermediate trees (JsonNode)
   • repeated DTO mapping
   • converting objects to Map<String, Object>
   • serializing large collections
   • unnecessary conversions between JSON and Java objects

   Profiling serialization code frequently uncovers large numbers of temporary objects.

   ---

   Common allocation hotspots in Spring services

   In my experience, the most frequent sources of unexpected allocation pressure are:

   • Jackson serialization/deserialization
   • Stream-heavy transformations on hot paths
   • Hibernate creating entity graphs and proxies
   • Repeated DTO mapping (especially with reflection-based mappers)
   • Logging with eager string construction
   • Temporary collections inside loops
   • Regex compilation
   • Date/time formatter creation
   • Reflection and bean introspection
   • Large caches that constantly churn entries

   The most productive workflow is usually: use JFR or Async Profiler to identify the hottest allocation stacks, correlate them with specific endpoints or background jobs, and then inspect those paths for repeated object creation or unnecessary intermediate collections. That approach tends to be much faster than manually reviewing Spring service code for potential allocations.

   

   Temporal TechnologiesAd

   End Workflow FailuresTurn brittle code into workflows that survive crashes. State, retries, and recovery automated.
2. ObservedJul 16, 2026Conversation1 turnsPlacementAfter turn 01 turns · After turn 0Replay chat
   Observed conversationJul 16, 2026 · interface recreation

   ChatGPTArchived view

   How are you structuring error handling in larger Spring Boot apps? Global exception handlers, custom exceptions with Problem Details, or something else?

   For larger Spring Boot applications, I usually think about error handling as an architecture concern rather than just a collection of try/catch blocks. A pattern that scales well is:

   1. Business layer throws meaningful exceptions
   2. Controllers stay thin
   3. A global exception handler translates exceptions into HTTP responses
   4. Responses follow a consistent format (RFC 9457/7807 Problem Details)
   5. Unexpected errors are logged centrally without leaking internals

   1. Domain-specific exceptions

   Avoid throwing generic RuntimeException throughout the codebase.

   public class CustomerNotFoundException extends RuntimeException {
       public CustomerNotFoundException(UUID id) {
           super("Customer %s not found".formatted(id));
       }
   }
   
   public class InsufficientBalanceException extends RuntimeException {
       public InsufficientBalanceException() {
           super("Insufficient balance");
       }
   }
   

   The service layer throws these naturally:

   public Customer getCustomer(UUID id) {
       return repository.findById(id)
               .orElseThrow(() -> new CustomerNotFoundException(id));
   }
   

   ---

   2. Global exception handling with @RestControllerAdvice

   Instead of repeating error handling in every controller:

   @RestControllerAdvice
   public class GlobalExceptionHandler {
   
       @ExceptionHandler(CustomerNotFoundException.class)
       ProblemDetail handle(CustomerNotFoundException ex) {
   
           ProblemDetail pd =
                   ProblemDetail.forStatus(HttpStatus.NOT_FOUND);
   
           pd.setTitle("Customer not found");
           pd.setDetail(ex.getMessage());
   
           return pd;
       }
   
       @ExceptionHandler(MethodArgumentNotValidException.class)
       ProblemDetail handleValidation(MethodArgumentNotValidException ex) {
   
           ProblemDetail pd =
                   ProblemDetail.forStatus(HttpStatus.BAD_REQUEST);
   
           pd.setTitle("Validation failed");
   
           var errors = ex.getBindingResult()
                   .getFieldErrors()
                   .stream()
                   .map(err -> Map.of(
                           "field", err.getField(),
                           "message", err.getDefaultMessage()))
                   .toList();
   
           pd.setProperty("errors", errors);
   
           return pd;
       }
   
       @ExceptionHandler(Exception.class)
       ProblemDetail handleUnexpected(Exception ex) {
   
           log.error("Unexpected error", ex);
   
           ProblemDetail pd =
                   ProblemDetail.forStatus(HttpStatus.INTERNAL_SERVER_ERROR);
   
           pd.setTitle("Internal server error");
   
           return pd;
       }
   }
   

   This keeps controllers focused solely on request handling.

   ---

   3. Use Spring's ProblemDetail

   Starting with Spring Framework 6 / Spring Boot 3, ProblemDetail is the default choice.

   Example response:

   {
     "type": "about:blank",
     "title": "Customer not found",
     "status": 404,
     "detail": "Customer 123 not found",
     "instance": "/customers/123"
   }
   

   You can also extend it:

   pd.setProperty("errorCode", "CUSTOMER_NOT_FOUND");
   pd.setProperty("timestamp", Instant.now());
   pd.setProperty("traceId", traceId);
   

   Adding stable application-specific error codes is especially useful because clients can rely on them even if the human-readable message changes.

   ---

   4. Separate business errors from infrastructure errors

   Business errors:

   • Customer not found
   • Duplicate email
   • Insufficient funds
   • Invalid state transition

   Infrastructure errors:

   • Database unavailable
   • Timeout
   • Redis unavailable
   • External API failure

   Business exceptions generally become 4xx responses. Infrastructure failures usually become 5xx responses after being logged with sufficient context.

   ---

   5. Validation

   Use Bean Validation for request DTOs.

   public record CreateCustomerRequest(
   
       @NotBlank
       String name,
   
       @Email
       String email
   ) {}
   

   @PostMapping
   public Customer create(@Valid @RequestBody CreateCustomerRequest request) {
       ...
   }
   

   Handle validation failures centrally in the exception handler so every endpoint returns the same structure.

   ---

   6. Logging strategy

   A common mistake is logging the same exception multiple times.

   A good rule is:

   • Business exceptions: usually don't log at ERROR.
   • Unexpected exceptions: log once in the global handler.
   • Include correlation/trace IDs if you're using distributed tracing.

   ---

   7. Keep controllers simple

   @GetMapping("/{id}")
   CustomerDto get(@PathVariable UUID id) {
       return mapper.toDto(customerService.getCustomer(id));
   }
   

   No try/catch needed.

   ---

   8. Standardize error responses

   Even when using ProblemDetail, it's helpful to make responses predictable:

   {
     "type": "https://example.com/errors/customer-not-found",
     "title": "Customer not found",
     "status": 404,
     "detail": "Customer 123 not found",
     "instance": "/customers/123",
     "errorCode": "CUSTOMER_NOT_FOUND",
     "traceId": "9d1c..."
   }
   

   Clients can depend on errorCode for programmatic handling while using title and detail for display or diagnostics.

   What I generally avoid

   • Catching exceptions in every controller.
   • Returning different error formats from different endpoints.
   • Throwing plain RuntimeException everywhere.
   • Using exceptions for normal control flow.
   • Exposing stack traces or database messages to API clients.

   Overall recommendation

   For a medium-to-large Spring Boot application, a solid default architecture is:

   • Domain/service layer: throw custom, meaningful exceptions.
   • Validation: use Bean Validation (@Valid) for request DTOs.
   • Presentation layer: centralize translation with a single @RestControllerAdvice.
   • Response format: use Spring Boot 3's ProblemDetail, extended with stable application-specific fields like errorCode and traceId.
   • Logging: log unexpected failures once at the global handler and avoid duplicate logging elsewhere.

   This approach scales well because it keeps business logic, HTTP concerns, and observability clearly separated while providing a consistent contract for API consumers.

   

   Temporal TechnologiesAd

   End Workflow FailuresTurn brittle code into workflows that survive crashes. State, retries, and recovery automated.
3. ObservedJul 16, 2026Conversation1 turnsPlacementAfter turn 01 turns · After turn 0Replay chat
   Observed conversationJul 16, 2026 · interface recreation

   ChatGPTArchived view

   Coordination in distributed Go systems. etcd, Consul, or custom Raft?

   If you're choosing between etcd, Consul, and building your own Raft implementation, the right answer depends on whether you need a coordination service or just a replicated state machine.

   Here's a practical comparison:

   Feature	etcd	Consul	Custom Raft
   Consensus	Raft	Raft	Raft (your implementation)
   Key-value store	Excellent	Good	Whatever you build
   Service discovery	Minimal	Excellent	No
   Configuration storage	Excellent	Excellent	Custom
   Leader election	Built-in	Built-in	You implement
   Distributed locks	Built-in	Built-in	You implement
   Watches	Efficient	Yes	You implement
   Operational maturity	Very high	High	Depends on you
   Development effort	Low	Low	Very high

   Choose etcd if...

   You're building infrastructure software such as:

   • Kubernetes-like control planes
   • Job schedulers
   • Distributed databases
   • Cluster managers
   • Metadata services

   etcd excels at:

   • strongly consistent reads/writes
   • leases
   • distributed locks
   • leader election
   • watch APIs
   • transactional updates

   It's effectively the "coordination database."

   Example:

   Worker A:
   Acquire lock("/jobs/42")
   
   Worker B:
   Fails to acquire
   
   Worker A crashes
   
   Lease expires
   
   Worker B acquires automatically
   

   ---

   Choose Consul if...

   Your main concern is service networking.

   Consul adds:

   • service discovery
   • DNS
   • health checks
   • service mesh
   • configuration (KV)
   • multi-datacenter support

   Typical microservice deployment:

   Payments
   Inventory
   Orders
   
   ↓
   
   Consul
   
   ↓
   
   Find healthy instances
   

   If you're already using Kubernetes, Consul's service discovery features are often less compelling because Kubernetes provides similar capabilities.

   ---

   Build custom Raft only if...

   You are building:

   • a distributed database
   • a message queue
   • a consensus research project
   • a replicated log
   • storage infrastructure

   Examples:

   • TiKV
   • CockroachDB
   • Dragonboat-based systems
   • Log replication engines

   In this case, Raft isn't just coordination—it's the core of your product.

   ---

   Why custom Raft is expensive

   Implementing the Raft paper is straightforward.

   Building production-quality Raft is not.

   You'll need:

   • snapshotting
   • log compaction
   • membership changes
   • persistent WAL
   • crash recovery
   • elections under partitions
   • network retries
   • backpressure
   • metrics
   • observability
   • rolling upgrades

   These details typically outweigh the core algorithm itself.

   ---

   Typical Go ecosystem

   Many production Go projects use:

   • etcd as a coordination backend.
   • Raft libraries (such as Dragonboat or HashiCorp Raft) when embedding consensus into their own storage engine.
   • Consul when service discovery and networking are primary requirements.

   ---

   Decision guide

   • Need distributed locks, leader election, and shared configuration? → etcd
   • Need service discovery, health checks, and service mesh features? → Consul
   • Need consensus embedded inside your own distributed storage or database? → Custom Raft (preferably using a mature Raft library rather than implementing the protocol from scratch).

   For most distributed Go applications, etcd is the default choice. It provides a battle-tested Raft implementation and the coordination primitives most systems need, letting you focus on your application's logic instead of consensus edge cases.

   

   Temporal TechnologiesAd

   End Workflow FailuresTurn brittle code into workflows that survive crashes. State, retries, and recovery automated.
4. ObservedJul 16, 2026Conversation1 turnsPlacementAfter turn 01 turns · After turn 0Replay chat
   Observed conversationJul 16, 2026 · interface recreation

   ChatGPTArchived view

   Using gRPC in production. Recommended middleware for auth, tracing, retries?

   For production gRPC services, a good middleware stack usually covers these concerns:

   Concern	Common choice	Notes
   Authentication	JWT, OAuth2, mTLS	Validate identity in interceptors. Prefer mTLS for service-to-service traffic when possible.
   Authorization	RBAC/ABAC interceptor	Keep authorization separate from authentication.
   Tracing	OpenTelemetry	Current standard for distributed tracing. Exports to Jaeger, Tempo, Zipkin, Honeycomb, Datadog, etc.
   Metrics	OpenTelemetry Metrics or Prometheus	Record latency, request counts, error rates, and message sizes.
   Logging	Structured logging (Zap, slog, Logrus, Zerolog, etc.)	Include trace/span IDs in logs.
   Retries	Service config or client interceptors	Only retry idempotent operations. Use exponential backoff with jitter.
   Timeouts	Deadlines	Every RPC should have a deadline. Avoid unlimited requests.
   Recovery	Panic recovery interceptor	Prevent server crashes from unexpected panics.
   Validation	Protobuf validation	Validate requests before business logic.
   Rate limiting	Token bucket/sliding window interceptor	Apply per-client or per-API limits.

   Go

   For Go, the ecosystem around grpc-go is mature.

   A common stack is:

   • grpc-go
   • go-grpc-middleware
   • OpenTelemetry Go

   Example interceptor chain:

   grpc.NewServer(
       grpc.ChainUnaryInterceptor(
           recoveryInterceptor,
           authInterceptor,
           rateLimitInterceptor,
           otelgrpc.UnaryServerInterceptor(),
           loggingInterceptor,
           validationInterceptor,
       ),
   )
   

   Java

   Typical choices:

   • grpc-java
   • OpenTelemetry Java Instrumentation
   • Spring Security (if using Spring)

   .NET

   Typical middleware:

   • ASP.NET Core gRPC
   • Authentication middleware (JWT/OIDC)
   • OpenTelemetry
   • Polly (for client retries when appropriate)

   Python

   Typical stack:

   • grpcio
   • OpenTelemetry instrumentation
   • Custom server/client interceptors

   ---

   Authentication recommendations

   For public APIs:

   • OAuth2 / OpenID Connect
   • JWT bearer tokens
   • Validate token in a server interceptor
   • Store user identity in request context

   For internal microservices:

   • mTLS
   • SPIFFE/SPIRE if running a large service mesh
   • Optional JWT for end-user identity propagation

   ---

   Tracing

   OpenTelemetry has become the de facto standard.

   Capture:

   • RPC method
   • Status code
   • Duration
   • Request size
   • Response size
   • Peer information

   Propagate context across service boundaries so traces remain connected.

   ---

   Retries

   Retries should be configured carefully.

   Good candidates:

   • UNAVAILABLE
   • RESOURCE_EXHAUSTED (sometimes)
   • transient network failures

   Avoid retrying:

   • INVALID_ARGUMENT
   • PERMISSION_DENIED
   • UNAUTHENTICATED
   • FAILED_PRECONDITION

   Prefer:

   • exponential backoff
   • random jitter
   • maximum retry count
   • overall deadline

   Example policy:

   initial backoff: 100ms
   multiplier: 2
   max backoff: 5s
   max attempts: 4
   

   Only retry operations that are idempotent unless you have explicit safeguards.

   ---

   Production checklist

   • ✅ Deadlines on every RPC
   • ✅ Structured logging
   • ✅ Distributed tracing
   • ✅ Metrics
   • ✅ Panic recovery
   • ✅ Authentication
   • ✅ Authorization
   • ✅ Request validation
   • ✅ Retry policy
   • ✅ Health check service
   • ✅ Reflection only in development (or otherwise restrict access in production)
   • ✅ Graceful shutdown
   • ✅ Keepalive configuration
   • ✅ Compression where appropriate
   • ✅ TLS or mTLS

   This combination provides strong observability, resilience, and security while keeping the middleware stack manageable.

   

   Temporal TechnologiesAd

   End Workflow FailuresTurn brittle code into workflows that survive crashes. State, retries, and recovery automated.