When the tests passed alone, but failed together
During a full sequential CI run, a familiar failure kept appearing: a test sent a message, waited for the expected output, and eventually timed out. Some of the same scenarios passed when I executed them individually. The assertions had not changed. Something about the full run was changing what the tests could observe.
I was working on a Python test automation framework for a backend integration service. The service has no user interface. It consumes domain events from a message broker, applies business rules, persists state in PostgreSQL, and publishes transformed output to another queue. Other flows accept requests and produce acknowledgements, rejections, or data replies.
The basic test sounds simple: send a message, verify the outbound message, then verify the stored state. In practice, the interesting behavior happens asynchronously, with no HTTP response to mark the end of a transaction. And in the main publication flow there was an additional complication: the emitted output did not carry the correlation ID from the input.
That detail shaped the framework. Later, it also helped explain why a large sequential run could fail even when smaller runs behaved normally.
Three observable contracts
AMQP supports a correlation ID, and it is useful in request–reply workflows: attach an ID to a request, receive a reply carrying that ID, and match the two. But not every emitted event is a reply, and a service is not required to copy the incoming identifier onto every new publication.
The service I was testing exposed three different behaviors. Request flows preserved a usable correlation ID, so the harness could wait for a specific reply. Publication flows emitted new events without that link, so the test had to detect that a new message arrived after it sent its input. Negative scenarios could legitimately produce no outbound message at all: silence within a defined observation window was part of the expected result.
Those are three different test contracts: match by ID, detect a new arrival, or establish that no expected output appeared during the wait. Treating all three as one generic “send and receive” operation would have hidden important assumptions.
An event-driven client with a synchronous test interface
The scenarios were written in pytest and pytest-bdd. At the feature level, a step should read naturally: send an event, then verify the result. The AMQP client, built with Apache Qpid Proton, works differently. A reactor handles connection events, opened links, incoming deliveries, and receiver credit.
I kept that event-driven machinery behind a synchronous-looking facade. A receiver runs on a background thread and maintains an in-memory view of received messages. Test steps call higher-level operations instead of handling AMQP callbacks themselves. The facade can wait by correlation ID, wait for a new arrival, or collect several messages associated with one request.
For the publication flow, the crucial rule is to record the receiver’s position before sending the input. Otherwise, a fast output might arrive after the send but before the test starts counting, leaving it waiting for a message that has already been received.
Conceptually, the operation looks like this. This is illustrative pseudocode, not code copied from the framework:
position = inbox.snapshot_arrival_count()
send_message(payload)
result = inbox.wait_for_message_after(position, timeout=TIMEOUT)The snapshot must represent actual message arrivals, not just the number of distinct correlation-ID keys in a dictionary. Multiple uncorrelated events can share an empty correlation-ID value, so counting unique keys would miss subsequent messages. A real implementation also needs to handle synchronization between the receiver thread and the test thread.
Even with correct counting, the strategy has a serious limitation: “the next message” means “my output” only when no other in-flight scenario—or unrelated producer—can put a competing message on that outbound queue.
The clue was in Artemis
When I investigated the intermittent timeouts, I ran the automated tests sequentially while monitoring the relevant queues in ActiveMQ Artemis. I also sent messages manually and compared what I saw in the broker. During manual testing, the consumer count on the outbound queue stayed at one. During the automation run, I noticed that the count increased above one.
That was the turning point. The framework created clients based on the combination of publisher and subscriber addresses. Different scenarios could publish to different inbound destinations while listening to the same outbound queue. Those scenarios therefore obtained different client instances, and each instance opened a receiver on that shared queue.
From the broker’s perspective, there were multiple competing consumers. The broker could distribute deliveries among them. But a test was only polling the message store associated with the client it had been given. If the output went to a different receiver, the active test saw silence and timed out, even though the service had emitted a message.
That also explained the contrast with manual testing. One manually established consumer had no other consumer to compete with. The full automation run gradually introduced competing receivers as different scenarios requested clients with different address combinations.
One receiving owner per outbound queue
The fix was to make receiver ownership explicit. Within a test session, an outbound queue gets one receiving owner. The first client that needs that queue opens the receiver; clients created later for the same outbound queue can still send to their own inbound destinations, but they do not open another receiver. Instead, they read from the owner’s shared inbox.
The message store is reset between sequential scenarios so that one scenario does not accidentally assert on an earlier one’s output. The key change is that the framework no longer competes with itself for deliveries.
Nothing about the system under test needed to change to address this class of timeouts. The framework had been listening in the wrong way. Once the receiving topology reflected the test’s assumptions, the intermittent failures caused by competing in-session receivers stopped.
Why I cannot just turn on parallel execution
It would be tempting to take the sequential suite and enable pytest-xdist to reduce execution time. But the system design has four separate constraints that make uncontrolled parallel execution unsafe. Fixing the consumer ownership issue inside one process does not solve the others.
01Arrival-based waiting has no scenario identity
If two scenarios send through the same outbound queue at the same time, both may wake when either output arrives. Without a correlation field or a reliably echoed business identifier, the framework cannot prove which message belongs to which scenario. Assertions may fail for the wrong reason or, worse, pass because the unexpected output happens to match.
02Parallel workers would reintroduce competing receivers
Each parallel worker is a separate process with its own session-scoped client cache. Each process could independently believe it owns the only receiver on a shared queue. The original competing-consumer problem would return across workers, even if it had been fixed within each worker.
03The tested service has shared configuration
Operational mode and output format are controlled through cluster-level configuration. Scenario setup can patch Kubernetes ConfigMaps and wait for the service to pick up the change. One worker changing the mode or format could alter the behavior of the service while another worker is halfway through a test. The same risk exists between separate pipeline builds; simply separating messaging clients would not eliminate it.
Receiving an output is not the end of the test
Even after the correct outbound message arrives, the test still needs to validate the database state. Message publication and persistence do not necessarily become observable at the same instant. Positive verification steps therefore poll for the expected record within a bounded window and then compare relevant stored fields against the input and expected behavior.
Negative scenarios require a different kind of reasoning. A timeout can establish that no expected outbound message was observed during the chosen window. The test can then check that the previously stored state remains unchanged. A single state check is a snapshot, however, not proof that an update could never occur later. If the system’s contract requires an extended period of stability, the assertion should observe that stability explicitly.
The same precision matters when comparing output payloads. Generated timestamps and other dynamic fields need targeted normalization before an exact comparison. When diagnostic text varies by locale, assertions should use stable identifiers or codes instead of matching an entire translated sentence. The goal is not to make assertions loose; it is to be strict about the behavior that is actually part of the contract.
What I would carry into the next framework
Treat connection topology as test logic
Which process owns a receiver, which queue it listens to, and which tests can observe its messages are part of the test’s semantics. They are not incidental setup details.
Make correlation assumptions visible
If a test matches by arrival instead of by identity, document the requirement for a quiet, single-in-flight path and enforce it. “The next output” is only a meaningful assertion under controlled conditions.
Observe before acting
For arrival-based waits, take the inbox snapshot before sending, and count arrivals correctly. A tiny race window can produce failures that look like network or service instability.
Define what silence proves
Negative tests can legitimately expect no message, but only within a stated observation window. Pair message-level assertions with state verification appropriate to the system’s timing guarantees.
Debug the harness as seriously as the product
When tests pass alone but fail together, I look at shared state, fixture lifetimes, environment changes and resource ownership before assuming there are dozens of unrelated product defects. In this case, watching the broker’s consumer count was more revealing than another round of test retries.
A correlation ID is a small piece of metadata. Its absence forced me to think much more carefully about what my tests were observing, and the Artemis investigation reminded me that the automation framework itself can change the behavior of a message-driven system.
This case study uses generalized system descriptions and illustrative pseudocode; it does not reproduce client source code, message data, or internal configuration.
