S3 Payloads
How Tarmac detects and retrieves large message payloads offloaded to S3.
SQS caps message bodies at 256 KB, so systems that move larger payloads put the real body in S3 and send a pointer through the queue — the pattern implemented by the Amazon SQS Extended Client and by plenty of homegrown variants. Tarmac detects these pointers, retrieves the S3 object, and shows you the actual payload instead of an opaque pointer blob.
What Tarmac Detects
Detection runs on every received message, trying these shapes in order:
- SQS Extended Client pointers — the canonical
["software.amazon.payloadoffloading.PayloadS3Pointer", {"s3BucketName": …, "s3Key": …}]body format (the reservedExtendedPayloadSizeattribute raises confidence when present). - SNS-wrapped pointers — if the body is an SNS notification envelope, detection re-runs on the inner message, so topics feeding queues work transparently.
- Custom full-path properties — body fields you configure that hold a full S3 location as
s3://bucket/keyor an S3 HTTPS URL. - Bucket + key property pairs — your configured property names (matched exactly) first, then common conventions (
s3BucketName/s3Bucket/bucketName/bucketwiths3Key/s3ObjectKey/objectKey/key, case-insensitive), searched up to 12 levels deep in the JSON body.
If your producers use their own field names, add them under Settings ▸ External payloads — custom bucket, key, and full-path property names all take precedence over the built-in conventions. Custom names are matched case-sensitively.
Automatic vs. Manual Retrieval
When a pointer is detected, Tarmac checks the object's size with a HeadObject call:
- At or below the auto-retrieve threshold (256 KiB by default, configurable in Settings) → the object is fetched immediately and the message shows an Auto-retrieved chip.
- Above the threshold, or size unknown → the message is marked Manual retrieval required with the object size; click Retrieve in the inspector to fetch it on demand. Manual retrieval is hard-capped at 10 MiB — Tarmac uses ranged reads and refuses objects beyond the cap rather than pulling gigabytes into a message viewer.
- Retrieval errors (permissions, deleted object) → a Retrieval failed chip with the error; Retrieve retries.
Retrieved payloads are stored in the local cache alongside the message, appear in a dedicated S3 tab (pretty-printed like the body), and get their own copy button. Rows with external payloads show an S3 chip in the message list, colored by status. A manually retrieved payload survives re-fetching the message — Tarmac won't throw away an object you already pulled.
You need s3:GetObject on the payload bucket
Payload retrieval uses your Tarmac credentials, not the producer's. If messages show "Retrieval failed" with an access error, grant s3:GetObject on the offload bucket to the identity Tarmac is connected as.
The pointer and the object have different lifetimes
Deleting a queue message does not delete its S3 object, and redriving/resending a pointer message sends the pointer, not a copy of the object. Two consequences: (1) a resent pointer message is only useful while the S3 object still exists — check the bucket's lifecycle rules before replaying old messages; (2) staged deletes in Tarmac never remove S3 objects, so offloaded payloads linger until your lifecycle policy cleans them up.
LocalStack
The AWS-compatible connection's LocalStack preset enables S3 path-style addressing, so extended-client payloads round-trip correctly against LocalStack's S3 too. See Connecting to AWS.