This is how you can consume large SQS messages with JMS and Spring Boot
Spring Boot became ubiquitous in recent years and provided an opinionated way of integrating various pieces of technology. Working with JMS is no exception to that. Altought Amazon has got it own Java API for interacting with SQS, using it through JMS ensures that we’ll be able to use the same piece of code with another messaging infrastructure. After taking a look at a basic message consumer and producer setup, we dive into a more advanced use case, which is consuming large messages.
Get the working example
If you’re like most people, perhaps you’d like to see the big picture first and go into the details after that. I prepared a fully working example, which is available on GitHub and here you go.
% git clone firstname.lastname@example.org:springuni/springuni-examples.git
% cd springuni-examples
Create a local configuration file
.env with the following contents in the root of the project (
After starting the demo app, it will listen on port 5000. I would have liked to keep it simple and it just handles bare text messages. Spring’s automated message converter infrastructure doesn’t play a role here, as the article’s focus is how to deal with large SQS messages.
% mvn spring-boot:run -pl springuni-jms-sqs
% curl -H 'Content-Type: text/plain' http://localhost:5000/message -XPOST -d 'test'
If everything went well, you should see the following messages.
2017-08-09 19:13:01.443 INFO 29525 --- [nio-5000-exec-1] c.s.examples.jms.MessageProducer : Sending message test..
2017-08-09 19:13:02.069 INFO 29525 --- [nio-5000-exec-1] c.a.s.javamessaging.SQSMessageProducer : Message sent to SQS with SQS-assigned messageId: 1001f7ba-55fc-4bdb-8732-8f4d40343068
2017-08-09 19:13:02.069 INFO 29525 --- [nio-5000-exec-1] com.amazon.sqs.javamessaging.SQSSession : Shutting down SessionCallBackScheduler executor
2017-08-09 19:13:02.243 INFO 29525 --- [enerContainer-1] c.s.examples.jms.MessageConsumer : Received message test
I’m hoping you’re still with me and interested in seeing the details.
Nuances of working with SQS
SQS is an odd-one out from the point of view how message brokers operate in general.
SQS maximizes the size of messages at 256K
For exchanging data in larger chunks than that can be achieved in various ways. When I first faced this limitation, I applied gzip compression and encoded the compressed binary data with base64. That solution worked just fine for textual (JSON) data, however it required customization on both the producer’s and the consumer’s side. Furthermore compression itself doesn’t guarantee that the size of messages to be sent will never exceed the 256K limit. Amazon SDK provides an extended SQS client, which enables end users to exchange messages larger than 256K transparently without having to apply customization themselves. The extened SQS client leverages S3 for storing messages and only a reference to the stored object is being sent to queues.
SQS isn’t transactional
However, Spring Boot tries to set up JmsListenerContainerFactory as transactional. When JMS autoconfiguration is enabled JmsAnnotationDrivenConfiguration delegates configuring a DefaultJmsListenerContainerFactory to DefaultJmsListenerContainerFactoryConfigurer and that expects either a
JtaTransactionManager be present or is set the container factory’s
sessionTransacted property to true.
Annotation driven message listener configuration requires the queue name to be defined upfront
Arbitrary methods of a managed bean can be annotated with
@JmsListener(destination = "queueName") or alternatively
javax.jms.MessageListener can be implemented instead. Nevertheless, going for the first option is much more convenient as Spring intelligently converts the received message to various user defined data type through its MessageConverter infrastructure.
In SQS endpoint URLs identify queues and they also contain the queue’s name.
Such an URL looks like this:
Obviously, we can extract the queue name from an URL like this, however the way JMS can be setup with Spring Boot requires you to define the queue’s name directly.
In order to do be able to leverage Spring’s messaging infrastructure without having to hard code a JMS destination in the message consumer or having to repeat the queue’s name in the application’s configuration, we need to implement a custom DestinationResolver. That
DestinationResolver will eventually parse the endpoints URL of a SQS queue and we’ll have to fiddle with only a single application property.
Basic setup with SQS
For the showcasing how Amazon’s JMS messaging library plays with Spring, we need to setup the following dependencies.
amazon-sqs-java-extended-client-lib is only required if you would like to send large messages and 256K might not be enough. For basic use cases however, you can omit that.
The example is using
spring-boot-starter-web, because we produce and consume messages in the same application. Real world solutions however have these functionalities separated.
Producing JMS messages
For the sake of simplicity, we’ll be exchanging simple text messages. The aforementioned official Spring tutorial covers sending structured messages and it also explains how JSON messages are getting converted to/from simple Java POJOs.
We’re however focusing on the details of integrating SQS as our message broker instead.
For producing messages a simple REST controller (MessageProducer) is used which in turn puts the HTTP request’s body to an SQS queue. It’s fairly trivial to do just that, eventually there’s nothing specific to SQS in that piece of code.
Receiving JMS messages
Altought receiving messages (MessageConsumer) looks equally trivial at a first sight, the challenge here was to eliminate the requirement of having to define a hard-coded message destination.
Spring gives support for resolving destinations based on that destination name which is supplied by the destination attribute of the
@JmsListener annotation (DynamicDestinationResolver). As the application is listening on messages coming from a single queue, we don’t want to do that. Instead the queue’s name is to be determined at that time when the application boots.
StaticDestinationResolver gets initialized with a fixed queue name and it resolves destinations against that same queue name every time.
SqsProperties encapsulate all of the required properties required for creating an SQS client. Basically we need to define an AWS region, queue endpoint URL and AWS access/secret keys, thought the two latter can be omitted if the underlying EC2 container has got the necessary IAM roles.
In application.yml these properties are mapped to individual, upper cased application configuration options. Most of the time these kind of applications are deployed on EB (Elasticbeantalk) or ECS (Elastic Container Services) which supply configuration data as environment variables.
All of the configurations steps below are taken from AbstractSqsConfiguration. They were implemented as reusable building blocks and we would be inspecting them one-by-one in what follows.
Creating an AWS credentials provider
Connecting to AWS usually starts with authenticating to one of its services. As I mentioned above, the steps of having to supply access and secret keys can be omitted in case of IAM roles, yet creating an AWSCredentialsProvider (one way or another) is necessary.
When access and secret keys are supplied we try to authenticate with those static credentials first with a fallback to fetching credentials from ECS directly.
Creating a SQS client
Creating a SQS connection factory
SQSConnectionFactory is the concrete implementation of
javax.jms.ConnectionFactory and as such it’s the gateway between the standard JMS API and native access to SQS through its Java SDK.
Spring Boot makes working with JMS very easy. Under normal circumstances it’s perfectly enough to create register a single
javax.jms.ConnectionFactory bean and then it takes care of creating a message listener container and a JmsTemplate.
In case of SQS however, there some nuances we should take care of ourselves. I mentioned that SQS wasn’t transactional, but Spring Boot’s autoconfiguration mechanism tried to create a listener container factory that way.
JmsTemplate needs a destination and and that can be extracted from the given SQS endpoint URL and also it’s more convenient to have the queue name resolved automatically in contrast to having to hard-code it with
Using the SQS extended client
We’ve convered how to setup a basic JMS config, if you want to produce and consume message larger than 256K, keep reading.
Creating an S3 client
Amazon SQS client relies on S3 as a means of persisting large messages and only a reference is sent over SQS. In order to be able to leverage S3, a bucket and credetials for accessing S3 are required. The following piece of code demonstrates how an AmazonS3Client can be built.
Creating the bucket automatically upon the first initialization of the app is also taken care of, although that’s optional.
Creating an extended SQS client
Thereafter these two are linked through AmazonSQSExtendedClient and ExtendedClientConfiguration provides a way to customize how large messages should be handled. By default, only messages larger than 256K will be sent over S3, but a user-defined message size threshold can also be specified. It’s also possible to configure it in a way that all of the messages go through S3 regardless of their size.
I read somewhere that Spring makes simple things easy and complex things possible. This assertion proved to be true on many occasions (including this one), when I was dealing with configuring a sophisticated application infrastructure with it.
It took me approximately a half day of tweaking to get every aspect SQS large message handling with JMS and Spring Boot right. Eventually difficulties stemmed from that fact that SQS is a bit different from other message queuing solutions. For example, as I mentioned earlier, it isn’t transactional, but Spring Boot – being an opinionated framework – tries to configure it that way, as most messages brokers are transactional.
So in conclusion I can say that this setup haven’t been working just fine for a couple of month in three applications using SQS with large messages.