SymfonyCasts stands united with the people of Ukraine
If you liked what you've learned so far, dive in!
Subscribe to get access to this tutorial plus
video, code and script downloads.
With a Subscription, click any sentence in the script to jump to that part of the video!
Login SubscribeBy default, a message will be retried three times then lost forever. Well... in a few minutes... I'll show you how you can avoid even those messages from being lost.
Anyways... the process... just works! And it's even cooler than it looks at first. It's a bit hard to see - especially because there's a sleep in our handler - but this message was sent for retry #3 at the 13 second timestamp and it was finally handled again down at the 17 second timestamp - a 4 second delay. That delay was not caused by our worker just being busy until then: it was 100% intentional.
Check it out: I'll hit Ctrl+C to stop the worker and then run:
php bin/console config:dump framework messenger
This should give us a big tree of "example" configuration that you can put under the framework messenger config key. I love this command: it's a great way to find options that you maybe didn't know existed.
Cool! Look closely at the transports key - it lists an "example" transport below with all the possible config options. One of them is retry_strategy where we can control the maximum number of retries and the delay that should happen between those retries.
This delay number is smarter than it looks: it works together with the "multiplier" to create an exponentially growing delay. With these settings, the first retry will delay one second, the second 2 seconds and the third 4 seconds.
This is important because, if a message fails due to some temporary issue - like connecting to a third-party server - you might not want to try again immediately. In fact, you might choose to set these to way higher values so that it retries maybe 1 minute or even a day later.
Let's also try a similar command:
php bin/console debug:config framework messenger
Instead of showing example config, this tells us what our current configuration is, including any default values: our async transport has a retry_strategy, which is defaulting to 3 max retries with a 1000 millisecond delay and a multiplier of 2.
Let's make this a bit more interesting. In the handler, let's make it always fail by adding || true.
| ... lines 1 - 13 | |
| class AddPonkaToImageHandler implements MessageHandlerInterface, LoggerAwareInterface | |
| { | |
| ... lines 16 - 30 | |
| public function __invoke(AddPonkaToImage $addPonkaToImage) | |
| { | |
| ... lines 33 - 46 | |
| if (rand(0, 10) < 7 || true) { | |
| throw new \Exception('I failed randomly!!!!'); | |
| } | |
| ... lines 50 - 56 | |
| } | |
| } |
Now, under messenger, let's play with the retry config. Wait... but the async transport is set to a string... are we allowed to include config options under that? No! Well, yes, sort of. As soon as you need to configure a transport beyond just the connection details, you'll need to drop this string onto the next line and assign it to a dsn key. Now we can add retry_strategy, and let's set the delay to 2 seconds instead of 1.
| framework: | |
| messenger: | |
| ... lines 3 - 5 | |
| transports: | |
| # https://symfony.com/doc/current/messenger.html#transports | |
| async: | |
| dsn: '%env(MESSENGER_TRANSPORT_DSN)%' | |
| retry_strategy: | |
| delay: 2000 | |
| ... lines 12 - 20 |
Oh, and I also want to mention this service key. If you want to completely control the retry config - maybe even having different retry logic per message - you can create a service that implements RetryStrategyInterface and put its service id - usually its class name - right here.
Anyways, let's see what happens with the longer delay: restart the worker process:
php bin/console messenger:consume -vv
This time, upload just one photo so we can watch it fail over and over again. And... yep! It fails and sends for retry #1... then fails again and sends for retry #2. But check out that delay! 09 to 11 - 2 seconds - then 11 to 15 - a 4 second delay. And... if... we... are... super... patient... yea! Retry #3 starts a full 8 seconds later. Then it's "rejected" - removed from the queue - and lost forever. Tragic!
Retries are great... but I don't like that last part: when the message is eventually lost forever. Change the delay to 500 - it'll make this easier to test.
| framework: | |
| messenger: | |
| ... lines 3 - 5 | |
| transports: | |
| # https://symfony.com/doc/current/messenger.html#transports | |
| async: | |
| dsn: '%env(MESSENGER_TRANSPORT_DSN)%' | |
| retry_strategy: | |
| delay: 500 | |
| ... lines 12 - 20 |
Next, let's talk about a special concept called the "failure transport": a better alternative than allowing failed messages to simply... disappear.
Hey @Sven!
Sorry for the slow reply :). Hmm. It looks like I did that https://github.com/symfony/symfony/blame/b4d215c96fee8e208cdc82945b657a62913f9320/src/Symfony/Component/Messenger/Retry/MultiplierRetryStrategy.php#L31 ;)
The reason is mostly that, in Symfony, we tend to make things final until someone has a legitimate case for sub-classing them. And the answer is usually that you should use decoration. But I see your point about the service being abstract: https://github.com/symfony/symfony/blob/b4d215c96fee8e208cdc82945b657a62913f9320/src/Symfony/Bundle/FrameworkBundle/Resources/config/messenger.php#L157
However, you should be able to decorate the concrete version of this: a concrete service is made for each transport: https://github.com/symfony/symfony/blob/b4d215c96fee8e208cdc82945b657a62913f9320/src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php#L2275 - so decorate THAT service id, and I think it should work.
Let me know if that works :). It looks doable to me - though needing to hunt down these specific, concrete service ids is a bit tricky.
Cheers!
You mentioned in the video a way to override the way of retrying messages and implementing one on our own.
Apparently, it's terribly what I needed right now on my project and the scarce info regarding the subject, even after checking the documention, isn't really making it easy to implement.
Do you have any study material regarding it?
Hey @Vandarkholme
I haven't implemented a custom Retry strategy but I think you can do it without much troubles.
First, you have to create a class that implements Symfony\Component\Messenger\Retry\RetryStrategyInterface
Then, put your new services id in messenger config
// config/packages/messenger.yaml
framework:
messenger:
transports:
name:
retry_strategy:
service: App\Your\ServiceStrategy
And last, you have to implement the methods defined by such interface. You can give a look to this class to get an idea of what to do Symfony\Component\Messenger\Retry\MultiplierRetryStrategy
I hope this helps. Cheers!
Hey!
I think i found a bug in the messenger:
- i sent my messages as persistent (delivery_method=2) to RabbitMQ
- those messages in the actual queue were persistent (correct)
- when the worker consumes a message but it fails, the retry mechanism is called and a temporary queue is created where the message was sent
- this is correct, BUT and here comes the bug
- the message in that temp queue is not persistent, so if my retry mechanism has a delay of days... and the server/service stops/restarts those messages will be lost
Did i do something wrong or the bug really exists?
Thank you for your great work on symfonycasts!
Hey @Richard!
Ah, it is very possible this is a bug! But... it there *may* be a finished PR that will fix this - https://github.com/symfony/...
If you're interested, you could hack that onto your system and see if you get a better result - I would actually love to know if that fixes things for you. If it doesn't fix it, I would need to look closer, but indeed - my guess is that this would be a bug.
Cheers!
Thank Symfony Cast for the great video!
After we send a message with a DelayStamp to tell to execute the message in next 5 days
If later we want to extend the delay into 10 days, is it possible ?
Thanks,
Alex
Hey Nfq A.!
Hmm, so you want to send the message with a 5 day delay. Then, *before* it's processed, you want the ability to *extend* that delay to 10 days? I don't think that's easily possible. Basically, the nature of queues are that once you put them into a queue... you are just supposed to "dumbly" read them out when they become available. There would be no way to "ask" the queue for a specific message so that you could change the stamp.
However, this *is* possible... you just need to handle it in your app :). For example, you could set some database flag that says the message isn't ready to be processed yet (or even a date when it *should* be processed). Then, when the message is consumed after 5 days, your handler would check for this. If the message *is* ready to be processed, it would handle it. If not, it would take that message and re-send it into the message bus with a new, 5 day delay (the original 5 day delay + the new 5 day delay = 10 days).
I hope that helps!
Cheers!
Thank you Weaverryan,
That solution is very simple and totally fix my need!
(by the way I am very happy when received your answer the first time, I has followed your tutors for a long time, which helped improving my skills and mindset a lot! ^^ )
Happy day and keep the good work SymfonyCasts team!
Cheers!
Hey Nfq A.!
> That solution is very simple and totally fix my need
Yay, perfect!
> Happy day and keep the good work SymfonyCasts team!
Thanks to you for following us ❤️
Cheers!
// composer.json
{
"require": {
"php": "^7.1.3",
"ext-ctype": "*",
"ext-iconv": "*",
"composer/package-versions-deprecated": "^1.11", // 1.11.99
"doctrine/annotations": "^1.0", // v1.8.0
"doctrine/doctrine-bundle": "^1.6.10", // 1.11.2
"doctrine/doctrine-migrations-bundle": "^1.3|^2.0", // v2.0.0
"doctrine/orm": "^2.5.11", // v2.6.3
"intervention/image": "^2.4", // 2.4.2
"league/flysystem-bundle": "^1.0", // 1.1.0
"phpdocumentor/reflection-docblock": "^3.0|^4.0", // 4.3.1
"sensio/framework-extra-bundle": "^5.3", // v5.3.1
"symfony/console": "4.3.*", // v4.3.2
"symfony/dotenv": "4.3.*", // v4.3.2
"symfony/flex": "^1.9", // v1.18.7
"symfony/framework-bundle": "4.3.*", // v4.3.2
"symfony/messenger": "4.3.*", // v4.3.4
"symfony/property-access": "4.3.*", // v4.3.2
"symfony/property-info": "4.3.*", // v4.3.2
"symfony/serializer": "4.3.*", // v4.3.2
"symfony/validator": "4.3.*", // v4.3.2
"symfony/webpack-encore-bundle": "^1.5", // v1.6.2
"symfony/yaml": "4.3.*" // v4.3.2
},
"require-dev": {
"easycorp/easy-log-handler": "^1.0.7", // v1.0.7
"symfony/debug-bundle": "4.3.*", // v4.3.2
"symfony/maker-bundle": "^1.0", // v1.12.0
"symfony/monolog-bundle": "^3.0", // v3.4.0
"symfony/stopwatch": "4.3.*", // v4.3.2
"symfony/twig-bundle": "4.3.*", // v4.3.2
"symfony/var-dumper": "4.3.*", // v4.3.2
"symfony/web-profiler-bundle": "4.3.*" // v4.3.2
}
}
@weaverryan or anybody who knows; is there a specific reason why the MultiplierRetryStategy is marked @final? I want to reuse all of the logic, except for when a specific exception is thrown (in that case I want to change the waiting time). I can't decorate the service either, since it's marked abstract.