Async Emails with Messenger

Hello, I have the issue when the email sent by queue. Even if I added
$this->entrypointLookup->reset();
before each send email, I have no css in the one I got.
I spent lot of hours to find a solution but I have no idea how to debug...

SF 5.3 with all up to date
`
public function newEmail(string $email_to, string $name_to = ''): TemplatedEmail

{
    $this->entrypointLookup->reset(); //HERE

    return (new TemplatedEmail())
        ->to(new Address($email_to, $name_to));
}

`

SOmes screenshots :<a href="https://vgy.me/u/XOYsnu&quot;&gt; Console email</a> -<a href="https://vgy.me/u/coYjpI&quot;&gt; Console Message</a> -<a href="https://vgy.me/u/XQry8r&quot;&gt;Email got html source</a>

Thank you!

Hey Christophe R.!

Hmm, yes this sounds tricky. Ok, a few questions:

A) You mentioned that your email is sent by a queue. When you have this problem, are you actually routing the emails to some sort of "async" queue? And then when you run bin/console messenger:consume, THAT is when the email is sent (and the styles are missing)? Or are you sending to Messenger, but then the message is actually handled "sync". If it is this situation, you might have this problem: https://github.com/symfony/symfony/issues/34972

B) In general, I think that your message is either being rendered multiple times for some reason (and so, the 2nd render has no styles) or you are handling your messages in a worker... and after it sends the first email, encore needs to be reset so that it works properly for the 2nd message... though I think that Encore is automatically reset between handling messages via messenger:consume... but I'm not 100% positive.

To help debug, I would put some debugging code inside this method - https://github.com/symfony/symfony/blob/5.4/src/Symfony/Component/Mailer/Messenger/MessageHandler.php#L29-L32 - that's the Messenger handler class that is called to actually SEND the email message. I would be interested to see if it's being called multiple times.

Let me know what you find out :).

Cheers!

Thanks for your answer!

This morning, in local dev I did another test, I Put a dump($message) in MessageHandler.php and I ask 2 requests passwords (email). They are in the queue (so no email sent) and in the profiler, the email section looks to be ok.

When I consumed, the first message has the css and the second not (but it is 2 differents email because 2 tokens differents for the reset)

Here the dump

I follow all your courses about API Platform, async email and symfony stuff so my code inspired by you and I added the reset :
$this->entrypointLookup->reset();
in the constructor of my Mailer service. (May be it need to be another place?)

If I send message with MESSENGER as doctrine default, I have the same issue.

If I remove messenger from composer, It looks to be ok each time, So I am thinking the issue is with Messenger and Mailer...

I am not sure to understand the A), but I think it is the case where the email looks to be good when sending in the queue and when I consume, css is missing with the second one (with my morning example)

Tricky fact, If when I sent another email in the queue while consume is running, the css is missing.
IF I stop consume, resent email, reconsume, the email has his css...

Hey Christophe R.!

This is excellent debugging :). So the issue is fairly simple. When you run Messenger (without restarting), the first email will look correct. But then, when it sends a second email, nothing has called $entrypointLookup->reset() yet... so the CSS is missing. Adding that line to your Mailer service won't help, because that code isn't being executed in the worker: it's calling Symfony's mailer service directly.

So, hmm. I know how to fix this... but I'm surprised that I haven't seen this before as this should be affecting anyone who is using Encore in their emails with Messenger. I thought (but now I think I was wrong) that Messenger was "resetting" the Symfony services between messages (which would call reset() on the entrypoint lookup. We use emails and Messenger on our site, but we don't use Encore yet (we had a system already that built the email HTML & CSS outside of Symfony).

Anyways, here's the fix: register a listener/subscriber on the WorkerMessageReceivedEvent event. Inside, if the message is an instance of SendEmailMessage - https://github.com/symfony/symfony/blob/5.4/src/Symfony/Component/Mailer/Messenger/SendEmailMessage.php - then call $this->entrypointLookup->reset().

That'll reset the Encore stuff before each email is actually delivered. On my end, I need to add a note about this to the tutorial. Let me know how it goes and if you need any more guidance!

Cheers!

Actually, here is the class you'll need: https://gist.github.com/wea... - try that out and let me know if you have any issues :)

Wowwww Ryan, you are too strong! It's WORK ! Thank you!
I will test in prod next week but no more issue in local dev!

Awesome! Thank you for confirming the fix! We'll add a note to the video for it soon - I appreciate you brining up the issue :)

Trying to use the starter code when I'm running symfony 4.4 has caused me so many composer headaches
On another note, I'm getting this error in my console from the messenger:

[messenger] Error thrown while handling message Symfony\Component\Mailer\Messenger\SendEmailMessage. Sending for retry #1 using 1000 ms delay. Error: "Connection to "smtp.mailtrap.io:2525" timed out."
["message" => Symfony\Component\Mailer\Messenger\SendEmailMessage { …},"class" => "Symfony\Component\Mailer\Messenger\SendEmailMessage","retryCount" => 1,"delay" => 1000,"error" => "Connection to "smtp.mailtrap.io:2525
" timed out.","exception" => Symfony\Component\Messenger\Exception\HandlerFailedException { …}]

And it tries this 3 times before removing the message from the transport.

BUT the emails ARE being sent. So why is there an error?

There was also an error message mentioning an unexpected response code:

[messenger] Error thrown while handling message Symfony\Component\Mailer\Messenger\SendEmailMessage. Sending for retry #2 using 2000 ms delay. Error: "Expected response code "354" but got code "250",
with message "250 2.1.0 Ok"." ["message" => Symfony\Component\Mailer\Messenger\SendEmailMessage { …},"class" => "Symfony\Component\Mailer\Messenger\SendEmailMessage","retryCount" => 2,"delay" => 2000,"error" => "Expecte
d response code "354" but got code "250", with message "250 2.1.0 Ok".","exception" => Symfony\Component\Messenger\Exception\HandlerFailedException { …}]

Hey Nick F.!

Oh no! Do you remember (other than the xsl extension) what Composer issues you had? Did you download the "start" code and then upgrade to 4.4? Or just start a new 4.4 project and "adapt" the code into it? If there is an issue, I'd like to know so that we can smooth it out.

[messenger] Error thrown while handling message Symfony\Component\Mailer\Messenger\SendEmailMessage. Sending for retry #1 using 1000 ms delay. Error: "Connection to "smtp.mailtrap.io:2525" timed out."
["message" => Symfony\Component\Mailer\Messenger\SendEmailMessage { …},"class" => "Symfony\Component\Mailer\Messenger\SendEmailMessage","retryCount" => 1,"delay" => 1000,"error" => "Connection to "smtp.mailtrap.io:2525 " timed out.","exception" => Symfony\Component\Messenger\Exception\HandlerFailedException { …}]
And it tries this 3 times before removing the message from the transport.
BUT the emails ARE being sent. So why is there an error?

Hmm. Is the email sent just ONE time, or is it sent 3 times? I suspect just once - but I want to be sure. Also, if you stop running messenger:consume and THEN trigger the email to be sent (i.e. register), is the email sent? I can't think of why this would be happening: you are clearly "routing" the email to an async transport, which means that Messenger will no longer be handling it sync. However, my first guess is that is IS somehow being handed sync and then also sent to the transport. But... as I said, that's just a guess.

[messenger] Error thrown while handling message Symfony\Component\Mailer\Messenger\SendEmailMessage. Sending for retry #2 using 2000 ms delay. Error: "Expected response code "354" but got code "250",

Hmm. So, in the SMTP internals, when Symfony sends a message it does this:

1) it Sends a "DATA" line, which tells the SMTP server that you're about to starting sending data. the SMTP server should reply with a 354 code, which means "Start mail input"

2) After Sending the data, Symfony sends a special ending syntax and expects the SMTP server to reply with 250, meaning "Requested action was completed".

For some reason, the SMTP server that you're using seems to be replying to the first with a 250. I have no idea why it would do that. Here's the core part of Symfony that I'm referencing: https://github.com/symfony/symfony/blob/083a32dee34bc0cb67f4d81262a39c132f8097e0/src/Symfony/Component/Mailer/Transport/Smtp/SmtpTransport.php#L202-L207

Is this coming from Mailtrap? Or something else? Let me know what you find out!

Cheers!

Hey Victor,

first of all thanks a lot for your tutorials. They helped me a lot to move my mindset from symfony 3.4 to 4.4. Big step. Your video helped especially for the main concepts of sf4.

As it seems, i run into a problem which is not very common, measured by search results i found on google. Today i found this tutorial and you use the same Mail setup as i do. So, hopefully you can help me out.
Over all i have a working setup after switching from SwiftMailer with a self written spooling system to the brand new mailer component with async dsn (sendgrid). Thanks to that, i could delete a lot of code ;)

After sending an email via mailer, i store the base email in the database to use the activity callback provided from sendgrid (like: deliverd, opened, clicked). Sendrid uses their own message id to communicate the status from the specific message. This is stored in the response header 'x-message-id'. In the code debug i can see that this header is correctly set by the sendgrid transport (SendgridApiTransport:59). It's stored in the SentMassge Object.

Here is the thing: I use your code from the "messenger" tutorial with the EmailMiddleware. With this approach i cant access the SentMessage Object anymore. I only got the TemplatedEmail Object. The TemplatedEmail Object does not store the Header, given by Sendgrid response.
The return value of getResult() from HandledStamp is null.

I hope you have a hint what i do wrong at the moment. When i read the changes from Fabien, related to this topic, it should work (https://github.com/symfony/... but i have no clue what i'm doing wrong.

Hopefully you have an idea.

Hey Matthias,

Thank you for kind words about SymfonyCasts tutorial! I'm really happy to hear it!

Hm, EmailMiddleware is something that you created, right? You created this for getting access to SentMessage? Are you sure your middleware is called in the correct order? And are you sure your middleware is called at all? Probably you need to call you middleware a bit later?

Did you try to debug the stacktrace with Blackfire? It might help to track the chain of calls that you can follow and if there's any chances to get that SentMessage object.

P.S. I looked into code a bit, and it looks like when we're talking about Messenger, the message is sent in MessageHandler::__invoke(), i.e. there's where the send() method is called which in turn return SentMessage object, see: https://github.com/symfony/... - but I don't see any return statements in this MessageHandler::__invoke() that would help to access SentMessage with Messenger :/

I hope this helps somehow. Please, let us know if it's not.

Cheers!

OMG. It seems that it's the missing "return" in the default MessageHandler. When i update the Class with the return statement, i can access the SentMessage Object from the getResult Method delivered by HandledStamp. Cool.

So that i can go forward with my project code without change symfony codebase, i created my own MessageHandler to handle the SendEmailMessage. Registered it over the services tag "mailer.messenger.message_handler". A kind of expected, the handlers are both registered.

services.yaml
`
services :
[...]

App\MessageHandler\SendEmailMessageHandler:
    tags:
        - name: mailer.messenger.message_handler
          handles: Symfony\Component\Mailer\Messenger\SendEmailMessage

`

result of debug:messenger
`
The following messages can be dispatched:

Symfony\Component\Mailer\Messenger\SendEmailMessage

  handled by App\MessageHandler\SendEmailMessageHandler  
  handled by mailer.messenger.message_handler            

`

Is there an other way to really overwrite or replace the default MessageHandler? Because at the moment both handler sending an Email ... again: as expected :)

Looking really forward to the "final fix".

Hey Matthias,

Glad this workaround works for you! I wonder if that "return" statement should be in the Symfony code base by default... probably it's a good idea to create a PR about it and see the feedback of community.

Hm, I suppose it should be sent only once... how does your messenger.yaml file config look like? Especially your routing section there. Also, I though the correct tag should be "messenger.message_handler" according to the docs, no? See https://symfony.com/doc/cur...

Cheers!

I used to send mails synchronous and all worked. Now I tried to make the switch to async.

Now $mailer->send($email); throws an exception stating Serialization of 'Closure' is not allowed ultimately originating from PhpSerializer.php line 61.

I clicked through many steps of code with Xdebug but cannot figure out where that damn closure comes from. Any suggestions on how to fix the issue?

Upon further investigation it looks like Symfony is trying to serialize all the Doctrine Objects passed to the mail via 'context' and though I haven't found the closure yet, it is probably somewhere inside those entities. But IMHO it is definetly not neccessary to serialize all of that just for an async mail to be sent. I'd still like to know how to handle closures with Messenger but for now it should suffice to render the mail template and then serialize it. Is there a way to do so?

EDIT: Okay so I've found at least a closure. As mentioned above all the Doctrine Objects are passed to the serializer. And with them Entity Manger Object(s ?). They have LazyObjectState and there is an initializer that is a Closure.

... Leading me back to my question: Is it really neccessary to serialize ALL THAT? And how come this is not a common problem? I am probably not the only one using Doctrine Object when creating an email.

EDIT2: And a similar problem was just described over at Stackoverflow. Looks like those Lazy Ghost Entity Managers introduced in 6.2 are causing some trouble...

Hey Eric,

Yes, the problem is that when you make your messages async - you force them to be saved somewhere, e.g. in the DB, before sending via Messenger, and so to be properly saved they should serialize objects you pass to the emails. But thankfully, there's a simple solution for this! Just stop passing objects into your email template :) If you need a user first name and email in that - you don't need to pass the whole object like you do with page templates - just pass simple scalar values of user's first name and email.

Otherwise, if you really don't want to get rid of passing objects to the email template - then you probably need to worry about configuring your entities to be serialized properly. But I think the best practice is to pass just simple scalar values to the template instead.

Cheers!

Hey Victor,
thanks for the reply. I actually didn't take into consideration just passing scalar values. But as I also happen to be a lazy dev, I made it work with passing objects. It is probably a bit context dependend but my usecase is as follows: After handling a form a confirmation mail is send. This mail makes use of objects from the form data.
All this Lazy Ghost EntityManager Shenanigan gets inserted into the objects after they are flushed. So just before I do a clone
` $entityDTO = clone $entityObject;

        $entityManager->persist($entityObject);
        $entityManager->flush();
        ...
        $email = (new TemplatedEmail())
                    -> [...]
                    -> context([
                        'entity' => $entityDTO
                    ]);`

Looks like it all works just fine that way.

Hey Eric,

Interesting solution, thanks for sharing it with others! Glad to hear it works for you, but just in case add some tests to be calm on upgrades :)

Cheers!

Is there a way to get the event when a "Symfony\Component\Mailer\Messenger\SendEmailMessage" is handled? My purpose is for each email sent ( async ) successfully, I can run some code for that event. Is this possible ?

Hey Dang

An option would be to listen to the event Symfony\Component\Messenger\Event\WorkerMessageHandledEvent. Then, you can check if the message instance inside the envelop is equals to your email message. Does it make sense?

Cheers!

Hey MolloKhan ,
Thanks a lot for that hint. I've created a Event Subsciber to listen to that Symfony\Component\Messenger\Event\WorkerMessageHandledEvent.
It works. But actually, my email is sending inside a MessageHandler of "other" Message (I call the mailer->send(email) inside MessageHandler). So when I listen to WorkerMessageHandledEvent, I get back the Message when it done, but not the event of the mailer it self. I can see the line "Message Symfony\Component\Mailer\Messenger\SendEmailMessage handled by Symfony\Component\Mailer\Messenger\MessageHandler::__invoke" while the worker running. And after that is my MeesageHandler is handled.
So can I catch the event while the workers is running and for exact the mailer event?

Hey Trinh, sorry for my slow reply.

So, the email is sent via another message handler, in that case, your subscriber should be triggered twice, isn't it? one per each message been handled, unless, Symfony Messenger does not trigger the event when a message is dispatched inside a handler... I'm not sure about that but if that's the case, an option would be that you dispatch a custom event inside your MessageHandler class so you can listen to it

I have the most bizarre issue with the messenger component here. I have followed this guide to the letter, using Symfony 5.2.5, PHP 7.3, and have configured it to use doctrine async. I'm using sass and have this at the top of my base template:

{% apply inky_to_html|inline_css(encore_entry_css_source('email')) %}

When I send an email, it is stored in the messenger_messages table as expected, and then when I run messenger:consume -vv it picks out the email and sends it successfully, complete with all the styling from the sass files (the 'email' entry also contains foundation for emails).

BUT...

When I leave messenger:consume -vv running and send another email, the email is still sent, but none of the CSS is applied inline. It's still storing the email in the database, because the primary key is incrementing, and Inky still works to convert everything into tables, but there is no inline CSS at all - just the raw <table>, <tr>, <th> elements.

After trying for hours and hours to work out what was going on, I eventually removed the messenger component completely using composer, and now it works perfectly. I can only assume there is a problem with messenger being able to use encore_entry_css_source()?

Any advice would be greatly appreciated!

Hey Steven,

Yes, that's a known WebpackEncoreBundle problem, you need to reset its state every time you send an email. Take a look at the docs: https://github.com/symfony/... . And actually, we covered this issue in this course: https://symfonycasts.com/sc... - you probably missed this :)

So, bring back the Messenger and try this solution ;)

I hope this helps!

Cheers!

Thank you Victor! I had skipped the PDF video as I wasn't working with PDFs.

Hey Steven,

Ah, now I see why you missed this :) Yeah, a known problem for Webpack Encore, you just should be aware of it to be able to solve when needed. It's not a bug actually, it's just a compromise that makes Encore shine in other cases - it prevents rendering script and link tags twice when you use template inheritance in Twig :)

Cheers!

Hello, I use async messenger to send emails but now I have problems with the translator component. I assume that this is a known problem. The translation does not take place and I get the string I entered in the trans function:
$this->translator->trans('subject')
I'm injecting TranslatorInterface to a MessagHandlerInterface implementation class. I think it looses context druing processing. What can I do?

Hey Metin,

Yeah, I think I know your problem. It's similar to request context when you generate URLs in async emails - because there's no request, you need to set the request context to generate links properly, i.e. see https://symfony.com/doc/4.1... . What about translation - it's the same, no request, so no locale. This probably mean that you have to set the locale manually. You can do it via translator globally e.g. "$translator->setLocale('fr');" for example, or pass the correct locale to the "trans()" method - it should be possible via the last argument, see method signature. I'd recommend to try the first way and set the correct local manually on the translator before trying to translate.

I suppose you can even do it globally via events, listen to WorkerMessageReceivedEvent::class and set the correct locale - it should give you correct context when your emails will be sent async. I'm not sure, but probably you will also need to set the locale on ConsoleEvents::class event as well, but it might be redundant if your problem is only in async emails.

I hope this helps!

Cheers!

Is it possible to use use SwiftMailer with Messenger to queue emails?

Hey Shaun,

I don't know for sure, have never done this before, but I'd say no. Thanks to the new Symfony Mailer, messages are just data objects now, and so you can easily serialize/deserialize them. It was a problem with SwiftMailer because its messages were "heavy". I'd recommend you to refactor to Symfony Mailer to be able to use it with Messenger component that should work out of the box without extra coding. And most probably it shouldn't be a hard refactor, but depends on your project though.

I hope this helps!

Cheers!

Hello guys,

I'm having an issue with what seems to be "cache" when developing my email templates using Inky.

--> When modifying any template, I need to restart the messenger worker for my changes to be taken into account. Is it linked to messenger or is it linked to the "inky_to_html" method not parsing the template twice ? (like the styles where we need to manually reset)

I haven't found any related issues anywhere in the comments nor google and even though I could continue restarting my worker on local, it is getting somewhat annoying and I would like to find a more effective solution :)

Thanks in advance !

Hey Thibault V.!

I believe that this is, unfortunately, just the "nature of worker processes". It's not just your inky templates that are affected - if you change any PHP code while developing locally, you need to restart your workers so that your workers see the new code. This is why we gracefully restart workers on deploy. But, I can give you 2 options:

1) In dev, use a "sync://" transport. Then only use your real transport in the prod environment (I can point you how to do this if you need a little pointer).

2) If you use the symfony binary, it has a little-known feature to "watch" for changes and restart your workers for you. It's this:

symfony run -d --watch=config,src,templates,vendor symfony console messenger:consume async

That runs your messenger:consume command in the background as a daemon and restarts it automatically whenever it sees changes to the listed directories.

Let me know if any of this helps ;).

Cheers!

Hey weaverryan,

Thanks for this command, it worked like a charm :)

Cheers !

Hi everybody,

I used this tutorial, and adapt it for a CMS I develop with Symfony 4.4

For this project, I need to be able to send emails to multiple users.
For example, send a reminder for an event to all users.

I'm working with Messenger too, but I was wondering... How can we manage a potential error ?
Let's say that on the 20 emails that has to be send, 2 emails are not send (whatever the reason)...
Is it better to manage this with Mailer or with Messenger ? And how ?

Thanks for your ideas/suggestions because the documentation for this 2 "new" functionality is quite silent about it.

Hi Christina V.!

Excellent question :). So if you're sending a reminder for an event, my guess is that you would do this with some sort of CRON job? For example, you would run a CRON job every 1 minute... or 5 minutes, query the database to find all emails that need to be sent and then send them? If you were thinking of some totally different way, then let me know.

Assuming I'm right, then, as you said, you might find that you need to send 20 emails all at once. If you did not use Messenger and use Mailer directly, then, if something fails, you'll get a normal exception. You could certainly wrap the sending in a "try-catch" and NOT mark it as sent, so that it would be tried again on the next loop. Here is some fake code to illustrate:

$eventRegistrants = [...];
foreach ($eventRegistrants as $eventRegistrant) {
    // ...

    try { 
        $this->mailer->send(...);
        $eventRegistration->setReminderEmailSent(true); // used to know that this was sent!
    } catch (\Exception $e) {
        // do nothing, don't mark the "registrant email" as sent so that
        // we will try to send it again on the next loop
        // maybe log an error :) 
    }
}

This is a fine setup. Like anything, you need to code carefully to avoid problems if suddenly 10,000 emails need to be sent at once, but there are no real problems with this. Also, Mailer itself as a way to specify a "fallback" transport. This allows Mailer to automatically try sending through a 2nd transport if the first transport fails for some reason.

If you decide to use Messenger, then the error handling is pushed into a different spot. First, you would not need the try-catch: you would just do $this->mailer->send() and that's it. In reality, this would "queue" the email to be sent. Then, if there IS a problem delivering the email from your "worker", Messenger's automatic retry system will takeover. By default, Messenger will automatically try to "process" a message 3 times (with some delays in between) before sending it to a "failure queue" so you can inspect what happened and retry.

So in both situations, you can handle your failures gracefully. Messenger would really be more about performance or scalability in this setup: in theory your one CRON job could very quickly queue 1000 new emails to be sent, and then multiple workers could share the load of sending those emails quickly. But, you may not have that problem :).

Let me know if this helps!

Cheers!

Hey @weaverryan.

Sorry for the late reply. With this corona crisis, I'm out of space !

Well, I don't have a CRON (yet) but yes, at the end, this is something that we plan to have.
After you reply, I get deeper in the new Messenger Component, I missed some stuff and now it's all crystal clear!

Thanks for your reply and coments.
Always a pleasure to learn from here.

Cheers

Hello!

I got a strange behavior when enabling the messenger to send email async:

1st case:
-> I sent the mail to the queue
-> Checking email in database, the email looks correct (links / translation)
-> Start the messenger:consume (-vv)

=> I got the error "Unable to generate a URL for the named route "admin_homepage" as such route does not exist." but the route exists

2nd case:
-> I start the messenger:consume (-vv)
-> I sent email to the queue (here I can not check if the email looks correct in database as the consumer is handling the email directly)

=> I reveiced in mailtrap the email but no link are generated and translation is not done. No error in messenger:consume output.

Any idea of what could miss?

Thx!

huh tried again today and it works correctly. Nothing has changed in my config. Could it be a "cache" issue?

Hey Lydie,

Hm, weird... yeah, sounds like a cache issue... Especially if we're talking about prod. In dev mode cache should recompile... but in case you use some virtualization tools and sharing files between host and target machines - it might not always work correctly, though works most of the time. Anyway, in any weird situation it would be always a good idea to try to clear the cache first and see if it helps. If not - dig further :)

I'm happy it works for you! And thanks for mentioning it works now!

Cheers!

Hello!

In fact I was talking about local development (but with db on a remote server). I do not really understand what happened but even clearing manually the cache did not work. I think it worked again after closing phpstorm and stopping local server. If it appears again, I will just try that and see if it helps :)

Hey Lydie,

Hm, interesting. I wonder what platform do you use? Are you on Windows? Probably start with stop/start the webserver... if it does not help, only then try to close/open PhpStorm. Though, it's weird that you have to do these steps.

Cheers!