SymfonyCasts stands united with the people of Ukraine
If you liked what you've learned so far, dive in!
Subscribe to get access to this tutorial plus
video, code and script downloads.
With a Subscription, click any sentence in the script to jump to that part of the video!
Login SubscribeSo... how does all of this work on production? It's a simple problem really: on production, we somehow need to make sure that this command - messenger:consume - is always running. Like, always.
Some hosting platforms - like SymfonyCloud - allow you to do this with some simple configuration. You basically say:
Yo Cloud provider thingy! Please make sure that
bin/console messenger:consumeis always running. If it quits for some reason, start a new one.
If you're not using a hosting platform like that, it's ok - but you will need to do a little bit of work to get that same result. And actually, it's not just that we need a way to make sure that someone starts this command and then it runs forever. We actually don't want the command to run forever. No matter how well you write your PHP code, PHP just isn't meant to be ran forever - eventually your memory footprint will increase too much and the process will die. And... that's perfect! We don't want our process to run forever. Nope: what we really want is for messenger:consume to run, handle... a few messages... then close itself. Then, we'll use a different tool to make sure that each time the process disappears, it gets restarted.
The tool that does that is called supervisor. After you install it, you give it a command that you always want running and it stays up all night constantly eating pizza and watching to make sure that command is running. The moment it stops running, for any reason, it puts down the pizza and it restarts the command.
So let's see how Supervisor works and how we can use it to make sure our worker is always running. Because I'm using a Mac, I already installed Supervisor via Brew. If you're using Ubuntu, you can install it via apt. By the way, you don't actually need to install & configure Supervisor on your local machine: you only need it on production. We're installing it so we can test and make sure everything works.
To get it going, we need a supervisor configuration file. Google for "Messenger Symfony" and open the main documentation. In the middle... there's a spot that talks about supervisor. Copy the configuration file. We could put this anywhere: it doesn't need to live in our project. But, I like to keep it in my repo so I can store it in git. In... how about config/, create a new file called messenger-worker.ini and paste the code inside.
| [program:messenger-consume] | |
| command=php /path/to/your/app/bin/console messenger:consume async --time-limit=3600 | |
| user=ubuntu | |
| numprocs=2 | |
| autostart=true | |
| autorestart=true | |
| process_name=%(program_name)s_%(process_num)02d |
The file tells Supervisor which command to run and other important info like which user it should run the process as and the number of processes to run. This will create two worker processes. The more workers you run, the more messages can be handled at once. But also, the more memory & CPU you'll need.
Now, locally, I don't need to run supervisor... because we can just manually run messenger:consume. But to make sure this all works, we're going to pretend like my computer is production and change the path to point to use my local path: /Users/weaverryan/messenger... which if I double-check in my terminal... oop - I forgot the Sites/ part. Then, down here, I'll change the user to be weaverryan. Again, you would normally set this to your production values.
Oh, and if you look closely at the command, it's running messenger:consume async. Make sure to also consume async_priority_high. The command also has a --time-limit=3600 option. We'll talk more about this and some other options in a bit, but this is great: it tells the worker to run for 60 minutes and then exit, to make sure it doesn't get too old and take up too much memory. As soon as it exits, Supervisor will restart it.
Now that we have our config file, we need to make sure Supervisor can see it. Each Supervisor install has a main configuration file. On a Mac where it's installed via Brew, that file is located at /usr/local/etc/supervisord.ini. On Ubuntu, it should be /etc/supervisor/supervisord.conf.
Then, somewhere in your config file, you'll find an include section with a files line. This means that Supervisor is looking in this directory to find configuration files - like ours - that will tell it what to do.
To get our configuration file into that directory, we can create a symlink: ln -s ~/Sites/messenger/config/messenger-worker.ini then paste the directory.
ln -s ~/Sites/messenger/config/messenger-worker.ini /usr/local/etc/supervisor.d/
Ok! Supervisor should now be able to see our config file. To run supervisor, we'll use something called supervisorctl. Because I'm on a Mac, I also need to pass a -c option and point to the configuration file we were just looking at. If you're on Ubuntu, you shouldn't need to do this - it'll know where to look already. Then say reread: that tells Supervisor to reread the config files:
supervisorctl -c /usr/local/etc/supervisord.ini reread
By the way, you may need to run this command with sudo. If you do, no big deal: it will execute the processes themselves as the user in your config file.
Cool! It sees the new messager-consume group. That names comes from the key at the top of our file. Next, run the update command... which would restart any processes with the new config... if they were already running... but our's aren't yet:
supervisorctl -c /usr/local/etc/supervisord.ini update
To start them, run start messenger-consume:*:
supervisorctl -c /usr/local/etc/supervisord.ini start messenger-consume:*
That last argument - messenger-consume:* isn't very obvious. When you create a "program" called messenger-consume, this creates what's called a "homogeneous process group". Because we have processes=2, this group will run two processes. By saying messenger-consume:* it tells Supervisor to start all processes inside that group.
When we run it... it doesn't say anything... but... our worker commands should now be running! Let's go stop our manual worker so that only the ones from Supervisor are running. Now,
tail -f var/log/messenger.log
This will make it really obvious whether or not our messages are being handled by those workers. Now, upload a few photos, delete a couple of items, move over and... yea! It's working! It's actually working almost twice as fast as normal because we have twice the workers.
And, now we can have some fun. First, we can see the process id's created by Supervisor by running:
ps -A | grep messenger:consume
Tip
You can also use ps aux, which will work on more operating systems.
There they are: 19915 and 19916. Let's kill one of those:
kill 19915
And run that again:
ps -A | grep messenger:consume
Yes! 19916 is still there but because we killed the other one, supervisor started a new process for it: 19995. Supervisor rocks.
Next, let's talk more about the options we can use to purposely make workers exit before they take up too much memory. We'll also talk about how to restart workers on deploy so that they see the new code and a little detail about how things can break if you update your message class.
Hey Tim,
Thank you for sharing your workaround with others! Good idea to start the command via CRON ;)
Cheers!
Hello again !
A tiny precision, wich I faced : on ubuntu server I had to precise the directory where the supervisor should find the app so my worker looks like :
[program:messenger-consume]
command=php bin/console messenger:consume async --time-limit=3600
directory=/var/www/html/my-app
One other thing about supervisor, it seems he prefer now working with .conf files and not .ini (easy to fix, but you have to think about this --> http://supervisord.org/runn...
Are aware of a trouble for the FlySystem to delete files in a handler? I can't do it on production while I'm ok on local dev (still searching btw)
Hey Remi,
Yeah, it makes sense. In our example we use the absolute path so it works. If you use a relative path like only "bin/console" - you have to specify the directory of your project. Thank you for some tips! Might be useful for other users.
About problem with deleting files in a handler. Hm, most probably you have different permissions, so it sounds like permission problem. Is it for all the folders? Or some of them are removed successfully but some are not?
Cheers!
Hm, most probably you have different permissions, so it sounds like permission problem. Is it for all the folders? Or some of them are removed successfully but some are not?
Mmm... Not sure because my filesystem works perfectly in the controller in production... I'll explore further and keep you update (i didn't try it syncronously yet to make a differential diagnosis)
Hey @Rémi Dck
I also believe that the file deletion problem is due to permissions. I think you just have to tell supervisor to run your worker as your web user (usually named as "www"). Here is an example of how to do it but I didn't give it a try: https://stackoverflow.com/a...
Cheers!
OMG you're the best Diego. Thanks a lot !!!
I dug in and the tail is a little bit tricky (to me, i'm not a unix user...)
The first step is finding the web user, like you said and the best command for this is less /etc/passwd. For Ubuntu 18.04 seems to be www-data
Next in messenger-worker.conf change user name and add environnement targeting the home directory of www-data
Aaaand done ! Love it.
Awesome! Excellent job man because I wrote the name wrong, it usually is www-data not just www :p
Dang
| posted 2 months ago
Hello guys,
I'm using Docker for deployment and my image have supervisor installed within. Now in order to execute these three command :
supervisorctl reread
supervisorctl update
supervisorctl start messenger-consume:*
Where should I put them? In Dockerfile or a entrypoint command? They should be run after the container is up if I'm not wrong ?
Yo @Dang!
Hmm. I am FAR from an expert on Docker, especially in a production environment. However, I believe that you would have a specific (PHP) container set up for running supervisor and that supervisor would be your main CMD on that container (and you'd run supervisor with nodaemon=true). Again, this is NOT my area of expertise, so I could be misguided, but that's my impression :).
Cheers!
Hi. What is/are the possible strategy/strategies to use if the Consumer throws an exception for any reason ? Assuming that the environment is Production. Thanks.
Hey Marko,
The best strategy is to do a few more retries after some time, and if still no success - move that message into a failed messages queue where you can retry the message manually when the problem will be fixed. That's the strategy we use on SymfonyCasts, and IIRC we covered it in this course :)
Cheers!
Hello Victor! Thanks a lot for your quick answer. Does Supervisor have any role in this issue (in production) ?
Hey Marko,
Supervisor is about a different problem, it helps to restart the worker on the server, so it's a bit different. If for some reasons you mean that your worker crashes, i.e. stops on the production that you have to restart it manually again - then yes, you need that supervisor that will restart the worker for you. But the problems you described relates to different things if I understand you correctly.
Cheers!
Rainer S.
Yeah that's a good question, I'm hot sure but probably this link https://github.com/alexsilv... will help you, but I think you already saw this link. Unfortunately I can't advise you some real production example for windows, also there is not so many windows servers, that's why there is not so many solutions for it!
Cheers! Hope it will help!
Has anyone by any chance run supervisor on an AWS EB setup? (Elastic Beanstalk, using .ebextension files)
Hey Chad,
Unfortunately, I have not run it personally. Though it looks like there's some solutions around the internet, e.g.: https://gist.github.com/vra...
I hope this helps!
Cheers!
Hey Chad,
Ah, great! Sorry, can't say something more about it, have never done it before. Probably someone who use it could suggest you a good tip about it.
Cheers!
the following .ebextension config file is working to install supervisor and start messenger:consume, but the .env file is required to be populated with all the database credentials and such
files:
"/tmp/messenger-consume.conf":
mode: "000644"
owner: root
group: root
content: |
[program:messenger-consume]
command=php /var/www/html/bin/console messenger:consume async --time-limit=1800 --memory-limit=340M --sleep=30
user=webapp
numprocs=1
autostart=true
autorestart=true
process_name=%(program_name)s_%(process_num)02d
"/tmp/supervisord.conf":
mode: "000644"
owner: root
group: root
content: |
; supervisor config file
[unix_http_server]
file=/var/run/supervisor.sock ; (the path to the socket file)
chmod=0700 ; sockef file mode (default 0700)
[supervisord]
user=root
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir=/var/log/supervisor ; ('AUTO' child log dir, default $TEMP)
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL for a unix socket
; The [include] section can just contain the "files" setting. This
; setting can list multiple files (separated by whitespace or
; newlines). It can also contain wildcards. The filenames are
; interpreted as relative to this file. Included files *cannot*
; include files themselves.
[include]
files = /etc/supervisor/conf.d/*.conf
; Change according to your configurations
commands:
01_01_install_supervisord:
command: |
sudo easy_install supervisor
sudo mkdir -p /etc/supervisor/conf.d/
sudo mkdir -p /var/log/supervisor
sudo touch /var/log/supervisor/supervisord.log
ignoreErrors: true
container_commands:
01_01_copy_supervisor_configuration_files:
command: |
sudo mv /tmp/supervisord.conf /etc/supervisord.conf
sudo mv /tmp/messenger-consume.conf /etc/supervisor/conf.d/messenger-consume.conf
ignoreErrors: true
02_01_start_supervisord:
command: |
sudo /usr/local/bin/supervisord -c /etc/supervisord.conf
sudo /usr/local/bin/supervisorctl reread
sudo /usr/local/bin/supervisorctl update
sudo /usr/local/bin/supervisorctl restart all
sudo /usr/local/bin/supervisorctl start messenger-consume:*
ignoreErrors: true```
Hey Chad,
Thank you for sharing this solution with others! I'm glad you were able to get it working.
Cheers!
Hello,
When running supervisorctl -c /usr/local/etc/supervisord.ini start messenger-consume:*, I get zsh: no matches found: messenger-consume:*. But it seems the workers are actually working, because I tried to upload an image and it worked. In the video, there's no message after you start the supervisor with messenger-consume:*. So I'm just wondering if I'm doing something wrong...
Thanks!
Hey Ajie62!
Hmm. Try this command to see what's going on:
supervisorctl -c /usr/local/etc/supervisord.ini start
I definitely don't like that "no matches found". Also try using the stop command to see if it stops anything. Here's a screenshot of what starting and stopping looks on my machine - you can see how "start" sometimes doesn't have any output (if it's already started) but sometimes does.
Also, the zsh: no matches found: messenger-consume:* is odd... the error should come from supervisorctl... not from "zsh". Try surrounding messenger-consume:* with quotes. I don't know zsh well... but I'm wondering if it's somehow interpreting something differently...
Cheers!
Hey, thanks for you answer! The problem was oh-my-zsh. I removed it an now everything works fine. I assume I could have solved the problem and keep oh-my-zsh, but I wanted to save time and keep learning :) Thank you again, weaverryan
// composer.json
{
"require": {
"php": "^7.1.3",
"ext-ctype": "*",
"ext-iconv": "*",
"composer/package-versions-deprecated": "^1.11", // 1.11.99
"doctrine/annotations": "^1.0", // v1.8.0
"doctrine/doctrine-bundle": "^1.6.10", // 1.11.2
"doctrine/doctrine-migrations-bundle": "^1.3|^2.0", // v2.0.0
"doctrine/orm": "^2.5.11", // v2.6.3
"intervention/image": "^2.4", // 2.4.2
"league/flysystem-bundle": "^1.0", // 1.1.0
"phpdocumentor/reflection-docblock": "^3.0|^4.0", // 4.3.1
"sensio/framework-extra-bundle": "^5.3", // v5.3.1
"symfony/console": "4.3.*", // v4.3.2
"symfony/dotenv": "4.3.*", // v4.3.2
"symfony/flex": "^1.9", // v1.18.7
"symfony/framework-bundle": "4.3.*", // v4.3.2
"symfony/messenger": "4.3.*", // v4.3.4
"symfony/property-access": "4.3.*", // v4.3.2
"symfony/property-info": "4.3.*", // v4.3.2
"symfony/serializer": "4.3.*", // v4.3.2
"symfony/validator": "4.3.*", // v4.3.2
"symfony/webpack-encore-bundle": "^1.5", // v1.6.2
"symfony/yaml": "4.3.*" // v4.3.2
},
"require-dev": {
"easycorp/easy-log-handler": "^1.0.7", // v1.0.7
"symfony/debug-bundle": "4.3.*", // v4.3.2
"symfony/maker-bundle": "^1.0", // v1.12.0
"symfony/monolog-bundle": "^3.0", // v3.4.0
"symfony/stopwatch": "4.3.*", // v4.3.2
"symfony/twig-bundle": "4.3.*", // v4.3.2
"symfony/var-dumper": "4.3.*", // v4.3.2
"symfony/web-profiler-bundle": "4.3.*" // v4.3.2
}
}
In case anybody has the same problem, that the installation of SUPERVISOR is not allowed on production-server, I solved this with a workaround by using a cronjob (starting every 5 minutes) with this bash-script:
Happy coding!