SymfonyCasts stands united with the people of Ukraine
If you liked what you've learned so far, dive in!
Subscribe to get access to this tutorial plus
video, code and script downloads.
With a Subscription, click any sentence in the script to jump to that part of the video!
Login SubscribeCheck this out: I'm going to turn off my Wifi! Gasp! What do you think will happen? I mean, other than I'm gonna miss all my Tweets and Instagrams! What will happen when I refresh? The page will load, but all the images will be broken, right?
In the name of science, I command us to try it!
Woh! An error!?
Error executing ListObjects on https://sf-casts-spacebar ... Could not contact DNS servers.
What? Why is our Symfony app trying to connect to S3?
Here's the deal: on every request... for every thumbnail image that will be rendered, our Symfony app makes an API request to S3 to figure out if the image has already been thumbnailed or if it still needs to be. Specifically, LiipImagineBundle is doing this.
This bundle has two key concepts: the resolver and the loader. But there are actually three things that happen behind the scenes. First, every single time that we use |imagine_filter(), the resolver takes in that path and has to ask:
Has this image already been thumbnailed?
And if you think about it, the only way for the resolver to figure this out is by making an API request to S3 to ask:
Yo S3! Does this thumbnail file already exist?
If it does exist, LiipImagineBundle renders a URL that points directly to that image on S3. If not, it renders a URL to the Symfony route and controller that will use the loader to download the file and the resolver to save it back to S3.
Phew! The point is: on page load, our app is making one request to S3 per thumbnail file that the page renders. Those network requests are super wasteful!
What's the solution? Cache it! Go back to OneupFlysystemBundle and find the main page of their docs. Oh! Apparently I need Wifi for that! There we go. Go back to their docs homepage and search for "cache". You'll eventually find a link about "Caching your filesystem".
This is a super neat feature of Flysystem where you can say:
Hey Flysystem! When you check some file metadata, like whether or not a file exists, cache that so that we don't need to ask S3 every time!
Actually, it's even more interesting & useful. LiipImagineBundle calls the exists() method on the Filesystem object to see if the thumbnail file already exists. If that returns false, the cached filesystem does not cache that. But if it returns true, it does cache it. The result is this: the first time LiipImagineBundle asks if a thumbnail image exists, Flysystem will return false, and Liip will know to generate it. The second time it asks, because the "false" value wasn't cached, Flysystem will still talk to S3, which will now say:
Yea! That file does exist.
And because the cached adapter does cache this, the third time LiipImagineBundle calls exists, Flysystem will immediately return true without talking to S3.
Tip
If you're using version 4 of oneup/flysystem-bundle (so, flysystem v2),
the league/flysystem-cached-adapter will not work - it was not updated to support
flysystem v2. Someone has created a cached adapter - https://github.com/Lustmored/flysystem-v2-simple-cache-adapter -
but configuring it requires extra steps.
To get this rocking, copy the composer require line, find your terminal and paste to download this "cached" Flysystem adapter.
composer require league/flysystem-cached-adapter
While we're waiting, go check out the docs. Here's the "gist" of how this works, it's 3 parts. First, you have some existing filesystem - like my_filesystem. Second, via this cache key, you register a new "cached" adapter and tell it how you want things to be cached. And third, you tell your existing filesystem to process its logic through that cached adapter. If that doesn't totally make sense yet, no worries.
For how you want the cached adapter to cache things, there are a bunch of options. We're going to use the one called PSR6. You may or may not already know that Symfony has a wonderful cache system built right into it. Anytime you need to cache anything, you can just use it!
Start by going to config/packages/cache.yaml. This is where you can configure anything related to Symfony's cache system, and we talked a bit about it in our Symfony Fundamentals course. The app key determines how the cache.app service caches things, which is a general-purpose cache service you can use for anything, including this! Or, to be fancier - I like being fancy - you can create a cache "pool" based on this.
Check it out. Uncomment pools and create a new cache pool below this called cache.flysystem.psr6. The name can be anything. Below, set adapter to cache.app.
| framework: | |
| cache: | |
| ... lines 3 - 17 | |
| pools: | |
| cache.flysystem.psr6: | |
| adapter: cache.app |
That's it! This creates a new cache service called cache.flysystem.psr6 that, really... just uses cache.app behind the scenes to cache everything. The advantage is that this new service will automatically use a cache "namespace" so that its keys won't collide with other keys from other parts of your app that also use cache.app.
In your terminal, run:
php bin/console debug:container psr6
There it is! A new fancy cache.flysystem.psr6 service.
Back in oneup_flysystem.yaml, let's use this! On top... though it doesn't matter where, add cache: and put one new cached adapter below it: psr6_app_cache. The name here also doesn't matter - but we'll reference it in a minute.
| ... line 1 | |
| oneup_flysystem: | |
| cache: | |
| psr6_app_cache: | |
| ... lines 5 - 21 |
And below that add psr6:. That exact key is the important part: it tells the bundle that we're going to pass it a PSR6-style caching object that the adapter should use internally. Finally, set service to what we created in cache.yaml: cache.flysystem.psr6.
| ... line 1 | |
| oneup_flysystem: | |
| cache: | |
| psr6_app_cache: | |
| psr6: | |
| service: cache.flysystem.psr6 | |
| ... lines 7 - 21 |
At this point, we have a new Flysystem cache adapter... but nobody is using it. To fix that, duplicate uploads_filesystem and create a second one called cached_uploads_filesystem. Make it use the same adapter as before, but with an extra key: cache: set to the adapter name we used above: psr6_app_cache.
| ... line 1 | |
| oneup_flysystem: | |
| ... lines 3 - 13 | |
| filesystems: | |
| ... lines 15 - 17 | |
| cached_uploads_filesystem: | |
| adapter: uploads_adapter | |
| cache: psr6_app_cache |
Thanks to this, all Filesystem calls will first go through the cached adapter. If something is cached, it will return it immediately. Everything else will get forwarded to the S3 adapter and work like normal. This is classic object decoration.
After all of this work, we should have one new service in the container. Run:
php bin/console debug:container cached_uploads
There it is: oneup_flysystem.cached_uploads_filesystem_filesystem. Finally, go back to liip_imagine.yaml. For the loader, we don't really need caching: this downloads the source file, which should only happen one time anyways. Let's leave it.
But for the resolver, we do want to cache this. Add the cached_ to the service id. The resolver is responsible for checking if the thumbnail file exists - something we do want to cache - and for saving the cached file. But, "save" operations are never cached - so it won't affect that.
| liip_imagine: | |
| ... lines 2 - 13 | |
| resolvers: | |
| flysystem_resolver: | |
| flysystem: | |
| # use the cached version so we're not checking to | |
| # see if the thumbnailed file lives on S3 on every request | |
| filesystem_service: oneup_flysystem.cached_uploads_filesystem_filesystem | |
| ... lines 20 - 69 |
Let's try this! Refresh the page. Ok, everything seems to work fine. Now, check your tweets, like some Instagram photos, then turn off your Wifi again. Moment of truth: do a force refresh to fully make sure we're reloading. Awesome! Yea, the page looks terrible - a bunch of things fail. But our server did not fail: we are no longer talking to S3 on every request. Big win.
Next, let's use a super cool feature of S3 - signed URLs - to see an alternate way of allowing users to download private files, which, for large stuff, is more performant.
Hey Fox C.!
I appreciate the comment :). So just to make sure that we both understand each other, with the code in this chapter, the end result is that your images ARE stored on S3. And once an image has been stored on S3, there is no network request made to S3 in order to load the page. In essence, we are using AWS S3 as a "CDN" and the server is not responsible for delivering the messages (the server generates a link directly to S3, assuming the image is already created).
However (and this may be exactly what you're saying), it IS true that the thumbnails are not created until someone tries to view that thumbnail for the first time. If every thumbnail on my site has already been generated and is sitting on S3, that's fine! When I go a page, everything loads fast as my server is just rendering links that point directly at S3. But if you loaded a page that had 10 images that had NEVER been thumbnailed, then 10 requests will be fired back to the server and the server will work on creating those 10 thumbnails and sending them to S3. The original page would load fine, but the images would load slowly (and you would have 10 requests blocking other traffic while their thumbnails are being created). So this could *definitely* be a real-world problem - I agree. The pre-generation you talked about would be the solution here :). This is a fair criticism - and I think it would have been good to mention this. You could, at the moment you upload, just use the Liip services directly to "force" the thumbnail generation: https://github.com/liip/Lii... . It's more work, but it solves this problem.
Cheers!
Oh... It's caching *whether file exists* not caching a file on the symfony server and then serving that local file to the user. Great, thanks for that!
Hi Ryan,
After applying the changes in this video, I get the following error:
Attempted to load class "Psr6Cache" from namespace "League\Flysystem\Cached\Storage".
Did you forget a "use" statement for e.g.
"Symfony\Component\Validator\Mapping\Cache\Psr6Cache" or
"Symfony\Component\Cache\Simple\Psr6Cache"?
Any idea why?
Thanks for your help
Hey Diana,
Please, make sure you have this package installed: league/flysystem-cached-adapter . If not - do "composer require league/flysystem-cached-adapter". That namespace you're referencing is coming from that package.
I hope this helps.
Cheers!
Hey,
I have successfully used the S3 AWS service to store my images.
Wishing to set up a cache solution as advised by Ryan, I have a problem installing the bundle flysystem-cached-adapter which is not compatible with league / flysystem-bundle ":" ^ 2.0 ".
Do you know a solution to work around this problem?
Hey Stephane!
Ah, you're right! That package doesn't work with Flysystem 2 - what a shame! I will have to dig in deeper to see what a better solution is. On a high level, one option would be to create your own custom Flysystem adapter class - https://flysystem.thephplea...
Inside this, you would "decorate" a Flysystem adapter & inject CacheInterface. In the appropriate methods, you would check the cache before calling the internal adapter. You would then register your new Flysystem adapter as a service and use *it*.
That is... a lot of "quick talk" for something that will take some real code. Let me know if it makes sense. As I mentioned, I will probably need to dive in and see if I can code up a solution - but if you want to try it before then, I'd be happy to help or answer any questions along the way :).
Cheers!
Hello! Being in the same situation, I tried to apply your advice, but it is ... way too complicated for me. I can't seem to put this in place. Do you have a resource please?
EDIT : I found this resource for V2 : https://github.com/Lustmore... .What do you think ?
EDIT 2 : If there is no solution, could I save the URLs of the images coming from S3 in BDD? This would allow much fewer API calls to be made. And if the entity does not yet have a URL to the image from AWS S3, then I get it and put it back in DB for later. Good idea ?
Hey Kiuega!
> EDIT : I found this resource for V2 : https://github.com/Lustmore... .What do you think ?
At a quick glance, that looks like exactly what's needed!
> EDIT 2 : If there is no solution, could I save the URLs of the images coming from S3 in BDD? This would allow much fewer API calls to be made. And if the entity does not yet have a URL to the image from AWS S3, then I get it and put it back in DB for later. Good idea ?
Yup! That's a very cool idea. You do run the risk that, somehow, that flag in the database gets "out of date" with reality (somehow, the database says that there IS a thumbnail, so you use it, but it's not actually there). However, that's probably a minor thing.
Cheers!
Hello @weaverryan :)
>At a quick glance, that looks like exactly what's needed!
In fact I just noticed that in your video, without cache, if you cut your internet connection, then you have an error and can no longer access the page. At home, even without cache, if I cut my internet connection, I have no error (but the images are not displayed. In the end I have the same visual result as you when you put your system on. cover in place). Does this mean that there is already a built-in cache since the new version?
If not, then I first tried to set up the bundle that I presented to you above.
First, it seems to require "psr / cache": "^ 1.0" (while we are at v2), which requires me to downgrad my version.
Second, I don't understand how to set it up afterwards? I stupidly followed your video, but I feel like I'm on the wrong track.
Especially since since the last versions, probably, we can no longer, in the configuration file of oneup_flysystem.yaml, have under oneup_flysystem.filesystems.cached_uploads_filesystem the 'cache' key as you have in the video. Which causes this error:
"Unrecognized option" cache "under" oneup_flysystem.filesystems.cached_uploads_filesystem ". Available options are" adapter "," alias "," mount "," visibility "."
>Yup! That's a very cool idea. You do run the risk that, somehow, that flag in the database gets "out of date" with reality (somehow, the database says that there IS a thumbnail, so you use it, but it's not actually there). However, that's probably a minor thing.
To implement this solution, does this mean that I should override LiipImagineBundle events in order to do the processing described in my previous answer, rather than letting LiipImagine make the call?
Thanks ! :)
Hey Kiuega!
> At home, even without cache, if I cut my internet connection, I have no error (but the images are not displayed. In the end I have the same visual result as you when you put your system on. cover in place). Does this mean that there is already a built-in cache since the new version?
That's a good question! If there were some cache involved, it would be, I think, inside LiipImagineBundle. But I'm not aware of anything. It seems more likely that something (LiipImagineBundle or perhaps the OneupFlysystemBundle) is now simply failing gracefully in the background. But that is a total guess.
> Second, I don't understand how to set it up afterwards
You're right that it is, indeed, more complex. The previous version of OneupFlysystemBundle had built-in support for the cache adapter. This allowed you to install it and, sort of, "drop it in" - via that "cache" option. But that is gone in the latest version, since the cached adapter wasn't ported to Flysystem v2.
I see that you opened an issue about this https://github.com/Lustmore.... As the author states, it's now your job to register the cache adapter service manually... but it involves quite a lot of wiring. I've just replied there with a possible solution.
Cheers!
Hello @weaverryan !
Thank you very much for your answer, it helped me a lot! :)
I was able to set it up.
On the other hand when you say :
# config/packages/flysystem.yaml
flysystem:
storages:
# ... your other storages
cached_public_uploads_storage:
# point this at your service id from above
adapter: 'flysystem.cache.adapter'
<blockquote>># point this at your service id from above</blockquote>
is it just a comment pointing out what you are doing, or an instruction asking me to do something more?
Finally, is there a way to verify that the cache is really applied? Since if I cut the internet connection without cache, I don't get an error message.
Thanks!
Hey Kiuega !
> is it just a comment pointing out what you are doing, or an instruction asking me to do something more
Yea, sorry - this is just "pointing out what I am doing" - not an instruction to do something extra.
> Finally, is there a way to verify that the cache is really applied? Since if I cut the internet connection without cache, I don't get an error message.
Excellent question! Yes, the web debug toolbar can tell you this. Refresh a page (that has thumbnails on it), then click any button on the web debug toolbar to open the profiler for that page. Then click on the left to the "Cache" section. Here, you will be see all the cache activity during that request, which is filterable by the cache pool (so, flysystem.cache.adapter) in your case.
Another way to see this, if you want to triple check it on production, is to use a tool like Blackfire and check to see if your request contains any network calls to S3.
Let me know if you've got it hooked up and working!
Cheers!
Hello @weaverryan !
Okay I checked it all out!
So firstly, if I don't use the cache system, I have (as expected) nothing in the "Cache" section of the profiler.
And if I use the cache system that we just set up, then I have the "Cache" section of the profiler which is filled with what seems to be the right things!
Small precision, the name is 'cache.flysystem.psr6' and not 'flysystem.cache.adapter', but the result seems to be the same! The cache works perfectly: Yes!
Thank you so much !
I will just have to open an issue on github so that it updates the use of "psr / cache" to version 2 because it is currently version 1 that is used (and therefore when we install its bundle , we have to downgrad), but at least it works!
Thank you !
Hey Kiuega!
Woohoo! I'm so happy it's working - nice job :). This should help many other people who have been running into this.
Cheers!
Maybe this adapter is now unneeded on the V2? I've seen no movement on https://github.com/thephple... in the past months. Must be a conscious decision.
Edit: Well I migrated to Flysystem 2 to try it, had to remove the cached adapter, and the performance is now appalling. I understand that without the cached adapter it makes two calls, but I guess we could also solve that by saving a reference to the external S3 URL in the picturable entities instead of asking for it each time we need it (which was the goal of the cached adapter somehow, in a different way).
Hey weaverryan
Thank a lot for your message. I will study your suggestion about creation of custom Flysystem adapter class.
Cheers.
these * .yaml configs for services we configured looks to me too complicated, cause I can't find a kinda common pattern and where to use/provide the properties we made. I hope someday it will change
Hey Bagar,
Do you mean you don't know where to write new configuration? You can always use "Find in path" PhpStorm feature, and e.g. search for "liip_imagine:" key to find the proper file with the configuration you needed. Or another good trick, it works for me when I press "Shift" in PgpStorm two times - it opens a search by file names, and I can write "liip" for example and it will suggest me all the files that contain "liip" text in its filename.
Also, Symfony has a few commands that help you working with configs:
$ bin/console config:dump-reference
It dumps the *default* configuration of your project. Or add "liip_imagine" as an arg to this command to see all the configuration for that bundle. Another one:
$ bin/console debug:config
This dumps the *current* configuration of the project.
I hope this helps!
Cheers!
Thank you, Viktor!
The tricks you've described are super handy, but I think that my problem is in abstractions interaction and how to describe them in *.yaml.
Anyway thanks
Hey Bagar,
Difficult to understand what you mean exactly, if you have some simple examples - I could try to give you more advices. Anyway, I suppose you can always changed to PHP or XML config. XML configuration is pretty popular for OSS bundles.
Cheers!
When I ran "php bin/console debug:container cached_uploads", got an error saying "No services found that match "cached_uploads"."
// composer.json
{
"require": {
"php": "^7.1.3",
"ext-iconv": "*",
"aws/aws-sdk-php": "^3.87", // 3.87.10
"composer/package-versions-deprecated": "^1.11", // 1.11.99
"knplabs/knp-markdown-bundle": "^1.7", // 1.7.1
"knplabs/knp-paginator-bundle": "^2.7", // v2.8.0
"knplabs/knp-time-bundle": "^1.8", // 1.9.0
"league/flysystem-aws-s3-v3": "^1.0", // 1.0.22
"league/flysystem-cached-adapter": "^1.0", // 1.0.9
"liip/imagine-bundle": "^2.1", // 2.1.0
"nexylan/slack-bundle": "^2.0,<2.2.0", // v2.1.0
"oneup/flysystem-bundle": "^3.0", // 3.0.3
"php-http/guzzle6-adapter": "^1.1", // v1.1.1
"sensio/framework-extra-bundle": "^5.1", // v5.2.4
"stof/doctrine-extensions-bundle": "^1.3", // v1.3.0
"symfony/asset": "^4.0", // v4.2.3
"symfony/console": "^4.0", // v4.2.3
"symfony/flex": "^1.9", // v1.17.6
"symfony/form": "^4.0", // v4.2.3
"symfony/framework-bundle": "^4.0", // v4.2.3
"symfony/orm-pack": "^1.0", // v1.0.6
"symfony/security-bundle": "^4.0", // v4.2.3
"symfony/serializer-pack": "^1.0", // v1.0.2
"symfony/twig-bundle": "^4.0", // v4.2.3
"symfony/validator": "^4.0", // v4.2.3
"symfony/web-server-bundle": "^4.0", // v4.2.3
"symfony/yaml": "^4.0", // v4.2.3
"twig/extensions": "^1.5" // v1.5.4
},
"require-dev": {
"doctrine/doctrine-fixtures-bundle": "^3.0", // 3.1.0
"easycorp/easy-log-handler": "^1.0.2", // v1.0.7
"fzaninotto/faker": "^1.7", // v1.8.0
"symfony/debug-bundle": "^3.3|^4.0", // v4.2.3
"symfony/dotenv": "^4.0", // v4.2.3
"symfony/maker-bundle": "^1.0", // v1.11.3
"symfony/monolog-bundle": "^3.0", // v3.3.1
"symfony/phpunit-bridge": "^3.3|^4.0", // v4.2.3
"symfony/profiler-pack": "^1.0", // v1.0.4
"symfony/var-dumper": "^3.3|^4.0" // v4.2.3
}
}
This approach may be suitable for generating thumbnails for a small volume of frequently accessed images (although the initial viewer delay is still an issue), but it's not optimal for a large volume of infrequently accessed images (such as a personal library of images for each user - this might be a print-lab website, photography app etc) as it would produce considerable delay and server load. Particularly if the images are hosted on emphemeral servers (e.g. heroku) or across servers behind a load balancer (that would each be produce their own cache).
A more optimal approach (to reduce initial load delay and reduce symfony server load) would be to pre-generate the thumbnails and store them on S3 - using AWS S3 as the "CDN". This increases first load speed and reduces server load by not being responsible for delivering images.
Maybe this isn't your target audience - however I feel this architectural analysis in the videos (while there's some already) would benefit the training outcomes of students - as the "where and why" of an approach is often just as important as the "how to".