Flag of Ukraine
SymfonyCasts stands united with the people of Ukraine
TRACK

Symfony 5 >

COURSE

All about Uploading Files in Symfony

Buy Access to Course

S3 & Private Object via ACLs

Keep on Learning!

If you liked what you've learned so far, dive in!
Subscribe to get access to this tutorial plus
video, code and script downloads.

Start your All-Access Pass
Buy just this tutorial for $10.00
Previous Chapter

Next Chapter
Edit on GitHub

With a Subscription, click any sentence in the script to jump to that part of the video!

Login Subscribe

Head to /admin/article and log back in since we cleared our database recently: admin1@thespacebar.com, password engage. Edit any of the articles. Everything should work just fine: I'll select a few references to upload and... it works nicely. It is a bit slower now that the server is sending the files to S3 in the background, though that should be less noticeable once we're on production, especially if our server is also hosted on AWS.

So... can we download these? Try it! Yea, it works great! Open up ArticleReferenceAdminController and search for "download". Here it is: the download is handled by downloadArticleReference: we open a file stream from Flysystem - which is now from S3 - and stream that back to the user. By planning ahead and using Flysystem, when we switched to S3, nothing had to change!

But, there is one tiny problem. Back on the page, click the image. Access denied!? This should show us the full-size, original image. Hmm, the URL looks right. And, indeed! The problem isn't the path, the problem is with that file's permissions on S3.

Each file, or "object" on S3 can be set to be publicly accessible or private. File are private by default. In fact, the only reason that we can see the thumbnails, which are also stored in S3... is that LiipImagineBundle is smart enough to make sure that when it saves the files to S3, it saves them as public.

When an author uploads an article image, we need to do the same thing: we do want the original images to be public.

Giving the Images Public ACL

Head over to UploaderHelper and find uploadFile(). So far, we've been using the $isPublic argument to choose between the public and private filesystem objects. But when we changed to S3, I temporarily made these two filesystems identical. That wasn't on accident: with S3, we don't need two filesystems anymore! We can use the same one for both public and private files, and control the visibility on a file-by-file basis.

Check it out: remove the $filesystem = part and always use $this->filesystem.

131 lines src/Service/UploaderHelper.php
... lines 1 - 13
class UploaderHelper
{
... lines 16 - 108
$newFilename = Urlizer::urlize(pathinfo($originalFilename, PATHINFO_FILENAME)).'-'.uniqid().'.'.$file->guessExtension();
$stream = fopen($file->getPathname(), 'r');
$result = $this->filesystem->writeStream(
... lines 113 - 117
);
... lines 119 - 129
}

To tell Flysystem that a file should be public or private, add a third argument to writeStream(): an array of options. The option we want is visibility. If $isPublic is true, use AdapterInterface - the one from Flysystem - ::VISIBILITY_PUBLIC. Otherwise, AdapterInterface::VISIBILITY_PRIVATE.

131 lines src/Service/UploaderHelper.php
... lines 1 - 5
use League\Flysystem\AdapterInterface;
... lines 7 - 13
class UploaderHelper
{
... lines 16 - 108
$newFilename = Urlizer::urlize(pathinfo($originalFilename, PATHINFO_FILENAME)).'-'.uniqid().'.'.$file->guessExtension();
$stream = fopen($file->getPathname(), 'r');
$result = $this->filesystem->writeStream(
$directory.'/'.$newFilename,
$stream,
[
'visibility' => $isPublic ? AdapterInterface::VISIBILITY_PUBLIC : AdapterInterface::VISIBILITY_PRIVATE
]
);
... lines 119 - 129
}

Cool, right? That won't instantly change the permissions on the files we've already uploaded. So let's go upload a new one. Close the tab, select a new file, how about rocket.jpg and... update! The thumbnail still works and if you click it, yes! The original file is public!

By the way, you can see this setting when you're looking at the individual files in S3. Click back to the root of the bucket, find the rocket.jpg file and click it. Under "Permissions", here we go. My account has all permissions, of course, and under "Public Access", Everyone has "Read object" access.

Remove that Extra Private Filesystem!

Hey! This is awesome! Thanks to the object-by-object permissions super-power of S3, we don't need an extra "private" filesystem at all! We can do some serious cleanup! Start in config/packages/oneup_flysystem.yaml: remove the private_uploads_adapter and filesystem.

12 lines config/packages/oneup_flysystem.yaml
# Read the documentation: https://github.com/1up-lab/OneupFlysystemBundle/tree/master/Resources/doc/index.md
oneup_flysystem:
adapters:
public_uploads_adapter:
awss3v3:
client: Aws\S3\S3Client
bucket: '%env(AWS_S3_BUCKET_NAME)%'
filesystems:
public_uploads_filesystem:
adapter: public_uploads_adapter

Next, in services.yaml, because there's no private_upload_filesystem anymore, remove that bind.

60 lines config/services.yaml
... lines 1 - 10
services:
... line 12
_defaults:
... lines 14 - 20
bind:
$markdownLogger: '@monolog.logger.markdown'
$isDebug: '%kernel.debug%'
$publicUploadsFilesystem: '@oneup_flysystem.public_uploads_filesystem_filesystem'
$uploadedAssetsBaseUrl: '%uploads_base_url%'
... lines 26 - 60

That will break UploaderHelper because we're using that bind on top. But... we don't need it anymore! Remove the $privateFilesystem property and the $privateUploadFilesystem argument.

129 lines src/Service/UploaderHelper.php
... lines 1 - 13
class UploaderHelper
{
... lines 16 - 18
private $filesystem;
private $requestStackContext;
... lines 23 - 27
public function __construct(FilesystemInterface $publicUploadsFilesystem, RequestStackContext $requestStackContext, LoggerInterface $logger, string $uploadedAssetsBaseUrl)
{
$this->filesystem = $publicUploadsFilesystem;
$this->requestStackContext = $requestStackContext;
$this->logger = $logger;
$this->publicAssetBaseUrl = $uploadedAssetsBaseUrl;
}
... lines 35 - 127
}

But, we're still using that property in two places... the first is down in readStream. Now that everything is stored in one filesystem, delete that old code, remove the unused argument and always use $this->filesystem. Reading a stream is the same for public and private files.

125 lines src/Service/UploaderHelper.php
... lines 1 - 13
class UploaderHelper
{
... lines 16 - 75
public function readStream(string $path)
{
$resource = $this->filesystem->readStream($path);
... lines 79 - 84
}
... lines 86 - 123
}

Repeat that in deleteFile(): delete the extra logic & argument, and use $this->filesystem always.

125 lines src/Service/UploaderHelper.php
... lines 1 - 13
class UploaderHelper
{
... lines 16 - 86
public function deleteFile(string $path)
{
$result = $this->filesystem->delete($path);
... lines 90 - 93
}
... lines 95 - 123
}

Let's see... these two methods are called from ArticleReferenceAdminController. Take off that second argument for readStream().

200 lines src/Controller/ArticleReferenceAdminController.php
... lines 1 - 20
class ArticleReferenceAdminController extends BaseController
{
... lines 23 - 126
public function downloadArticleReference(ArticleReference $reference, UploaderHelper $uploaderHelper)
{
... lines 129 - 131
$response = new StreamedResponse(function() use ($reference, $uploaderHelper) {
$outputStream = fopen('php://output', 'wb');
$fileStream = $uploaderHelper->readStream($reference->getFilePath());
... lines 135 - 136
});
... lines 138 - 145
}
... lines 147 - 198
}

Then, search for "delete", and remove the second argument from deleteFile() as well.

200 lines src/Controller/ArticleReferenceAdminController.php
... lines 1 - 20
class ArticleReferenceAdminController extends BaseController
{
... lines 23 - 150
public function deleteArticleReference(ArticleReference $reference, UploaderHelper $uploaderHelper, EntityManagerInterface $entityManager)
{
... lines 153 - 158
$uploaderHelper->deleteFile($reference->getFilePath());
... lines 160 - 161
}
... lines 163 - 198
}

That felt great! There's one more piece of cleanup we can do, it's optional, but nice. Using the word "public" in the adapter and filesystem isn't accurate anymore! Let's use uploads_adapter and uploads_filesystem.

12 lines config/packages/oneup_flysystem.yaml
... line 1
oneup_flysystem:
adapters:
uploads_adapter:
... lines 5 - 8
filesystems:
uploads_filesystem:
adapter: uploads_adapter

We reference this in a few spots. In liip_imagine.yaml, take out the public_ in these two spots.

67 lines config/packages/liip_imagine.yaml
liip_imagine:
... lines 2 - 5
loaders:
flysystem_loader:
flysystem:
filesystem_service: oneup_flysystem.uploads_filesystem_filesystem
... lines 10 - 13
resolvers:
flysystem_resolver:
flysystem:
filesystem_service: oneup_flysystem.uploads_filesystem_filesystem
... lines 18 - 67

And in services.yaml, update the "bind" in the same way. Hmm, and I think I'll change the argument name it's binding to: just $uploadFilesystem.

60 lines config/services.yaml
... lines 1 - 10
services:
... line 12
_defaults:
... lines 14 - 20
bind:
... lines 22 - 23
$uploadsFilesystem: '@oneup_flysystem.uploads_filesystem_filesystem'
... lines 25 - 60

That will break UploaderHelper: we need to rename the argument there. But, let's just see what happens if we... "forget" to do that. Refresh the page:

Unused binding $uploadFilesystem in S3Client.

This is that generic... and somewhat "inaccurate" error that says that we've configured a bind that's never used! The error is even better if we temporarily delete the bind entirely. Ah, here it is:

Cannot autowire UploaderHelper: argument $publicUploadFilesystem references an interface, but that interface cannot be autowired.

This is saying: Hey! I don't know what you want me to send for this argument! Put the bind back, then, in UploaderHelper... here it is. Change the argument to match the bind: $uploadFilesystem.

125 lines src/Service/UploaderHelper.php
... lines 1 - 13
class UploaderHelper
{
... lines 16 - 27
public function __construct(FilesystemInterface $uploadsFilesystem, RequestStackContext $requestStackContext, LoggerInterface $logger, string $uploadedAssetsBaseUrl)
{
$this->filesystem = $uploadsFilesystem;
... lines 31 - 33
}
... lines 35 - 123
}

Oh, and there's one more thing we can get rid of! Do we need the public/uploads directory anymore? No! Delete it! And inside .gitignore, we can remove the custom public/uploads/ line we added.

So by putting things in S3... it simplifies things!

Next: now that I've been complimenting our S3 setup and saying how awesome it, I have a... confession to make! We've just introduced a hidden performance bug. Let's crush it!

Leave a comment!

Most Useful Newest Oldest
5
Login or Register to join the conversation
Mathew Avatar
Mathew Avatar Mathew | posted 1 year ago | edited

So I seem to be having some issues with this tutorial. I am adapting your tutorial, and instead of uploading public article images, I am attempting to upload public user profile pictures. I have been following the tutorial so far and everything is going great locally. I upload an image using Flysystem, and it works like a charm. Imagine bundle then grabs it, does its thing and puts them in the specified thumbnails folders.

Then, when I send my code up to my EC2 instance, it appears to be working but no.. Flysystem takes the file and uploads it to S3, no problem. Imagine Bundle though does not upload any of the thumbnails, but DOES try to read them as if they are there, so my site has a bunch of broken links. I have gone in to the AWS Policy checker to make sure the Id and Secret I use have access to the bucket and it all checks out. I have removed all the security options on my testing S3 bucket to make sure any of those are stopping the thumbnail uploads, and it still doesn't put the files in.

I'm running out of Ideas! Any thing you can recommend?
Here are some configs for you for context.

My latest attempt, I was creating a separate filesystem with hardcoded public visibility. Normally I just use the uploads_filesystem
oneup_flysystem.yaml


services:
    Aws\S3\S3Client:
        arguments:
            -   version: '2006-03-01' # or 'latest'
                region: 'us-west-2'
                #credentials:
                #    key: '%s3_key%'
                #    secret: '%s3_secret%'


oneup_flysystem:

    adapters:
        uploads_adapter:
            awss3v3:
                client: Aws\S3\S3Client
                bucket: '%s3_bucket%'
                
        imagine_adapter:
            awss3v3:
                client: Aws\S3\S3Client
                bucket: '%s3_bucket%'
                


    filesystems:

       uploads_filesystem:
           adapter: 'uploads_adapter'

       imagine_filesystem:
           adapter: 'imagine_adapter'
           visibility: public

Imagine Bundle config, again set up with my most recent attempt to force a seperate public filesystem specifically for imagine.


liip_imagine:
    # valid drivers options include "gd" or "gmagick" or "imagick"
    driver: "gd"
    loaders:
        flysystem_loader:
            flysystem:
                filesystem_service: 'oneup_flysystem.imagine_filesystem_filesystem'

    data_loader: flysystem_loader

    resolvers:
        flysystem_resolver:
            flysystem:
                filesystem_service: 'oneup_flysystem.imagine_filesystem_filesystem'
                root_url: '%uploads_base_url%'
                cache_prefix: thumbnails
                #visibility: public

    cache: flysystem_resolver




    filter_sets:

        squared_thumbnail_small:
            filters:
                thumbnail:
                    size:           [200,200]
                    mode:           outbound
                    allow_upscale:  true

        user_list_thumbnail:
            filters:
                auto_rotate: ~
                thumbnail:
                    size:           [45,45]
                    mode:           outbound
                    allow_upscale:  true

        squared_thumbnail_medium:
            filters:
                thumbnail:
                    size:           [500,500]
                    mode:           outbound
                    allow_upscale:  true

My uploads manager is literally straight out of your tutorial with no changes other then I'm focusing on user photos and not article images and references.
Props to Kiuega below for the Visibility tip.



    private function uploadFile(File $file, string $directory, bool $isPublic) : string
    {
        if ($file instanceof UploadedFile)
            $originalFileName = $file->getClientOriginalName();
        else
            $originalFileName = $file->getFilename();

        $newFileName = Urlizer::urlize(pathinfo($originalFileName, PATHINFO_FILENAME)).uniqid().'.'.$file->guessExtension();

        $fileSystem = $this->fileSystem;

        $stream = fopen($file->getPathname(), 'r');

        try {
            $fileSystem->writeStream(
                $directory.'/'.$newFileName,
                $stream,
                [
                    'visibility' => $isPublic ? Visibility::PUBLIC : Visibility::PRIVATE
                ]
            );
        } catch (FilesystemException $e) {
            throw new Exception(sprintf('Exception occurred: "%s"', $e));
        }
        if (is_resource($stream))
            fclose($stream);
        return $newFileName;
    }
Reply
Mathew Avatar

UGh.. I thought I proof read before I posted. I should have stated that I do have the S3 Client credentials commented out due to already having another account set up on the EC2 box for my AWS Secrets Manager service. I am using the same user to bring in all the different policies for AWS. By not providing the credentials it just defaults to the authenticated already established IAM User... I think... I read that somewhere.. Anyways, its working locally, just not on staging.

Reply
weaverryan Avatar

Hey @Mathew!

Darn! So close that you have it working locally... but not on staging!

Imagine Bundle though does not upload any of the thumbnails, but DOES try to read them as if they are there, so my site has a bunch of broken links.

Ok, so this is the part we need to debug. And this can be tricky. Remember, there are two possible URLs that ImagineBundle might output. They are of the form:

A) /media/cache/resolve/squared_thumbnail_small/... (taken from the tutorial example: https://symfonycasts.com/screencast/symfony-uploads/thumbnailing). This is NOT meant to be the real path to a real image: this is path to a Symfony route/controller whose job it is to thumbnail and upload the thumbnail to its final location. In theory, this path would render just ONE time (the first time a thumbnail is rendered). Every other time after (since the thumbnail will now already be created), then next pat his rendered:

B) https://some.path.to/the/real/physical/image. For you, this would be a URL to S3.

You mentioned that the thumbnails are failing to UPLOAD to S3 (vs they ARE being uploaded, but you can't access them). This means that something is going wrong when the (A) URL is rendered. If you can, try to isolate (and then copy) a URL in the (A) format. Depending on how things are setup, you might only see this once right after uploading an image, or you may ONLY see this URL form (since the system may correctly be realizing that the thumbnails did NOT upload... and so it's trying again and again and again).

Anyways, once you have this URL, you can paste it into your browser and start debugging. Ideally you can hack some code onto your staging server to see what's going wrong. If you can put your staging server into the "dev" environment, you may even be able to tail var/logs/dev.log: there could be details about what's going wrong there. For debugging code, I'd put some dd() statements into some core classes, like https://github.com/liip/LiipImagineBundle/blob/2.x/Binary/Loader/FlysystemLoader.php or https://github.com/liip/LiipImagineBundle/blob/2.x/Imagine/Cache/Resolver/FlysystemResolver.php#L118

That's some, somewhat generic instructions. But let me know if it helps you get anywhere :).

Cheers!

Reply
Kiuega Avatar
Kiuega Avatar Kiuega | posted 2 years ago | edited

Hello ! With the latest versions, we no longer have the AdapterInterface class, so we cannot run the following code:


          $this->filesystem->writeStream(
                $directory . '/' . $newFilename,
                $stream,
                [
                    'visibility' => $isPublic ? AdapterInterface::VISIBILITY_PUBLIC : AdapterInterface::VISIBILITY_PRIVATE
                ]
          );

Is there an alternative ?

<b>EDIT</b> : I thought of simply putting 'public' or 'private' rather than using constants from AdapterInterface (non-existent on the latest version).

But by digging a bit I found the <b>League\Flysystem\Visibility</b> class

It also contains two constants which do exactly the same thing.

I imagine this is the class that is used to replace the AdapterInterface?

Reply
sadikoff Avatar

Hey Kiuega

You are right! That will work! :)

Cheers!

1 Reply
Cat in space

"Houston: no signs of life"
Start the conversation!

This tutorial is built on Symfony 4 but works great in Symfony 5!
Symfony 4

What PHP libraries does this tutorial use?

// composer.json
{
    "require": {
        "php": "^7.1.3",
        "ext-iconv": "*",
        "aws/aws-sdk-php": "^3.87", // 3.87.10
        "composer/package-versions-deprecated": "^1.11", // 1.11.99
        "knplabs/knp-markdown-bundle": "^1.7", // 1.7.1
        "knplabs/knp-paginator-bundle": "^2.7", // v2.8.0
        "knplabs/knp-time-bundle": "^1.8", // 1.9.0
        "league/flysystem-aws-s3-v3": "^1.0", // 1.0.22
        "league/flysystem-cached-adapter": "^1.0", // 1.0.9
        "liip/imagine-bundle": "^2.1", // 2.1.0
        "nexylan/slack-bundle": "^2.0,<2.2.0", // v2.1.0
        "oneup/flysystem-bundle": "^3.0", // 3.0.3
        "php-http/guzzle6-adapter": "^1.1", // v1.1.1
        "sensio/framework-extra-bundle": "^5.1", // v5.2.4
        "stof/doctrine-extensions-bundle": "^1.3", // v1.3.0
        "symfony/asset": "^4.0", // v4.2.3
        "symfony/console": "^4.0", // v4.2.3
        "symfony/flex": "^1.9", // v1.17.6
        "symfony/form": "^4.0", // v4.2.3
        "symfony/framework-bundle": "^4.0", // v4.2.3
        "symfony/orm-pack": "^1.0", // v1.0.6
        "symfony/security-bundle": "^4.0", // v4.2.3
        "symfony/serializer-pack": "^1.0", // v1.0.2
        "symfony/twig-bundle": "^4.0", // v4.2.3
        "symfony/validator": "^4.0", // v4.2.3
        "symfony/web-server-bundle": "^4.0", // v4.2.3
        "symfony/yaml": "^4.0", // v4.2.3
        "twig/extensions": "^1.5" // v1.5.4
    },
    "require-dev": {
        "doctrine/doctrine-fixtures-bundle": "^3.0", // 3.1.0
        "easycorp/easy-log-handler": "^1.0.2", // v1.0.7
        "fzaninotto/faker": "^1.7", // v1.8.0
        "symfony/debug-bundle": "^3.3|^4.0", // v4.2.3
        "symfony/dotenv": "^4.0", // v4.2.3
        "symfony/maker-bundle": "^1.0", // v1.11.3
        "symfony/monolog-bundle": "^3.0", // v3.3.1
        "symfony/phpunit-bridge": "^3.3|^4.0", // v4.2.3
        "symfony/profiler-pack": "^1.0", // v1.0.4
        "symfony/var-dumper": "^3.3|^4.0" // v4.2.3
    }
}
Edit on GitHub
Previous Chapter
Next Chapter
userVoice