August 14, 2015

November 25, 2019

How to upload file to GCS from GAE/GCE using Python

It is possible to upload files to Google Cloud Storage on several ways. We can use Blobstore API, create upload url using create_upload_url method and POST a file to this url. We can use Google Cloud Storage Client and upload a file from our custom handler. In this post I will describe how to upload file using Google API Client Library.

There is a small difference between upload from Google App Engine and Google Compute Engine. For both platforms we need to use Google API Python Client library. On Google Compute Engine we can just install library using pip.

pip install --upgrade google-api-python-client

On Google App Engine you should add this library as third-party package as other custom libraries.

I assume two things:

  • GAE application or GCE instance have access to Google Cloud Storage and correct permissions to bucket
  • file is send in image field (self.request.POST['image'])

Example code.

import json
from apiclient import discovery
from apiclient.http import MediaIoBaseUpload
from google.appengine.api.blobstore import create_gs_key
from google.appengine.api.images import get_serving_url
from oauth2client.client import GoogleCredentials
from webapp2 import RequestHandler
class UploadHandler(RequestHandler):
def post(self):
image = self.request.POST.get('image', None)
"""
Validation ideas:
image.type - contains file mimetype
image.file.tell() - contains file size (image.file.seek(0, os.SEEK_END))
"""
# Retrieve credentials
credentials = GoogleCredentials.get_application_default()
# Create service object
service = discovery.build('storage', 'v1', credentials=credentials)
# Create media object
media = MediaIoBaseUpload(
image.file,
mimetype=image.type
)
# Create request
request = service.objects().insert(
bucket='MY_BUCKET',
name='test.jpg',
media_body=media
)
# Execute request - store file on GCS
response = request.execute()
# Build path to file on GCS
gcs_path = '/gs/{bucket}/{filename}'.format(
bucket=response['bucket'],
filename=response['name'],
)
# Build BlobKey
blob_key = create_gs_key(gcs_path)
# Create url to image
image_url = get_serving_url(blob_key)
return self.response.out.write(
json.dumps({'gcs_path': gcs_path, 'image_url': image_url})
)

GoogleCredentials.get_application_default() will retrieve credentials from environment so we don't need to care about them. Using discovery.build we can build service object. In this example upload is executed from webapp2 handler so we use MediaIoBaseUpload class to build object required for media_body parameter. First parameter is a file (io.Base) which is send from html form, second parameter is a mimetype of this file. We can pass to service.objects().insert method a bucket name, file name and previous prepared MediaIoBaseUpload object. At the end we should call request.execute() to store file on Google Cloud Storage.

Here is an example response from request.execute():

{
"kind": "storage#object",
"contentType": "image/jpeg",
"name": "test.jpg",
"etag": "CKiZ7Z+NcscCEAE=",
"generation": "1434332916697000",
"md5Hash": "DSSWHKT/hgkbQh75AxUGDw==",
"bucket": "MY_BUCKET",
"updated": "2015-08-11T22:43:16.697Z",
"owner": {
"entityId": "01b4903a17ed47fa59cf117324999ecf3d1be3ea0dd1263e5044bb7dc2f2ba4f",
"entity": "user-01b4903a17ed47fa59cf117324999ecf3d1be3ea0dd1263e5044bb7dc2f2ba4f"
},
"crc32c": "oBtnBc==",
"metageneration": "1",
"storageClass": "STANDARD",
"mediaLink": "https://www.googleapis.com/download/storage/v1/b/MY_BUCKET/o/test.jpg?generation=1433332991697000&alt=media",
"id": "MY_BUCKET/test.jpg/1433332991697000",
"selfLink": "https://www.googleapis.com/storage/v1/b/MY_BUCKET/o/test.jpg",
"size": "1008926"
}

Of course we can add some validation to handler before file upload like checking file type, file size, etc.

More at documentation

© 2020 Przemysław Kołodziejczyk