How to upload file to GCS from GAE/GCE using Python

It is possible to upload files to Google Cloud Storage on several ways. We can use Blobstore API, create upload url using create_upload_url method and POST a file to this url. We can use Google Cloud Storage Client and upload a file from our custom handler. In this post I will describe how to upload file using Google API Client Library.

There is a small difference between upload from Google App Engine and Google Compute Engine. For both platforms we need to use Google API Python Client library. On Google Compute Engine we can just install library using pip.

pip install --upgrade google-api-python-client

On Google App Engine you should add this library as third-party package as other custom libraries.

I assume two things:

GAE application or GCE instance have access to Google Cloud Storage and correct permissions to bucket
file is send in image field (self.request.POST[‘image’])

Example code.

import json

from apiclient import discovery
from apiclient.http import MediaIoBaseUpload
from google.appengine.api.blobstore import create_gs_key
from google.appengine.api.images import get_serving_url
from oauth2client.client import GoogleCredentials
from webapp2 import RequestHandler


class UploadHandler(RequestHandler):

    def post(self):
        image = self.request.POST.get('image', None)

        """
        Validation ideas:
        image.type - contains file mimetype
        image.file.tell() - contains file size (image.file.seek(0, os.SEEK_END))
        """

        # Retrieve credentials
        credentials = GoogleCredentials.get_application_default()
        # Create service object
        service = discovery.build('storage', 'v1', credentials=credentials)
        # Create media object
        media = MediaIoBaseUpload(
            image.file,
            mimetype=image.type
        )
        # Create request
        request = service.objects().insert(
            bucket='MY_BUCKET',
            name='test.jpg',
            media_body=media
        )
        # Execute request - store file on GCS
        response = request.execute()

        # Build path to file on GCS
        gcs_path = '/gs/{bucket}/{filename}'.format(
            bucket=response['bucket'],
            filename=response['name'],
        )
        # Build BlobKey
        blob_key = create_gs_key(gcs_path)
        # Create url to image
        image_url = get_serving_url(blob_key)

        return self.response.out.write(
            json.dumps({'gcs_path': gcs_path, 'image_url': image_url})
        )

GoogleCredentials.get_application_default() will retrieve credentials from environment so we don’t need to care about them. Using discovery.build we can build service object. In this example upload is executed from webapp2 handler so we use MediaIoBaseUpload class to build object required for media_body parameter. First parameter is a file (io.Base) which is send from html form, second parameter is a mimetype of this file. We can pass to service.objects().insert method a bucket name, file name and previous prepared MediaIoBaseUpload object. At the end we should call request.execute() to store file on Google Cloud Storage.

Here is an example response from request.execute():

{
  "kind": "storage#object",
  "contentType": "image/jpeg",
  "name": "test.jpg",
  "etag": "CKiZ7Z+NcscCEAE=",
  "generation": "1434332916697000",
  "md5Hash": "DSSWHKT/hgkbQh75AxUGDw==",
  "bucket": "MY_BUCKET",
  "updated": "2015-08-11T22:43:16.697Z",
  "owner": {
    "entityId": "01b4903a17ed47fa59cf117324999ecf3d1be3ea0dd1263e5044bb7dc2f2ba4f",
    "entity": "user-01b4903a17ed47fa59cf117324999ecf3d1be3ea0dd1263e5044bb7dc2f2ba4f"
  },
  "crc32c": "oBtnBc==",
  "metageneration": "1",
  "storageClass": "STANDARD",
  "mediaLink": "https://www.googleapis.com/download/storage/v1/b/MY_BUCKET/o/test.jpg?generation=1433332991697000&alt=media",
  "id": "MY_BUCKET/test.jpg/1433332991697000",
  "selfLink": "https://www.googleapis.com/storage/v1/b/MY_BUCKET/o/test.jpg",
  "size": "1008926"
}

Of course we can add some validation to handler before file upload like checking file type, file size, etc.