In some of my Apache Airflow installations, DAGs or tasks that are scheduled to run do not run even when the scheduler doesn’t appear to be fully loaded. How can I increase the number of DAGs or tasks that can run concurrently? Similarly, if my installation is under high load and I want to limit how qui…
Tag: google-cloud-platform
Connecting to Cloud SQL from Google Cloud Function using Python and SQLAlchemy
I read all documentation related to connecting to MysQL hosted in Cloud SQL from GCF and still can’t connect. Also, tried all hints in documentation of SQLAlchemy related to this. I am using the following connection The error I got was: (pymysql.err.OperationalError) (2003, “Can’t connect to…
BigQuery – Best way to DROP date-sharded tables
I have a few date-sharded tables I want to delete but they already have more than 100 shards for each table and can’t drop them manually. I’ve tried using wildcards but it does not seem to work. I finally used the python API: And it works, but I needed to create the tables array with the names of …
Google App Engine deployment issue- main app not found
I am trying to deploy my app using Google App Engine. I have edited app.yaml to reflect the flexible environment and also gave all the app information. Below is the app.yaml file. Once the deployment is in progress, I am getting the following error Please note that App Deployed is the line in my print stateme…
service account does not have storage.objects.get access for Google Cloud Storage
I have created a service account in Google Cloud Console and selected role Storage / Storage Admin (i.e. full control of GCS resources). gcloud projects get-iam-policy my_project seems to indicate that the role was actually selected: And documentation clearly indicates that role roles/storage.admin comprises …
Add GCP credentials to airflow via command line
Airflow allows us to add connection information via command-line airflow connections. This can help with automated deployment of airflow installations via ansible or other dev-ops tools. It is unclear how connections to google cloud platform (service accounts) can be added to ariflow via command line. Answer …
Running a python script on Google Cloud Compute Engine
For a machine learning task at school I wrote my own MLP network. The data set is quite big, and training takes forever. I was alerted to the option of running my script on the Google Cloud Compute Engine. I tried to set this up, but did not succeed (yet). The steps I undertook where: Create an account Create…
Python3 BigQuery or Google Cloud Python through HTTP Proxy
How to route BigQuery client calls through HTTP Proxy ? Before Posting this, I tried following but it is still not routing through http proxy. And the Google Cloud service credentials are set through shell environment variable GOOGLE_APPLICATION_CREDENTIALS Outgoing traffic ( 172.217.x.x belongs to googleapis…