In some of my Apache Airflow installations, DAGs or tasks that are scheduled to run do not run even when the scheduler doesn’t appear to be fully loaded. How can I increase the number of DAGs or tasks that can run concurrently? Similarly, if my installation is under high load and I want to limit how quickly my Airflow workers
Tag: google-cloud-platform
Connecting to Cloud SQL from Google Cloud Function using Python and SQLAlchemy
I read all documentation related to connecting to MysQL hosted in Cloud SQL from GCF and still can’t connect. Also, tried all hints in documentation of SQLAlchemy related to this. I am using the following connection The error I got was: (pymysql.err.OperationalError) (2003, “Can’t connect to MySQL server on ‘localhost’ ([Errno 111] Connection refused)”) (Background on this error at: http://sqlalche.me/e/e3q8)
BigQuery – Best way to DROP date-sharded tables
I have a few date-sharded tables I want to delete but they already have more than 100 shards for each table and can’t drop them manually. I’ve tried using wildcards but it does not seem to work. I finally used the python API: And it works, but I needed to create the tables array with the names of the tables
Google App Engine deployment issue- main app not found
I am trying to deploy my app using Google App Engine. I have edited app.yaml to reflect the flexible environment and also gave all the app information. Below is the app.yaml file. Once the deployment is in progress, I am getting the following error Please note that App Deployed is the line in my print statement. It is getting executed.
service account does not have storage.objects.get access for Google Cloud Storage
I have created a service account in Google Cloud Console and selected role Storage / Storage Admin (i.e. full control of GCS resources). gcloud projects get-iam-policy my_project seems to indicate that the role was actually selected: And documentation clearly indicates that role roles/storage.admin comprises permissions storage.objects.* (as well as storage.buckets.*). But when I try using that service account in conjunction
Add GCP credentials to airflow via command line
Airflow allows us to add connection information via command-line airflow connections. This can help with automated deployment of airflow installations via ansible or other dev-ops tools. It is unclear how connections to google cloud platform (service accounts) can be added to ariflow via command line. Answer Pre airflow 1.9 the following example outlines how to use a DAG to add
Running a python script on Google Cloud Compute Engine
For a machine learning task at school I wrote my own MLP network. The data set is quite big, and training takes forever. I was alerted to the option of running my script on the Google Cloud Compute Engine. I tried to set this up, but did not succeed (yet). The steps I undertook where: Create an account Create a
Python3 BigQuery or Google Cloud Python through HTTP Proxy
How to route BigQuery client calls through HTTP Proxy ? Before Posting this, I tried following but it is still not routing through http proxy. And the Google Cloud service credentials are set through shell environment variable GOOGLE_APPLICATION_CREDENTIALS Outgoing traffic ( 172.217.x.x belongs to googleapis.com ) not routing through HTTP Proxy , Answer Answering the question myself as I found