I installed airflow (v 2.2.4) on the ami below: The folder it currently lives in is here (But should I relocate this somewhere else per these docs?) AMI INFO: And the issue I am having is that after I run airflow initdb - which works great, I run airflow webserver and when I try to copy in the http value

Airflow: ‘ERR_CONNECTION_REFUSED’ on port 8080 & 8081

I installed airflow (v 2.2.4) on the ami below:

The folder it currently lives in is here (But should I relocate this somewhere else per these docs?)

/home/ec2-user/airflow/airflow.cfg
##WHEN I RUN echo $AIRFLOW_HOME I get the below:
/home/ec2-user/airflow

AMI INFO:

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
POSTGRES V = 10.17

And the issue I am having is that after I run airflow initdb – which works great, I run airflow webserver and when I try to copy in the http value after “Listening at”: I get a ERR_CONNECTION_REFUSED ‘this site can’t be reached’ error in google chrome (same with other browsers).

 ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ _ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  ____/____/|__/
[2022-04-05 18:08:55,769] {dagbag.py:500} INFO - Filling up the DagBag from /dev/null
[2022-04-05 18:08:55,819] {manager.py:779} WARNING - No user yet created, use flask fab command to do it.
[2022-04-05 18:08:56,185] {manager.py:512} WARNING - Refused to delete permission view, assoc with role exists DAG Runs.can_create User
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8081
Timeout: 120
Logfiles: - -
Access Logformat: 
=================================================================            
[2022-04-05 18:08:58 +0000] [9318] [INFO] Starting gunicorn 20.1.0
[2022-04-05 18:08:58 +0000] [9318] [INFO] Listening at: http://0.0.0.0:8081 (9318)
[2022-04-05 18:08:58 +0000] [9318] [INFO] Using worker: sync
[2022-04-05 18:08:58 +0000] [9321] [INFO] Booting worker with pid: 9321
[2022-04-05 18:08:58 +0000] [9322] [INFO] Booting worker with pid: 9322
[2022-04-05 18:08:59 +0000] [9323] [INFO] Booting worker with pid: 9323
[2022-04-05 18:08:59 +0000] [9324] [INFO] Booting worker with pid: 9324
[2022-04-05 18:09:00,798] {manager.py:779} WARNING - No user yet created, use flask fab command to do it.
[2022-04-05 18:09:00,845] {manager.py:779} WARNING - No user yet created, use flask fab command to do it.
[2022-04-05 18:09:01,084] {manager.py:779} WARNING - No user yet created, use flask fab command to do it.
[2022-04-05 18:09:01,182] {manager.py:779} WARNING - No user yet created, use flask fab command to do it.
[2022-04-05 18:09:01,575] {manager.py:512} WARNING - Refused to delete permission view, assoc with role exists DAG Runs.can_create User
[2022-04-05 18:09:01,605] {manager.py:512} WARNING - Refused to delete permission view, assoc with role exists DAG Runs.can_create User
[2022-04-05 18:09:01,888] {manager.py:512} WARNING - Refused to delete permission view, assoc with role exists DAG Runs.can_create User

This initially happened when it was on port 8080, but per this post I tried a new port – but still no luck. Airflow Config details below:

# If you set web_server_url_prefix, do NOT forget to append it here, ex:
# ``endpoint_url = http://localhost:8080/myroot``
# So api will look like: ``http://localhost:8080/myroot/api/experimental/...``
endpoint_url = http://localhost:8081

[webserver]
    # The base url of your website as airflow cannot guess what domain or
    # cname you are using. This is used in automated emails that
    # airflow sends to point links to the right web server
    base_url = http://localhost:8081

# Default timezone to display all dates in the UI, can be UTC, system, or 
# any IANA timezone string (e.g. Europe/Amsterdam). If left empty the
# default value of core/default_timezone will be used
# Example: default_ui_timezone = America/New_York
default_ui_timezone = UTC

# The ip specified when starting the web server
web_server_host = 0.0.0.0

# The port on which to run the web server
web_server_port = 8081

Answer

That is not an Airflow problem – it’s basic remote computing problem.

You run Airflow on remote machine.

If you use localhost, you try to connect to your local one. You need to make sure you connect to the remote machine not to your local host.

You need to setup some way to open your port to be accessible from your local machine where Browser is.

You can do it for example in the way described here:

https://aws.amazon.com/blogs/aws/new-port-forwarding-using-aws-system-manager-sessions-manager/

Advertisement

Answer