unable to install awscli using pip on dind 18.09.6-dind due to cython dependency

I saw a strange issue just most of yesterday where while running a simple jenkins build that uses pod template with container docker:18.09.6-dind (alpine linux) the build would fail while trying to install awscli using pip. Here is the sample code:-

apk add curl
apk add py-pip
pip install --user awscli

The error was :-

apk add py-pip
(1/11) Installing libbz2 (1.0.6-r7)
(2/11) Installing expat (2.2.8-r0)
(3/11) Installing libffi (3.2.1-r6)
(4/11) Installing gdbm (1.13-r1)
(5/11) Installing ncurses-terminfo-base (6.1_p20190518-r2)
(6/11) Installing ncurses-libs (6.1_p20190518-r2)
(7/11) Installing readline (8.0.0-r0)
(8/11) Installing sqlite-libs (3.28.0-r3)
(9/11) Installing python2 (2.7.18-r0)
(10/11) Installing py-setuptools (40.8.0-r1)
(11/11) Installing py2-pip (18.1-r0)
Executing busybox-1.30.1-r2.trigger
OK: 74 MiB in 54 packages
[Pipeline] withAwsCli
Executing sh script inside container dind of pod pod-custom-w741q
Executing command: "pip" "--version" 
exit
pip 18.1 from /usr/lib/python2.7/site-packages/pip (python 2.7)
Executing sh script inside container dind of pod pod-custom-w741q
Executing command: "pip" "show" "awscli" 
exit
ERROR: [withAwsCli] Command 'pip show awscli' returned the code 1
Executing sh script inside container dind of pod pod-custom-w741q
Executing command: "pip" "install" "--user" "awscli" 
exit
Collecting awscli
  Downloading https://files.pythonhosted.org/packages/d9/d3/1b76aa71fea8438f82a991ad94443d343836848f8b8c0dc413ed987745f8/awscli-1.19.51-py2.py3-none-any.whl (3.6MB)
Collecting botocore==1.20.51 (from awscli)
  Downloading https://files.pythonhosted.org/packages/3c/1d/7a3741f17a10cc599bb3728351992b10094328ee07b8a129250ba2039642/botocore-1.20.51-py2.py3-none-any.whl (7.4MB)
Collecting colorama<0.4.4,>=0.2.5 (from awscli)
  Downloading https://files.pythonhosted.org/packages/c9/dc/45cdef1b4d119eb96316b3117e6d5708a08029992b2fee2c143c7a0a5cc5/colorama-0.4.3-py2.py3-none-any.whl
Collecting PyYAML<5.5,>=3.10 (from awscli)
  Downloading https://files.pythonhosted.org/packages/a0/a4/d63f2d7597e1a4b55aa3b4d6c5b029991d3b824b5bd331af8d4ab1ed687d/PyYAML-5.4.1.tar.gz (175kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  Complete output from command /usr/bin/python2 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-mT9r0n --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel Cython:
  Collecting setuptools
    Downloading https://files.pythonhosted.org/packages/e1/b7/182161210a13158cd3ccc41ee19aadef54496b74f2817cc147006ec932b4/setuptools-44.1.1-py2.py3-none-any.whl (583kB)
  Collecting wheel
    Downloading https://files.pythonhosted.org/packages/65/63/39d04c74222770ed1589c0eaba06c05891801219272420b40311cd60c880/wheel-0.36.2-py2.py3-none-any.whl
  Collecting Cython
    Downloading https://files.pythonhosted.org/packages/d9/cd/0d2d90b27219c07f68f1c25bcc7b02dd27639d2180add9d4b73e70945869/Cython-0.29.23.tar.gz (2.1MB)
  Installing collected packages: setuptools, wheel, Cython
    Running setup.py install for Cython: started
      Running setup.py install for Cython: finished with status 'error'
      Complete output from command /usr/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-72Od1r/Cython/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('rn', 'n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-Qfbz7l/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-mT9r0n --compile:
      Unable to find pgen, not compiling formal grammar.
      running install
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-2.7
      copying cython.py -> build/lib.linux-x86_64-2.7
      creating build/lib.linux-x86_64-2.7/Cython
      copying Cython/TestUtils.py -> build/lib.linux-x86_64-2.7/Cython
      copying Cython/Utils.py -> build/lib.linux-x86_64-2.7/Cython
      copying Cython/CodeWriter.py -> build/lib.linux-x86_64-2.7/Cython
      copying Cython/Debugging.py -> build/lib.linux-x86_64-2.7/Cython
      copying Cython/StringIOTree.py -> build/lib.linux-x86_64-2.7/Cython
      copying Cython/__init__.py -> build/lib.linux-x86_64-2.7/Cython
      copying Cython/Coverage.py -> build/lib.linux-x86_64-2.7/Cython
      copying Cython/Shadow.py -> build/lib.linux-x86_64-2.7/Cython
      creating build/lib.linux-x86_64-2.7/Cython/Build
      copying Cython/Build/Dependencies.py -> build/lib.linux-x86_64-2.7/Cython/Build
      copying Cython/Build/__init__.py -> build/lib.linux-x86_64-2.7/Cython/Build
      copying Cython/Build/Distutils.py -> build/lib.linux-x86_64-2.7/Cython/Build
      copying Cython/Build/Inline.py -> build/lib.linux-x86_64-2.7/Cython/Build
      copying Cython/Build/BuildExecutable.py -> build/lib.linux-x86_64-2.7/Cython/Build
      copying Cython/Build/Cythonize.py -> build/lib.linux-x86_64-2.7/Cython/Build
      copying Cython/Build/IpythonMagic.py -> build/lib.linux-x86_64-2.7/Cython/Build
      creating build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/FlowControl.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Code.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/MemoryView.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Pipeline.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/AnalysedTreeTransforms.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/CythonScope.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/TypeSlots.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Errors.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/FusedNode.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/PyrexTypes.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/AutoDocTransforms.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/UtilityCode.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/StringEncoding.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Version.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/CmdLine.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/UtilNodes.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/ParseTreeTransforms.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Visitor.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/TypeInference.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/__init__.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Options.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/DebugFlags.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Scanning.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/ExprNodes.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Interpreter.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Builtin.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Annotate.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Symtab.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Optimize.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Parsing.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Naming.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/ModuleNode.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/TreeFragment.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Nodes.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Future.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/TreePath.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Main.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/CodeGeneration.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Buffer.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Pythran.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      copying Cython/Compiler/Lexicon.py -> build/lib.linux-x86_64-2.7/Cython/Compiler
      creating build/lib.linux-x86_64-2.7/Cython/Runtime
      copying Cython/Runtime/__init__.py -> build/lib.linux-x86_64-2.7/Cython/Runtime
      creating build/lib.linux-x86_64-2.7/Cython/Distutils
      copying Cython/Distutils/old_build_ext.py -> build/lib.linux-x86_64-2.7/Cython/Distutils
      copying Cython/Distutils/__init__.py -> build/lib.linux-x86_64-2.7/Cython/Distutils
      copying Cython/Distutils/build_ext.py -> build/lib.linux-x86_64-2.7/Cython/Distutils
      copying Cython/Distutils/extension.py -> build/lib.linux-x86_64-2.7/Cython/Distutils
      creating build/lib.linux-x86_64-2.7/Cython/Debugger
      copying Cython/Debugger/DebugWriter.py -> build/lib.linux-x86_64-2.7/Cython/Debugger
      copying Cython/Debugger/libpython.py -> build/lib.linux-x86_64-2.7/Cython/Debugger
      copying Cython/Debugger/__init__.py -> build/lib.linux-x86_64-2.7/Cython/Debugger
      copying Cython/Debugger/Cygdb.py -> build/lib.linux-x86_64-2.7/Cython/Debugger
      copying Cython/Debugger/libcython.py -> build/lib.linux-x86_64-2.7/Cython/Debugger
      creating build/lib.linux-x86_64-2.7/Cython/Debugger/Tests
......................................
......................................
      running build_ext
      building 'Cython.Plex.Scanners' extension
      creating build/temp.linux-x86_64-2.7
      creating build/temp.linux-x86_64-2.7/tmp
      creating build/temp.linux-x86_64-2.7/tmp/pip-install-72Od1r
      creating build/temp.linux-x86_64-2.7/tmp/pip-install-72Od1r/Cython
      creating build/temp.linux-x86_64-2.7/tmp/pip-install-72Od1r/Cython/Cython
      creating build/temp.linux-x86_64-2.7/tmp/pip-install-72Od1r/Cython/Cython/Plex
      gcc -fno-strict-aliasing -Os -fomit-frame-pointer -g -DNDEBUG -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/include/python2.7 -c /tmp/pip-install-72Od1r/Cython/Cython/Plex/Scanners.c -o build/temp.linux-x86_64-2.7/tmp/pip-install-72Od1r/Cython/Cython/Plex/Scanners.o
      unable to execute 'gcc': No such file or directory
      error: command 'gcc' failed with exit status 1
  
      ----------------------------------------
  Command "/usr/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-72Od1r/Cython/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('rn', 'n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-Qfbz7l/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-mT9r0n --compile" failed with error code 1 in /tmp/pip-install-72Od1r/Cython/
  
  ----------------------------------------
Command "/usr/bin/python2 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-mT9r0n --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel Cython" failed with error code 1 in None

It always worked fine until yesterday and also works fine now on same dind OS. I have no clue why was it trying to set up dependency cython just yesterday. I see a new version of cython was released yesterday https://pypi.org/search/?q=cython Not sure if that played any role I am not that familiar with how pip works when it tries to install something so would like to understand why it was looking for cython all of a sudden while its the same pip version 18.1 on that dind OS and trying to fetch latest awscli which happened to be released just yesterday 1.19.51 as per https://github.com/aws/aws-cli/releases Also yesterday same thing worked just fine all throughout on docker:18.06.1-ce-dind or other dind versions. So something specific to docker:18.09.6-dind and combination of pip 18.1 and awscli 1.19.51. kindly help me figure out what could have been the issue here.

Answer

Found out the root cause which is https://github.com/aws/aws-cli/issues/6096