Skip to content
Advertisement

Generator as function argument

Can anyone explain why passing a generator as the only positional argument to a function seems to have special rules?

If we have:

JavaScript
  1. This works, as expected.

    JavaScript
  2. This does not work, as expected.

    JavaScript
  3. This works, as expected

    JavaScript
  4. This works, but I don’t understand why. Shouldn’t it fail in the same way as 2)

    JavaScript

Advertisement

Answer

Both 3. and 4. should be syntax errors on all Python versions. However you’ve found a bug that affects Python versions 2.5 – 3.4, and which was subsequently posted to the Python issue tracker. Because of the bug, an unparenthesized generator expression was accepted as an argument to a function if it was accompanied only by *args and/or **kwargs. While Python 2.6+ allowed both cases 3. and 4., Python 2.5 allowed only case 3. – yet both of them were against the documented grammar:

JavaScript

i.e. the documentation says a function call comprises of primary (the expression that evaluates to a callable), followed by, in parentheses, either an argument list or just an unparenthesized generator expression; and within the argument list, all generator expressions must be in parentheses.


This bug (though it seems it had not been known), had been fixed in Python 3.5 prereleases. In Python 3.5 parentheses are always required around a generator expression, unless it is the only argument to the function:

JavaScript

This is now documented in the What’s New in Python 3.5, thanks to DeTeReR spotting this bug.


Analysis of the bug

There was a change made to Python 2.6 which allowed the use of keyword arguments after *args:

It’s also become legal to provide keyword arguments after a *args argument to a function call.

JavaScript

Previously this would have been a syntax error. (Contributed by Amaury Forgeot d’Arc; issue 3473.)


However, the Python 2.6 grammar does not make any distinction between keyword arguments, positional arguments, or bare generator expressions – they are all of type argument to the parser.

As per Python rules, a generator expression must be parenthesized if it is not the sole argument to the function. This is validated in the Python/ast.c:

JavaScript

However this function does not consider the *args at all – it specifically only looks for ordinary positional arguments and keyword arguments.

Further down in the same function, there is an error message generated for non-keyword arg after keyword arg:

JavaScript

But this again applies to arguments that are not unparenthesized generator expressions as evidenced by the else if statement:

JavaScript

Thus an unparenthesized generator expression was allowed to slip pass.


Now in Python 3.5 one can use the *args anywhere in a function call, so the Grammar was changed to accommodate for this:

JavaScript

and

JavaScript

and the for loop was changed to

JavaScript

Thus fixing the bug.

However the inadvertent change is that the valid looking constructions

JavaScript

and

JavaScript

where an unparenthesized generator precedes *args or **kwargs now stopped working.


To locate this bug, I tried various Python versions. In 2.5 you’d get SyntaxError:

JavaScript

And this was fixed before some prerelease of Python 3.5:

JavaScript

However, the parenthesized generator expression, it works in Python 3.5, but it does not work not in Python 3.4:

JavaScript

And this is the clue. In Python 3.5 the *splatting is generalized; you can use it anywhere in a function call:

JavaScript

So the actual bug (generator working with *star without parentheses) was indeed fixed in Python 3.5, and the bug could be found in that what changed between Python 3.4 and 3.5

Advertisement