Skip to content
Advertisement

Tweepy search seems to loop till rate limit is met / not respect the count in the search

I trust all is well with you and yours / thanks for taking a moment to read this. I am trying to figure out why this section of code does not respect 2 things.

  1. the count limit, it is supposed to only pull that many tweets, and it does whatever it wants to do.
  2. the code seems to loop indefinitely, and unless I put a hard stop, it doesn’t stop (hard stop is not currently included, though it would be something akin to adding a for in range loop that matches count. Though, I am not sure that would work either.
import tweepy
import Auth_Codes
import json
import os


def tweet_search(query=""):
    print(f"'Query = {query}'")

    twitter_auth_keys = {
        "consumer_key"          : Auth_Codes.consumer_key,
        "consumer_secret"       : Auth_Codes.consumer_secret,
        "access_token"          : Auth_Codes.access_token,
        "access_token_secret"   : Auth_Codes.access_token_secret
    }
    auth = tweepy.OAuthHandler(
        twitter_auth_keys["consumer_key"],
        twitter_auth_keys["consumer_secret"]
    )

    auth.set_access_token(
        twitter_auth_keys["access_token"],
        twitter_auth_keys["access_token_secret"]
    )

    api = tweepy.API(auth,wait_on_rate_limit=True)

    cursor = tweepy.Cursor(api.search_tweets,q = f'{query}',lang = 'en',result_type = 'recent',count = 1)
    print("Query Complete")

    tweet_payload = []

    for tweet in cursor.items():

        current_searches_remaining = api.rate_limit_status()['resources']['search']['/search/tweets']['remaining']
        current_search_limit = api.rate_limit_status()['resources']['search']['/search/tweets']['limit']
        max_searches = current_search_limit * .25

        if current_searches_remaining >= (max_searches):
            print("Checking rate limit")
            print("Searches Remaining: ", current_searches_remaining, "/", current_search_limit)
            print("Converting payload to a useable format")

            tweet_payload.append(tweet._json)

            with open("Tweet_Payload.json", "w") as outfile:
               outfile.write(json.dumps(tweet_payload,indent=4, sort_keys=True))
            tweet_filter()

        else:
            print("Rate Limit Hit, Moving On")
            tweet_filter()
    print("Search Complete")
    tweet_filter()

Advertisement

Answer

While I am still working out the finer details, after making the following change, it now loops nearly as expected.

cursor = tweepy.Cursor(api.search_tweets,q = f'{query}',lang = 'en',result_type = 'recent',count = 11)
    print("Query Complete")

    tweet_payload = []

    for tweet in cursor.items(11):
User contributions licensed under: CC BY-SA
7 People found this is helpful
Advertisement