Updates and Backfills¶
Once a search or stream has been run around a focal event, it is easy to update or backfill the tweets with additional data.
Updates and End Times¶
The focal event can be updated with all the tweets that have occurred since the stream/search was run. To do this, use the update
flag
python -m twitter.search event_name --update
By default, the update looks at the last focal event tweet that came from the stream/search and gets all tweets that occurred from then to the moment of running the update.
To change when the update ends, we can use the end_time
parameter. A specific time can be passed to the end_time
, and the update will run from the last search/stream tweet to that time. For example, if we want to run our update until 11am UTC on August 18th, 2021, then we can enter
python -m twitter.search event_name --update -end_time 2021-08-18T11:00:00.00Z
We can also set the end_time
to the value last_time
and use the parameter n_days_after
to modify how many days after the last search/stream tweet time that we want to run the update. For example, if we wanted to run the update for the 3 days following the last tweet from our search/stream, then we would do
python -m twitter.search event_name --update -end_time last_time -n_days_after 3
Backfills and Start Times¶
In addition to updating our dataset, we can also backfill it with tweets that occurred before the earliest tweet in our search/stream. By default, the backfill runs from the beginning of the day of the earliest tweet from the search/stream until the time of that tweet. We can run that basic backfill as
python -m twitter.search event_name --backfill
Like the update, we can also set the backfill start manually using the start_time
. The start_time
can either be a specific time or a time relative to the first_time
using the n_days_before
parameter. For example, if we wanted to get all the focal event tweets that occurred in the week prior to the first tweet in our search/stream, then we would enter
python -m twitter.search event_name --backfill -start_time first_time -n_days_before 7
Note
All time parameters need to be RFC 3339 format:
YYYY-MM-DDT00:00:00.00Z
Note
The start and end times can be used together for any search, not just updates and backfills. That is, a different start_time
can be used for an update, a different end_time
can be used for a backfill, and in general both parameters can be used to produce any time window.