Get Unlimited DATA from Twitter (Without API!)

Tyler Germain

6 Feb 202310:45

EducationalLearning

32 Likes 10 Comments

TLDRThe video discusses Twitter's API changes and the discontent among developers due to the new pricing model. It introduces an alternative tool called twint, a Python package that scrapes Twitter data without using the API, bypassing its limitations. The video provides a step-by-step guide on installing twint, using it in the terminal, and incorporating it into a Python script, highlighting its capabilities such as searching for tweets, analyzing user profiles, and setting parameters like tweet limits and popularity criteria.

Takeaways

🚀 Twitter is implementing significant changes to its API, causing discontent among developers.
💰 The new pricing model for the Twitter API is perceived as expensive and unaffordable, potentially impacting businesses relying on it.
📈 The API plans range from 500 to 10,000 requests per month, with enterprise-level plans requiring direct contact for pricing.
🐱 Elon Musk's response to feedback suggests Twitter will introduce a 'light right only' API for bots providing good content, which will be free.
🌐 Twitter's API has limitations, such as only allowing access to the last 3,200 tweets or tweets within the last seven days for keyword searches.
🛠️ Twint is a Python package that enables scraping of Twitter data without using the API, bypassing its limitations.
🔍 Twint allows users to pull tweets based on keywords and user profiles, without the restrictions imposed by the Twitter API.
📚 Documentation for Twint can be found on GitHub, providing details on how to use the tool effectively.
💻 Installation of Twint requires Python and can be done using the package installer for Python (PIP).
🔧 Basic usage of Twint in the terminal includes searching for tweets by keywords and viewing specific users' tweets.
📊 Twint can be utilized within a Python environment, such as VS Code, to perform more complex data scraping tasks.

Q & A

What is the main issue developers have with the new Twitter API pricing model?
-The main issue developers have is that the new pricing model is very expensive and unattainable, which could potentially put any software using the Twitter API out of business.
How does the new pricing structure for Twitter API work?
-The new pricing structure has different paid plans based on API usage, starting from 500 requests per month up to 10,000, with the Enterprise level requiring contact with customer support for pricing.
What was Elon Musk's response to the concerns about the Twitter API pricing?
-Elon Musk responded by saying that Twitter will enable a 'light right only' API for bots providing good content, which will be free.
What is the tool introduced in the script that can be used to scrape Twitter data without using the API?
-The tool introduced is called 'twint', a Python package that allows users to scrape tweets and get information about Twitter without going through the Twitter API.

Outlines

00:00

🚀 Twitter API Changes and Developer Concerns

This paragraph discusses the significant changes Twitter is making to its API, which has caused discontent among developers. The main issue is the new pricing model, which is considered expensive and unachievable, potentially putting software using the Twitter API out of business. The speaker mentions a chart showing the total request per month and the corresponding monthly prices for different API plans, highlighting that large apps with significant user bases will face substantial costs. Elon Musk's response to feedback, promising a free, read-only API for bots providing good content, is also mentioned. The speaker expresses hope for future read-only access for researchers and institutions.

05:00

🛠️ Introduction to Twint: A Twitter Scraping Tool

The speaker introduces a Python package called Twint, which allows users to scrape tweets and gather information about Twitter without using the official API. Twint bypasses the limitations imposed by the API, such as the number of tweets that can be accessed based on recency or time frame. The speaker provides a link to Twint's documentation on GitHub and offers a brief tutorial on how to install and use Twint in a terminal, including searching for tweets by keyword and viewing a specific user's tweets.

10:02

📊 Twint's Features and Practical Applications

This paragraph delves into the practical applications of Twint, demonstrating how it can be used to search for tweets by keyword, filter by popularity based on likes and retweets, and save the data to a pandas data frame or JSON file. The speaker provides examples of how to use Twint in the terminal and within a Python environment using Visual Studio Code, highlighting the flexibility and power of the tool for various use cases.

Mindmap

Keywords

💡Twitter API

Twitter API refers to the set of rules and protocols that allow developers to access Twitter's data and functionalities to build software applications. In the video, it is mentioned that Twitter is making significant changes to its API, which is causing discontent among developers due to the new pricing structure and limitations imposed on data access.

💡Pricing Model

The pricing model refers to the structure or system in which a company sets the prices for its products or services. In the context of the video, Twitter's new pricing model for its API is criticized for being costly and inaccessible for many developers, affecting their ability to create and maintain applications that rely on Twitter data.

💡Developers

Developers are professionals who create, maintain, and improve software applications. In the video, developers are the primary stakeholders affected by the changes in Twitter's API, as they rely on this service to build and operate their applications. Their dissatisfaction stems from the new limitations and high costs associated with the API.

💡Elon Musk

Elon Musk is an entrepreneur and CEO of multiple companies, including Tesla and SpaceX. In the context of the video, he is mentioned in relation to a Twitter account called 'Pepito the cat', which uses the Twitter API to post automated updates. Musk's potential response to provide a 'light right only' API for non-commercial bots producing good content is seen as a positive development.

💡twint

twint is a Python package that enables users to scrape data from Twitter without using the official Twitter API. It offers a way around the limitations and costs associated with the API by providing an alternative method to access Twitter data.

💡Open Source

Open source refers to a type of software licensing where the source code is made publicly available for anyone to view, use, modify, and distribute. The twint package mentioned in the video is an open-source tool hosted on GitHub, allowing developers to freely use and contribute to its development.

💡Python

Python is a high-level, interpreted programming language known for its readability and ease of use. In the context of the video, Python is the programming language used to write scripts and utilize the twint package for scraping Twitter data.

💡Data Scraping

Data scraping is the process of extracting data from websites or applications, often for the purpose of data analysis or information gathering. In the video, data scraping is discussed as a method to obtain Twitter data without using the official API, through the use of the twint package.

💡Limitations

Limitations refer to restrictions or constraints on the functionality or use of a system. In the video, limitations are associated with the Twitter API, such as the number of requests allowed per month and the time frame within which tweets can be accessed.

💡Pandas DataFrame

A Pandas DataFrame is a two-dimensional, size-mutable, and potentially heterogeneous tabular data structure with labeled axes (rows and columns) in the Python programming language. In the video, it is mentioned as a potential format for saving the data scraped from Twitter using the twint package.

💡JSON

JSON (JavaScript Object Notation) is a lightweight data interchange format that is easy for humans to read and write and for machines to parse and generate. In the context of the video, JSON is one of the formats in which Twitter data can be saved using the twint package.

Highlights

Twitter is implementing massive changes to its API, causing discontent among developers.

The new pricing model for the Twitter API is considered expensive and unattainable, potentially putting software using the API out of business.

A chart is referenced showing the total requests per month and the month-to-month price for different paid plans under the API usage.

Large apps or web apps with a significant user base are expected to fall into the higher pricing ranges of the API plans.

Elon Musk has responded to feedback about the API changes, proposing a free, read-only API for bots providing good content.

A tool called twint is introduced as an alternative to scraping Twitter data without using the API and avoiding its limitations.

Twinter is a Python package that bypasses Twitter API limitations, allowing users to pull tweets from as far back as the start of Twitter's existence.

Instructions are provided on how to install twint using the terminal and Python, assuming the user has Python already installed.

The tutorial demonstrates how to use twint in the terminal with different commands, such as searching for tweets by keyword and viewing a specific user's tweets.

The process of setting up twint in Visual Studio Code (VS Code) is outlined, including importing twint and configuring the twint object.

An example is given on how to run a search for tweets about 'pizza' using twint within a Python script in VS Code.

The ability to limit the number of tweets retrieved and to filter by minimum likes and retweets is highlighted as a feature of twint.

Twinter's capability to save the Twitter data scraping results to a pandas data frame or a JSON file is mentioned.

The speaker expresses willingness to answer questions about twint in the comments section.

A detailed guide on installing and using twint is provided, including troubleshooting tips for potential issues.

The impact of Twitter's API pricing changes on small developers and businesses is discussed, highlighting the need for cost-effective alternatives like twint.

Transcripts

Browse More Related Video

Get UNLIMITED Tweets by Python Without Twitter API

Scrape Twitter with 5 Lines of Code

How to get TWEETS by Python | Twitter API 2022

TWITTER SENTIMENT ANALYSIS (NLP) | Machine Learning Projects | GeeksforGeeks

Scraping Amazon With Python: Step-By-Step Guide

Bitcoin Sentiment Analysis Using Python & Twitter