Reading Social Media into Data: Manually, through JSON, and through R

James Cook from the University of Maine at Augusta introduces the concept of analyzing social media. He discusses the idea of breaking down social media into its constituent parts such as individuals, relations, affiliations, and content meaning. He explains the process of understanding how these pieces fit together and provides a walkthrough of analyzing a live social media platform, using Reddit as an example.

James Cook continues his discussion on social media analysis by diving deeper into Reddit's structure. He explains how Reddit is organized by subject and subreddits, and how users interact through discussions. He highlights the public nature of Reddit and the ethical considerations of working with public data. Cook also touches on the limitations of manually copying and pasting data from Reddit and suggests the idea of directly accessing Reddit's database, which is not possible due to privacy and security concerns.

The speaker shifts focus to WordPress, a platform for creating social media sites, to illustrate the backend of social media data management. He explains the structure of a MySQL database in PHP, using a hypothetical example of a criminology course. Cook outlines how posts and comments are stored in a database, emphasizing the complexity of social media data and the need for structured organization.

James Cook discusses the use of JSON (JavaScript Object Notation) as a text-based data format for transmitting data over the internet. He explains the simplicity of JSON compared to tabular databases and how it is designed for computer readability. Cook demonstrates the use of APIs to access data from websites like 'catfat.ninja' and shows the process of converting JSON data into a more readable format using R programming.

The speaker provides a step-by-step guide on how to use R and its libraries to analyze JSON data. He explains the process of installing and using packages like HTTR and jsonlite to fetch and convert JSON data into a readable format. Cook illustrates the process with an example of fetching data from a catfact API and converting it into a dataset that can be analyzed.

James Cook encounters a challenge when attempting to fetch data from the Reddit API. He explains that Reddit limits the number of requests to prevent abuse and maintain website functionality. This limitation, known as a '429 Too Many Requests' error, prevents the straightforward conversion of API data into a usable format.

To overcome the limitations of the Reddit API, James Cook introduces the 'Reddit Extractor' package in R. This package is specifically designed for working with the Reddit API and converting data into readable datasets. Cook demonstrates how to extract thread URLs and content from the 'introverts' and 'extroverts' subreddits, and how to save this data as CSV files for further analysis.

The speaker concludes by showcasing the organized datasets of posts and comments from the 'introverts' and 'extroverts' subreddits. He demonstrates how to load these datasets into Microsoft Excel for analysis, highlighting the relationships between different variables such as the number of comments, scores, and content. Cook emphasizes the importance of data organization in conducting research and analysis to understand patterns and themes in social media.