Nathan Spelts

Creating Videos from Reddit Posts and Comments

2023-05-14T00:00:00Z

Introduction

In this blog post, we will explore an intriguing project that utilizes AI and various libraries to create engaging videos from Reddit posts and comments. The project’s main goal is to transform text-based content into visually appealing videos by combining Playwright for web scraping, moviepy for video editing, and SQLite for data management. Let’s dive into the details and discover what makes this project interesting and valuable.

Source code can be found here.

Project Overview

The main.py file serves as the entry point for the project. It imports necessary libraries and modules, such as dataclasses, os, asyncio, knapsack, shutil, and several custom modules. The script then defines a Content data class and initializes a content_dict dictionary to store content information.

The project’s workflow starts by connecting to the Reddit API using the Reddit wrapper module. It retrieves posts from a specific subreddit and iterates through each post. If a video for a particular post doesn’t already exist, the script proceeds to create content for that post.

The create_content_post() function generates audio and captures a screenshot of the post using the take_screenshot_post() function. The resulting content is stored as a Content object in the content_dict dictionary.

If the post’s audio duration is longer than 60 seconds, the script inserts the video into a database and removes the generated content directories. Otherwise, it proceeds to process the comments associated with the post.

For each comment, the script creates content using the create_content_comment() function, which generates audio and captures a screenshot of the comment using the take_screenshot_comment() function. The resulting content is stored in the content_dict dictionary, along with duration, upvotes, and comment IDs.

To optimize the selection of comments, the script uses the knapsack algorithm to find the best combination of comments based on duration and upvote count. Once the selection is made, it creates a video by combining the post content and selected comments using the create_video() function.

The video is then uploaded to YouTube using the get_authenticated_service() and initialize_upload() functions from the upload_video module. Finally, the generated video is saved to the output_video.mp4 file.

Other modules in the project include take_screenshot.py, which uses the Playwright library to capture screenshots of posts and comments on Reddit, and create_video.py, which handles the video creation process, including audio synthesis and video editing.

Key Features and Technologies:

Web Scraping with Playwright: The project leverages Playwright to scrape Reddit posts and comments and capture their visual representation.
Text-to-Speech (TTS) Synthesis: The TTS library is used to convert text content into synthesized audio, enhancing the user experience when watching the videos.
Video Editing with Moviepy: Moviepy, a Python library for video editing, enables the combination of audio, images, and video clips to create compelling and dynamic video content.
Knapsack Algorithm: The knapsack algorithm is employed to select the most suitable comments based on their duration and upvote count, ensuring an optimal combination for the final video.
SQLite Database: The project utilizes an SQLite database to store video IDs, ensuring that duplicate videos are not generated for the same Reddit post.

Conclusion:

The project presented here showcases the integration of various technologies and libraries to automate the process of transforming text-based Reddit content into visually engaging videos. By leveraging AI, web scraping, text-to-speech synthesis, video editing, and database management, the project demonstrates the power of Python for content creation and automation.

The ability to convert text content into multimedia videos not only enhances the user experience but also opens up opportunities for content creators to repurpose existing textual content and reach a wider audience through visual storytelling.

With further refinement and customization, this project could be expanded to support different platforms, integrate additional AI capabilities such as sentiment analysis or image recognition, and provide more advanced video editing features.

ASR Capstone at Seasalt AI

2023-04-11T00:00:00Z

It’s been nearly a year since my last post, but I’m excited to share details about my latest project! As I approach the end of my senior year at WSU, I’ve spent the entire year working on a capstone project in collaboration with Seasalt AI. Our work has focused on two tasks: developing an Indonesian model from scratch during the first semester, and retraining Seasalt AI’s existing Spanish model this semester.

Indonesian Model

The process of building a model from scratch began with locating an open-source corpus containing segmented audio files and time-stamped captions. We utilized this corpus to train our initial model using Kaldi, an open-source ASR project. The training pipeline involves normalizing the data, extracting mel frequency cepstral coefficients (MFCC) to facilitate approximation and save space, and generating statistical models to make assumptions about the data. Subsequently, the data undergoes monophone training, which examines individual phonemes, followed by triphone training that considers neighboring phonemes for context. Additional steps include speaker normalization, adapting the model to a specific individual’s voice, and reducing the impact of background noise. Lastly, the model is fine-tuned via time-delayed neural network training, resulting in a baseline Indonesian speech-to-text model.

Spanish Model

Data Crawling

For the Spanish model, we began with a provided baseline model. Our primary task was to gather data and use it to retrain this baseline model. We downloaded thousands of hours of YouTube audio and captions using the youtube-dl Python library, with our chosen keywords guiding the crawling process. We developed a script that took these keywords, located videos with Spanish captions, and downloaded all videos from channels containing those keywords. This data was then converted into a Kaldi-compatible dataset.

Data Validation

We proceeded to validate the data using our model. This involved creating smaller datasets, each containing around 50 hours of audio, and employing the model to confirm the presence of actual Spanish audio. The model assessed the word error rate between the audio and existing captions. The outcome was a condensed list of segments and Kaldi data suitable for retraining the model.

Model Retraining

Finally, we used the validated data to retrain the baseline model. This relatively straightforward process involved running scripts to enhance the model using the validated data.

Conclusion

This project has been an invaluable learning experience in machine learning, speech recognition, and working within a corporate environment. It has significantly contributed to my growth as a programmer, honing my skills in Python, Bash, and command-line operations.

CANG Physics Simulator

2022-05-02T00:00:00Z

Welcome to my first post on my new blog! I’m just about finished up with my junior year in the computer science sequence at Washington State University, and I wanted to talk about my semester project I just presented.

We were allowed to choose any project we wanted, as long as the scope was large enough. So I convinced some of my friends to create something akin to powder game. I always loved that game as a kid, and was also always curious how it worked under the hood. I’ve got a lot of interest in low level graphics rendering as well, mainly due to this video where jdh creates minecraft using C and OpenGL in 48 hours. So naturally, I convinced everyone to use OpenGL for our physics simulator. It felt like we bit off more than we could chew at times, but ultimately the project turned out really cool and I’m extremely proud of it.

Time for some technical details for those interested.

My biggest contributions to the project were:

Creating the infrastructure for drawing things to the screen
Liquid, Gas, and Solid collisions
State transitions

Drawing infrastructure

This part was extremely tricky and completely new territory for me. I’d experimented with drawing single triangles to the screen, and rendering some 3d shapes.

But I’d never attempted anything where new objects were being drawn on the fly. Thankfully, on the site where I’ve learned everything I know about OpenGL, there’s a tutorial on how to create breakout! So I followed the tutorial and extracted the information I needed to accomplish what I wanted to. Unfortunately this code I got from the tutorial didn’t count towards my line requirement on the project, but I learned a ton so the time wasn’t entirely wasted.

Liquid, Gas, and Solid collisions

This part of the project I can say with confidence that I thought through it and implemented it all on my own. Unfortunately that means making a lot of mistakes along the way. I’m going to skip all the things I tried that didn’t work, and get straight to what did work. I ended up creating a double array of entities, an entity being a number that is unique to a particle drawn on the screen. Each entity stores its own components that determine how it should behave. I created a liquid component, gas component, gravity component etc.

Then I wrote rules for what should happen to an entity with a specific component, like the gravity component for example. When a particle has the gravity component, and there’s not a valid entity in the space below it, it moves down. But if the particle encounters another particle with the border component, it stops moving down. The grid allowed me to check the immediate surroundings of a particle wherever it was, which kept things from getting bogged down too much. From there I started writing a liquid algorithm.

I tried many things to get realistic behaving water, and what I found to be the most effective was random movement. Every water particle chooses a random number between 0 and 99, if that number is odd it moves left, if it’s even it moves right. The amount the particle moves is dependent on the size of the number chosen. As simple as this is, it took a long time to figure out!

Steam was fairly simple to figure out, it chooses a random number similarly to water and slowly moves to the left or right as it ascends. Once it reaches a solid object, there’s a probability that it will condense back into water every few seconds.

State Transitions

This part was my favorite because it was so simple. Once I had collisions and drawing particles all set up, turning one entity into another was as simple as taking away one component and adding another. There’s water touching lava? Turn the lava into stone and turn the water into steam! The possibilities for things that could be added are endless. Of course, by the time I got into the stage where I was just adding fun things, I ran out of time unfortunately. But I managed to add water, ice, steam, lava, fire, and stone. I also added birds as my last contribution to the project. I was able to implement them via Boid’s algorithm, courtesy of Ben Eater.

I had a lot of fun with this project and learned a lot about working with others. I feel like I didn’t get too technical with my explanations. If you think it was too technical or not technical enough, go ahead and send me an email at nate at nathanspelts dot com and tell me about it! Even if you don’t have a specific comment, I’d love to hear that someone actually read this and enjoyed it! Thanks for your time. :)