Enhancing Learning in Oil & Gas: SPE Podcast Analytics with Open-Source Tools

SPE has published more than a hundred technical podcast episodes, but there was no easy way to search them or find the right one to listen to. The author built a fully open-source pipeline that scrapes the audio, transcribes it with Whisper, mines the transcripts for topics and drivers of popularity, and serves recommendations through a LINE chatbot. The project shows how free tools can turn unstructured audio into a searchable learning resource, and it gives podcast producers practical guidance on what listeners respond to.

Key results

Episodes typically drew 1,000–2,000 downloads, with one outlier above 4,600 at the end of 2020. Production paused from 2020 to 2023
Episodes of about 25–35 minutes got the most downloads, while those under 15 or over 50 minutes did worse, which suggests about 30 minutes as the target length
The best download model was XGBoost with a Gamma loss, and transcript content was the most influential input
Technology was the most common theme, followed by drilling, production, reservoir and automation topics; episodes about real field operations drew the most interest

Approach

Compared web-scraping options and used the Chrome Web Scraper extension to collect episode audio and metadata
Transcribed episodes with OpenAI Whisper on free Kaggle and Colab GPUs, using ffmpeg for audio conversion
Mined the transcripts with NLTK preprocessing, TF-IDF, LDA topic modelling (5 topics, 1,000-term vocabulary) and silhouette-scored clustering against a curated petroleum-engineering keyword list
Trained supervised regression models to predict download counts, evaluated with Gamma deviance and MAE on cross-validation and hold-out data
Deployed a LINE chatbot recommender on a Flask backend exposed through Cloudflare Tunnel, with Google Apps Script for simple tasks; the code is published on GitHub

The problem

SPE's podcast library was not organised by topic, there was no recommendation system, and nobody had analysed what makes an episode popular.

Tools & methods

PythonChrome Web ScraperOpenAI WhisperffmpegKaggle / Google ColabNLTKscikit-learn (TF-IDF, LDA, K-means)XGBoostFlaskLINE Messaging APICloudflare TunnelGoogle Apps ScriptGitHub

Summary written for this site from the published paper. For figures, data and the full method, read the paper on OnePetro.