From data stream to prediction: Deploying real-time ML pipelines on Kubernetes

This talk, presented by Carles Arnal from IBM, explores the development of a real-time machine learning pipeline using open source technologies on Kubernetes. It outlines an end-to-end pipeline for real-time text classification using Reddit data, featuring data ingestion validated against JSON schemas via Apicurio Registry and streamed through Apache Kafka. The speaker discusses processing the data in micro-batches with Apache Spark Structured Streaming, utilizing both a fine-tuned DistilRoBERTa transformer and a scikit-learn pipeline that incorporates TF-IDF, latent semantic analysis, and logistic regression for parallel inference. The session highlights the deployment of the system on Kubernetes, addressing schema validation, model agreement for prediction reliability, and the challenges of handling evolving data without labeled ground truth.