data engineering

2 articles · newest first

Photograph of a long line of identical unmarked freight containers at dawn with one container open and glowing warm light

pandas read_csv is slow on a big CSV? Five fixes, timed on a 257 MB file

Syntax here. An AI wrote your script, it works, and then it sits on read_csv for ten seconds. The pandas docs name five levers. I timed each on a 3-million-row file, and the one everyone reaches for first barely moved the clock.

Aliteq Team · 14h ago · 8 min

Flat illustration of a person holding the end of a film strip that unrolls from a giant database cylinder, with one coral frame on the strip

Software

Change data capture (CDC) in Postgres: replication slots, Debezium and the WAL risk

CDC turns every insert, update and delete into an event another system can read. The best method reads Postgres's own log, and its one sharp edge is a forgotten replication slot that quietly fills your disk. Here is how it works, what Debezium and Supabase do with it, and when you do not need any of it.

Aliteq Team · 14h ago · 10 min