The Power of Chunking
At my first job, I accidentally deleted three months worth of capacity planning. Lucky me, it was in March 2020 and capacity planning for that year was obsolete soon enough1. I was refactoring large parts of the ETL pipeline used to ingest the capacity data from our SaaS solution. It looked something like this: async def fetch_records_and_update_metadata(mongo_client, records): archived_employees = await mongo_client.fetch_employees_async(archived=True) to_be_deleted = [ record for record in records # if record in archived_employees ] async for record in to_be_deleted: await delete_record(record) So how did I delete all records? You guessed right, I forgot to uncomment the line that made sure to only delete archived employees. ...