Early Learnings from and Replacing CDC with Data Contracts - Interview w/ Andrew Jones

Provided as a free resource by DataStax https://www.datastax.com/products/datastax-astra?utm_source=DataMeshRadio (AstraDB) In this episode, Scott interviews Andrew Jones, Tech Lead of the Data Infrastructure Team at GoCardless. Andrew shares the story of how operational system changes kept breaking downstream data consumption (sound familiar?), especially using CDC. The software engineers couldn't easily use the CDC tooling and the data engineers could easily use the data from it either as CDC didn't structure the data for easy consumption. Andrew wasn't really sure how other people were handling taking the API contract concept and leveraging it for data but started building out some generic simple tooling to let consumers and producers feel somewhat comfortable with their data contracts. A big revelation was in helping data consumers make better asks for data. The data consumers weren't used to asking the producers for data, especially in a reliable and scalable way (sound familiar?). GoCardless now has an actual standard form for data consumers to use to request data and that is working quite well. GoCardless plans to completely remove their CDC architecture by 3Q of this year to replace with data contracts. They are focusing on providing tooling to give domains the autonomy to serve data to consumers in their own way. While it isn't data mesh, especially with the lack of interoperability between data products and lack of source/producer-aligned data products, it seems to be working for GoCardless thus far. Andrew's Medium post called Improving Data Quality with Data Contracts: https://medium.com/gocardless-tech/improving-data-quality-with-data-contracts-238041e35698 (https://medium.com/gocardless-tech/improving-data-quality-with-data-contracts-238041e35698) LinkedIn: https://www.linkedin.com/in/andrewrhysjones/ (https://www.linkedin.com/in/andrewrhysjones/) Twitter: @andrewrjones / https://twitter.com/andrewrjones (https://twitter.com/andrewrjones) Data Mesh Radio is hosted by Scott Hirleman. If y