System Design Patterns
14. Checksum
Let's learn about checksum and its usage.
1 min read
Background
In a distributed system, while moving data between components, it is possible that the data fetched from a node may arrive corrupted. This corruption can occur because of faults in a storage device, network, software, etc. How can a distributed system ensure data integrity, so that the client receives an error instead of corrupt data?
Definition
Calculate a checksum and store it with data.
Solution
When a system is storing some data, it computes a checksum of the data, and stores the checksum with the data. When a client retrieves data, it verifies that the data it received from the server matches the checksum stored. If not, then the client can opt to retrieve that data from another replica.
Examples
HDFS and Chubby store the checksum of each file with the data.