[Linux] "Are those logs really at the same time?" Using clockdiff to expose server time differences in milliseconds

Have you ever experienced this while investigating failures in a distributed system? "The web server error log and the DB server slow query log. The timestamps are off, and I can't figure out the causal relationship..."

Even if NTP (time synchronization) is configured, time can temporarily drift during high load or network failures. In such cases, the command that can instantly measure the "time difference (delta)" with a target server is clockdiff, which I will introduce here.

A relative of Ping? The power of "ICMP TIMESTAMP"

clockdiff is similar to ping, which is used for network connectivity checks, but it works a bit differently. Instead of a regular Ping (Echo Request), it sends a special packet called "ICMP TIMESTAMP" to the target and calculates and returns the difference between the "send time" and "receive time".

Since it displays something like delta=5ms with a single command, you can instantly judge, "Oh, this server is 5 milliseconds behind."

How to handle cases where it "doesn't go through," explained on the blog

It is a very convenient command, but in modern security environments, it can sometimes be tricky. On my technical blog, I explained the following points.

• The permission barrier: Why sudo is required and why it cannot be executed by a regular user, as it handles raw sockets.

• The firewall barrier: Regarding ICMP Timestamp packet blocking settings, which cause "Ping works but clockdiff does not (times out)."

• IP options: How to use the -o flag, which utilizes IP header options when regular ICMP is blocked.

• Real-time monitoring: A technique to visualize how time is gradually synchronized after an NTP restart (how the delta approaches 0) by combining it with the watch command.

A detective tool for solving the mystery of "log times not matching." For when the time comes, please be sure to check how to use it in the article below.