Ftrace

The first tool many driver developers reach for is printk. Probe does not run, print a line. Interrupts do not arrive, print a line. DMA does not move, print a line. User space cannot read data, print another line.

That works, but only up to a point:

• too many logs hide the real failure
• high-frequency logs slow the system down
• printing in interrupt or locked paths changes timing
• production images cannot keep verbose logs enabled
• an intermittent bug disappears after adding logs
• multi-instance devices are hard to distinguish

A better model is layered instrumentation: