2008-08-27

Detecting record structure in a file

As a signal processing kinda person, I'm well acquainted with the properties of the Fourier transform. It is usually thought of as a tool that has mostly to do with the discretized continuous time signals. But in the abstract, it's founded in shift-(in)variance. That makes it pretty much the ultimate tool for detecting cyclic structures, even if they're discrete at the origin.

I'm also a file/data format freak. A data representation enthusiast, if you can call anybody that. That makes me think, how about simply taking a long term FFT of a data file, in order to bring about any periodicity/record structure in it? Perhaps that sort of operation could serve as a reverse engineering tool when repetitive, record structures are present. And, looking into the Fourier statistics, perhaps it could even help calculate some sort of implicit measure of fractal dimension, which could help figure out the dimensionality of the record structure being looked at.