Sai Ravela and Bill Freeman
April 12, 2006

This might be a good example of why to publish great results or ideas quickly, lest it become, as Peter Hagelstein coins the term, "stolen, fair and square." One of the ideas we had (and we aren't aware of prior ones--if you know of any please let us know so we can attribute) was to deform patches in example-based super-resolution, or in downscaling, which we applied to fluids. This enabled richer reconstructions, allowed smaller collections (or database of lowres-hires pairs), and much larger patches.

Shown above is a picture communicating the essential result. The query is a low-resolution patch and the rest (except one) are synthetically reconstructed high-resolution patches and your job is to figure out which one is the true high resolution patch. We were so impressed by the results of applying field alignment to super-resolution! Thanks of course to the growing internet and flattening universe, it is now possible to pull up such nuggets that weren't immediately shipped to a CVPR or ICCV in the usual mad rush that happens every few months, or even an ArXiV elsewhere, but to simply put it up and revise the overall record. The file annotation here from my disk says content created April 12, 2006, 7:44am, so that's what we'll date this work to, though the results could well have been many months prior.

Other work now seems to have appeared from the group, and elsewhere, so that if you are really interested in how our algorithm works, please send Sai an email. You can also cite this site if you wish.

From a  STICS perspective, Ensemble Downscaling is thus a great application of Field Alignment.

But how does it work?

The first step is to do low-resolution and high-resolution simulations. These simulations for example are a patch on the simulation of a passive tracer along the Antarctic Circumpolar Current in collaboration with John Marshall. The patches were simulated at two different diffusivities an order of magnitude apart, so that one field has rich structure and the other is rather smooth. We create a data based of these fields, say a 100x100 image for low-res and a 100x100 image for high res (the low-res will just be very blurry so to say).  We cut up these 100x100 images into moderate chunks, say 50x50 with some overlap, so that from a set of simulation snapshots, we can create a database; the examples of <low,high> vectors saved.

Now, when a new low-resolution simulation comes along, a snapshot at time t, we wish to produce plausible high resolution simulations. The idea is to produce ensemble members that are consistent in appearance, structure, and with a variance that we expect to see. The procedure involves cutting up the incoming low-res field into overlapping patches of 50x50, looking up nearest neighbor low-resolution patches in the database, retrieving their high resolution counter parts. Let us suppose that we are looking at one retrieved low-res patch matching a query patch. The two will in general have amplitude and deformation error, so that one can solve the joint field alignment problem between a retrieved and query patch. The deformation is then applied to the hi-res patch associated with the low-res retrieval and, further, the amplitude adjustment is downscaled as a random field and added to the deformed hi-res patch. Once this is done, we move to the next patch in the query which overlaps the previous one. We do this "copy-deform-paste" like operation again, but this time ensure that the boundaries between the two hi-res patches we are going to stitch together is seamless. And in this way, on we go, top-to-bottom, left to right, patch by patch stitching together a plausible hi-res realization of the low-res query. Since each query produces many retrievals, we can produce an ensemble of hi-res patches. That's just was done by me when working for Bill Freeman in 2005, building upon his example-based super-resolution work, but obviously incorporating deformable patches. It's of obvious value to fluids because fluids, if anything, deform. But what's most interesting is the combination of pattern search with fluid deformation. In the example on this page, can you identify the true high resolution patch? a beer if you do, send Sai an email with the answer.