Texture Space Browser for Remotely Sensed Imagery

Software Design
for a
Texture Space Browser
for
Remotely Sensed Imagery

Daniel X. Pape
dpape@canis.uiuc.edu
10/4/96
Version: 1.1

1.0 Introduction

This document describes the design of a texture space browser for remotely sensed imagery.

2.0 Assumptions and Conventions

2.1 Program Perspective

This program will be written as a stand-alone application to simplify development. Eventually, it is meant to be a part of the Interspace system architecture [8]. Specifically, it will be integrated into the searching and indexing system via a wrapper IU.

2.2 Implementation Language

This program will be implemented in the Smalltalk language using the Visualworks environment developed by ParcPlace-Digitalk, Inc. This will help with the eventual integration into the Interspace system as the Interspace is being implemented in Smalltalk as well.

This program will also be using the Visual Companion Object Developer's Kit [1] developed by Object/FX Corporation. The VCODK is a reusable Smalltalk framework which offers minimal GIS capabilities like visual data access and rudimentary spatial analysis.

2.3 Portability

This program will retain all the portability inherent in applications running under the VisualWorks Smalltalk Environment; this program will be able to run on UNIX-based machines, Macintosh computers, and Windows PCs.

2.4 Definitions

A number of terms need to be defined, so that their use in this document can be understood.

Concept space: A network of terms and their weighted associations for the underlying documents in the database.
Vocabulary switching: Searching for similar concepts across large databases of documents in different but related fields.
Texture: A specific pattern of different colored pixels in an image, usually within some defined area, like an 8x8 pixel square.
Texture space: A network of textures and their weighted associations for the underlying images in the database.
Texture switching: Searching for similar patterns of textures across different but related large image sets.
Remotely sensed imagery: Any images which were obtained with either active or passive digital sampling of the electromagnetic spectrum. For this document, this specifically means any images of the earth's surface taken by satellite or aerial cameras, although the program described here could be used for other kinds of images, like medical x-ray images or CAT scans.
GIS: Geographic Information System. A GIS consists mainly of two parts: a standard relational database which holds both quantitative (spatial) data, and qualitative (descriptive) data and a user interface which allows for the display of the above data on some kind of projection of the earth's surface.
Spatial resolution: The real-world size which a single pixel in a remotely sensed image represents. If a pixel represents a 20m by 20m area, then the spatial resolution is said to be 20m. Small spatial resolution usually denotes a zoomed in image. Large spatial resolution usually denotes a zoomed out image.

3.0 Program Description

3.1 Program Requirements

In previous research [5], a domain-specific thesaurus was automatically generated from a number of related scientific documents in the fields of molecular and genetic worm biology. This first required the development of a concept space for the database of documents. The methodology for this program will be adopted directly from that research.

The process of the generation will most likely include: image and texture collection, texture filtering and automatic indexing, and co-occurrence analysis of the collected textures.

3.1.1 Image and Texture Collection

In order to create a meaningful database of image textures, a number of images are needed. Later this summer a complete aerial photo collection of Arizona and California will be made available from the University of California at Santa Barbara, so there should be more than enough images to experiment on.

Small chunks (tiles) of the images, will be cut out and cataloged. This catalog will also contain other metadata, such as which image the texture chunk came from, and what the coordinates of the texture chunk were. The size of these chunks will have to determined experimentally, because there are a number of problems involved with choosing a chunking size without knowing beforehand the contents of the images. For example, as can be seen in Figure 1, if the spatial resolution of the image is very large, then an 8x8 pixel chunk of the image may incorporate more than one ground feature, and the resulting texture analysis will be skewed. On the other hand, if the spatial resolution of the image is very small, then an 8x8 pixel chunk may be so small that many redundant textures will be cataloged.

Figure 1

Analogous to collecting documents in a specific subject domain, I believe that all of the images must be of the same scale and resolution, and they must also be imagery from the same band of the electromagnetic spectrum (visible, infrared, microwave, etc.). This is to ensure that a specific texture in one image will likely correspond to the same real world ground feature as the same texture in another image.

3.1.2 Texture Filtering and Automatic Indexing

Another unknown aspect of this project is whether or not a way of labeling which textures correspond to what ground features will be made available. If this kind of a list can be made, then we can prune the original texture catalog and keep only the textures desired (i.e. only vegetation textures, or only urban textures). Currently, however, it appears that one of these lists will not be available, so analysis will have to be performed on the entire texture catalog.

Even without a texture/feature list, it is still possible to prune the texture catalog to save computation. Near-duplicate textures, within some user defined threshold, can be "eliminated." This alone could bring the texture catalog down to a fraction of the original size. Another possibility, depending on the interactivity of the texture collection process, could be the definition of "stop-textures," or textures which are known to be so common that they are not important. This would be useful for example, in defining a stop-texture for "surface water" for image sets of northern Minnesota or a stop-texture for "cornfield" for image sets of Iowa.

3.1.3 Co-occurrence Analysis

Again using Chen's methodology as a model, most likely co-occurrence analysis of the image textures will involve examining the texture frequency and image frequency for each texture in an image. The texture frequency, tf_ij, represents the number of occurrences of texture T_i in image i, and the image frequency, df_i, represents the number of images in a collection of n images in which term j occurs.

Then, the combined weights, d_ij, of texture T_j in image i, and d_ik, of texture T_k in image i, will be computed as follows:

where N represents the total number of images in the image set.

Next, tf_ijk will represent the number of occurrences of both T_j and T_k in image i, and df_jk represents the number of images (in a collection of N images) in which T_j and T_k occur together. Finally, the combined weight, d_ijk, of both T_j and T_k will be computed as follows:

Texture co-occurrence analysis will be performed based on the asymmetric "Cluster Function" developed by Chen and Lynch [2]. The following two equations indicate the similarity weights from T_j to T_k, and from T_k to T_j (first and second equations, respectively).

Also included in these equations is a WeightingFactor which was used to penalize terms which appeared in many documents while doing the co-occurrence analysis. I am not sure at this time if such a weighting factor will prove useful to co-occurrence analysis of image textures, since general textures will most likely be removed from the database during the texture filtering stage above.

3.1.4 Texture Thesauri

Using techniques developed in [6] and [7], a texture thesaurus will be generated for the set of images. It is this thesaurus, and other thesauri generated for other sets of images, and the metadata contained in the corresponding texture catalogs, that the texture space browser will use in order to function.

3.2 Functionality

3.2.1 Interface

The main texture space browser window may potentially look something like Figure 2. The important features of the browser window are the image display pane, the texture list pane, and the related-texture list pane.

Figure 2

Image display pane

The image display pane is the area where images and maps will be displayed. This pane will support various image operations such as: zoom in/out, view scrolling, view rotation, etc. Additionally, a single mouse click within the image will highlight the corresponding texture in the texture list.

Texture list pane

The texture list pane will be a complete list of iconic representations of all the textures present in the current image. This pane will be able to scroll up and down, and if a texture is clicked upon, the browser will search the texture thesaurus for related textures. These related textures will be displayed in the related-texture list pane below. The browser may also optionally highlight all the areas in the current image which are represented by the selected texture. An example of this is shown in Figure 3.

Figure 3

Related-texture list pane

The related-texture list pane will contain the related textures found for any selected texture in the texture list pane. If one of the listed textures is double-clicked, then another window will appear showing a list of available images which contain that texture. This window will give the user the ability to open and display any of these images. It will also stay open until the user closes it, so that the other images can be opened and examined if desired. If several of the textures in the related texture list are highlighted and double-clicked on, the window that appears will contain a list of available images which contain all of the highlighted textures.

3.2.2 Uses

The texture space browser will initially have four main uses: image browsing, intra-image texture searching, image set texture searching, and image set texture switching.

Image browsing

The browser will of course support simple image browsing - the user can open an image, look at it, and then close it.

Intra-image texture searching

There will be two different ways that the browser will support intra-image texture searching. First, the user can search for any one texture by clicking on the texture in the texture list pane. The image display pane can then highlight all the areas of the current image which correspond to that texture. This can also be done if the user selects multiple textures in the texture list pane. Second, the user can search for similar regions of groups of textures. This can be done by first selecting a texture from the texture list pane. This will automatically bring up a list of related textures in the related-textures list pane. If the user then selects one or more of these related textures, then the image display pane will highlight any areas on the image which have textures from the related-textures list pane in close proximity to the original texture from the texture list pane.

Image set texture searching

There will also be two different ways that the browser will support image set texture searching. They are analogous to the way in which it will support intra-image texture searching, except that instead of showing results by highlighting the current image, the browser will first show a list of available images which share the desired textures, and once an image is selected and opened, then the correct areas on the image will be highlighted.

Image set texture switching

In other research, Chen et al. [3] experimented with generating and integrating automatic thesauri. They proposed treating the thesauri as neural or semantic networks, and applying spreading activation algorithms for term-switching. Using these same methods, the browser should be able to take a set of user selected textures and find a matching (or closely related) set of textures in another set of images.

This may not work for any two random sets of images. The two sets may have to satisfy one of two constraints: they probably will either have to be different spectral types of images for the same geographic area (e.g., a set of near-infrared images and a set of middle-infrared images for Orange County in the state of California) or they will have to be the same spectral type of image for different geographic areas (e.g., two sets of visible light photographs, one for California and one for Arizona). The first is possible because images of the same land area taken in two different bands of the electromagnetic spectra should contain similar patterns of textures, even though the textures themselves are actually different. On the other hand, for images taken in the same spectra, different areas of the land should contain similar patterns of textures, simply because there will only be a finite number of total different textures, and the patterns are bound to repeat themselves.

4.0 Future Extensions

A number of possibilities for future extensions exist. Most of these extensions are just suggestions right now because I do not have enough information to determine if they are actually possible to implement.

First, in the texture collection phase, it may be possible to dynamically determine what the best pixel chunk size is. This way, for areas of an image with a lot of information, like within a city, the pixel chunk size can be made small, say 4x4 pixels. For areas of an image, however, with a low amount of information, like within a large lake, the pixel chunk size can be made large, say 32x32 pixels.

Second, the browser should eventually be able to support more complicated spatial GIS queries of the images through the use of the VCODK framework. Initially however, I will simply be using the framework for the image display and manipulation routines.

Third, when selecting multiple textures in either the textures list pane or the related-textures list pane, the browser should be able to support other Boolean operations other then just AND'ing the selections before searching the texture thesaurus.

Finally, it would be useful to be able to calculate some kind of metric for the textures present in a drag-selected region of an image. This would allow the browser to effectively classify this region, and then it could try to search the other image sets for regions matching the same classification.

References

[1] Object/FX Corporation, Visual Companion Object Developer's Kit - Developer's Guide, 1995.

[2] H. Chen and K. J. Lynch. Automatic construction of networks of concepts characterizing document databases. IEEE Transactions on Systems, Man and Cybernetics, 22(5):885-902, September/October 1992.

[3] H. Chen, K. J. Lynch, K. Basu, and D. T. Ng. Generating, integrating, and activating thesauri for concept-based document retrieval. IEEE EXPERT, Special Series on Artificial Intelligence in Text-based Information Systems, 8(2):25-34, April 1993.

[4] H. Chen, J. Martinez, A. Kirchoff, T. D. Ng, and B. R. Schatz. Alleviating Search Uncertainty Through concept Associations: Automatic Indexing, Co-occurrence Analysis, and Parallel Computing. October 1995.

[5] H. Chen, J. Martinez, T. D. Ng, and Bruce R. Schatz. A Concept Space Approach to Addressing the Vocabulary Problem in Scientific Information Retrieval: An Experiment on the Worm Community System. Nov. 1, 1995.

[6] H. Chen, B. R. Schatz, J. Martinez, and D. T. Ng. Generating a domain-specific thesaurus automatically: an experiment on FlyBase. In Center for Management of Information, College of Business and Public Administration, University of Arizona, Working Paper, CMI-WPS 94-02, 1994.

[7] H. Chen, B. R. Schatz, T. Yim, and D. Frye. Automatic thesaurus generation for an electronic community system. Journal of the American Society for Information Science, 46(3):175-193, April 1995.

[8] B. R. Schatz, and K. Powell. Interspace System Architecture. Internal Document, 2/26/96.

[9] Y. Q. Chen. Novel Techniques for Image Texture Classification. Doctoral Thesis, University of Southampton, March 1995.

Software Design for a Texture Space Browser for Remotely Sensed Imagery