Multivariate Data Analysis
Previously known as multivariate statistical analysis

There are essentially only four steps here:

1. Low-pass filtration
2. Alignment in two dimensions
3. Dimension-reduction -- expression of a mxn image using only a few terms, i.e., eigenvectors
4. Classification

The low-pass filtration is optional, but if you plan to look at individual particles, this step will help.

For the classification below to be sensible, the images will need to have been aligned. The alignment step here is optional if the images have been aligned already.

The dimension-reduction step is even optional, in theory. In principle, one could classify the raw images (which is what SPIDER operation 'AP C' does). As an example here, I'm using correspondence analysis for the dimension-reduction. A similar method is principal-component analysis (PCA); to run PCA, one needs to change an option under SPIDER operation 'CA S' (here, in the batch file ca-pca.spi).

For classification, there are three methods illustrated here: Diday's method, Ward's method, and K-means. The individual classification operations are described in more depth in the classification tutorial.

Getting started

• Download and unpackage the tarball containing the batch files
  The files will be written to the current directory, so create a new one, if so desired.

Procedure

1. Low-pass filtration
   • BATCH FILE: filtershrink.spi
   • INPUT PARAMETERS: filter radii, decimation factor
   • INPUTS: selection file, unfiltered particles
   If you don't have a selection file, run mkfilenums.py (W.T.B. & T.R.S.), substituting the appropriate filenames:
   mkfilenums.py   listparticles.dat   win/ser*.dat
   • OUTPUTS: filtered particles

2. Reference-free alignment. -- choose one of these two options:
   1. Using 'AP SR'
      • BATCH FILE: apsr4class.spi
      • INPUT PARAMETER: object diameter (pixels, after decimation)
      • INPUTS: unaligned particles, selection file
      • OUTPUTS: aligned particles, averages

        There may to be a memory limit in 'AP SR'. If you get a core dump, truncate the selection file and try again.

   2. Using pairwise alignment
      • BATCH FILE: pairwise.spi
      • INPUT PARAMETER: object diameter (in pixels, after decimation)
      • INPUTS: unaligned particles, selection file
      • OUTPUTS: aligned particles, averages

        Conceptually, this alignment first aligns pairs of images and averages them. Then, it aligns pairs of averages of those pairs and averages them, and so forth. This type of alignment appears to be less random than does 'AP SR', which chooses seed images as alignment references.

        Reference: Marco S, Chagoyen M, de la Fraga LG, Carazo JM, Carrascosa JL (1996) Ultramicroscopy 66: 5-10.

3. Dimension-reduction
   • 1. (optional) Create a custom mask for your particle. The alternative is that a circular mask will be used below.
   • BATCH FILE: custommask.spi
   • PARAMETERS:
   • Fourier filter radius for input image
   • first threshold, in units of standard deviations
     That is, the threshold will be set to the average intensity plus some number times the standard deviation.
   • Fourier filter radius for the thresholded image
     The first thresholded image is often too jagged, so this filtration serves to smooth it.
   • second threshold, for the filtered mask, between 0 and 1
     The lower the number, the more generous the mask will be.
   • INPUT: average image, such as pairwise/rfreeavg001
   • OUTPUT: stkmask, whose slices are as follows:
   1. filtered image
   2. thresholded image
   3. filtered mask
   4. final mask
   5. mask-multipled image
   6. inverted mask
   7. inverted mask-multiplied image
   
   You may need to fiddle with the parameters to get a nice mask.

   slices of the custom-made mask stack file

   • 2. Run correspondence analysis or principal component analysis
   • BATCH FILE: ca-pca.spi
   • uses Python script eigendoc.py (J.S.L.) and Gnuplot script ploteigen.gnu (J.S.L. & T.R.S.)
   • INPUT PARAMETER: number of eigenfactors to calculate (more than 99 will require some user modification)
   • INPUT: aligned particles
   • OUTPUTS: eigenvalue histogram, eigenimages, factormaps, reconstituted images
     

     To switch to PCA (or iterative PCA), change the register x28 in ca-pca.spi to 2 (PCA) or 3 (iterative PCA).

     After running, examine the eigenimages and decide which ones to use. Typically all but the first few are noisy. If not, increase the number of eigenfactors to calculate, and re-run this batch file.

   eigenvalue histogram (click to enlarge)

   factormap (click to enlarge)

4. Classification -- choose one of three options:
   1. Diday's method, using 'CL CLA' -- I hear that this method works exceedingly well. In practice though, I find that I have limited control over the number of classes, which may or may not be a problem depending on the application. Also, I sometimes get errors with large data sets with this method.
      • BATCH FILE: cluster.spi
        • INPUT PARAMETER: number of eigenfactors to use
        • OUTPUT: dendrograms (PostScript and SPIDER formats)

          After running, decide how many classes to include. using WEB/ JWEB (Commands -> Dendrogram) and clicking on Show averaged images.

        dendrogram PostScript format (click to enlarge)

      • BATCH FILE: classavg.spi
        • INPUT PARAMETER: desired number of classes
        • OUTPUT: class averages

   2. Ward's method, using 'CL HC' -- The advantage is that, unlike Diday's method above, the dendrogram branches to any desired number of classes, down in size to individual particles. The disadvantage is that the dendrogram is unreadable if there are so many branches. You can truncate the dendrogram in WEB/JWEB as described below.
      • BATCH FILE: hierarchical.spi
        • INPUT PARAMETER: number of eigenfactors to use
        • OUTPUT: dendrograms (PostScript and SPIDER formats)

          After running, decide how many classes to use. The PostScript file may be highly branched, and nodes may be unreadable.

          Untruncated dendrogram (click to enlarge bottom row)

          The SPIDER-format dendrogram document can be viewed with WEB/JWEB and truncated. In WEB, go to Commands -> Dendrogram (example). In JWEB, go to File -> Open SPIDER Document File.

          Dendrogram in X-Window WEB (click to enlarge)

      • BATCH FILE: classavg.spi
        • INPUT PARAMETER: desired number of classes
        • OUTPUT: class averages

   3. K-means classification, using 'CL KM' -- The primary input is the number of classes to divide the particles into.
      • BATCH FILE: kmeans.spi
      • INPUT PARAMETERS: number of eigenfactors, number of classes
      • OUTPUT: class averages

        It can be informative to look at the individual particles from a class. You can use WEB/ JWEB, or montagefromdoc.py.
        Usage:
        ./montagefromdoc.py   KM/docclass001.dat
        If you have requested too many classes, there will be similar-looking class averages. If you have requested too few, there will be dissimilar particles within a class.

Miscellaneous tools:

• There is a Python utility, classavg.py, that upon clicking on a class average displays the constituent individual particles.
• Binary tree -- It is often not clear where to truncate the dendrogram. In X-Window WEB, one only sees the terminal nodes in the dendrogram averaged. (In JWEB, averaged images in not implemented at the time of this writing, although Bill Rice says that if the prefix is two characters long, it works.)
  • BATCH FILE: binarytree.spi
    • SUBROUTINES: averagenode.spi, update_lut.spi
    • INPUT PARAMETER: tree depth (number of averages will be (2**depth - 1))
    • INPUTS: dendrogram (from hierarchical.spi), images
    • OUTPUTS: averages (unlabeled and labeled)
  • Visualize the output using tree.py. (Requires Spiderutils.py, part of the SPIRE installation.) Syntax:
    tree.py    labeled001.dat    4    2    1024
    where:
    • labeled001.dat is an example filename (without a wild card)
    • 4 (optional) is the tree depth (default is 6)
    • 2 (optional) is the margin width (default is 2)
    • 1024 (optional) is the canvas width
  • If Spiderutils.py is not installed, try tree.spi. The output is a SPIDER-format image. However, the file size may be very large.
    • INPUT PARAMETERS: tree depth (number of averages will be (2**depth - 1))
    • INPUTS: averages from binarytree.spi
    • OUTPUTS: SPIDER-format tree image

    tree.py, depth=4 (click to enlarge)

    tree.spi, depth=4 (click to enlarge)

Source: techs/MSA/index.html     Page updated: 4/13/12     Tanvir Shaikh