SEGMENTATION AND MODELING OF TOMOGRAPHIC VOLUMES

Introduction

These are the methods currently used at Albany for visualization of tomographic volumes.

First, the individual objects of interest in the volume are segmented using Sterecon, which results in a stack of contours for each object or class of objects.

Next, the contour stacks are processed by any of three methods, which make Inventor-format "geometry" files. The methods are: (1) converting the contours directly to Inventor format using "ssrtoiv", (2) tiling of objects using Iris Explorer's "IsosurfaceLat" after creation of SPIDER volumes from the contour stacks, and (3) direct tiling of objects based on contour stacks, using MOVIE.BYU's "tiling" or "sheet", followed by "byutoexp". The choice of method depends on the nature of the object, as described below.

Then, a model or scene is made, which can contain objects created using any combination of the three methods. The final 3-D model is surface-rendered and displayed using Iris Explorer's "Render" or SGI's "SceneViewer".

Finally, the 3-D model can be converted to VRML format for web-based interactive display using "Cosmoplayer", or animation frames from a rotation or translation series can be made into a movie or video.

Currently only the SGI workstations have all the software used here.

Updated 03/04/00, M. Marko

Preliminary steps

Step 1. Prepare the tomographic volume for slice tracing.

Step 2. Trace contours on slices using Sterecon.

Method 1

Step 3. Convert Sterecon contours directly to Inventor format.

Method 2

Step 4. Transfer contours of individual objects or classes to SPIDER.

Step 5. Create SPIDER volumes of individual objects or classes.

Step 6. Smooth and cap SPIDER volumes.

Step 7. Tile SPIDER volumes using Explorer's "IsosurfaceLat".

Step 8. Adjust the translation and scale of Inventor geometry files.

Method 3

Step 9. Transfer Sterecon contour stacks to MOVIE.BYU format.

Step 10. Tile objects using "tiling" or "sheet"

Step 11. Smooth MOVIE.BYU-format objects.

Step 12. Convert MOVIE.BYU-format objects to Inventor geometry files

Final steps

Step 13. Reduce the number of tiles in Inventor geometry files.

Step 14. Smooth Inventor geometry files.

Step 15. Combine objects and make the final surface-rendered model.

Step 16. Determine surface area from models.

Step 17. Make presentations from models.

Step 18. Miscellaneous tips

Before doing this first step, the volume should be windowed on all sides to just the area to be traced. Be careful, however, not to cut off too many top and bottom z-slices for spherical objects, since the tomographic "missing wedge/pyramid" effect makes the top and bottom of spheres difficult to recognize. The contrast should be adjusted for optimum viewing, using the SPIDER operation TH, for example.

In b80.dat, the number of z-slices in the reconstruction is reduced by half by adding together adjacent slices. This reduces the number of slices to be traced, and is justified since the z-resolution of a typical tomographic reconstruction is less than the x-y resolution by about a factor of about three. The slices are made into individual images, with double the pixel size, to make it easier to trace fine details. A series of slice images, in SGI (-.rgb) format are also made, which are to be made into a "movie" by using the program movieplayer_old. This movie is essential as an aid to assigning correct connectivity when tracing with Sterecon. Alternatively, the tracing can be done in stereo, which also helps with connectivity questions. In that case, the stand-alone routine tostereo is used instead of b80.dat to prepare the volume for tracing. See M. Marko or A. Leith for advice on using tostereo.

You will need to edit b80.dat according to the comments written on the appropriate lines. Only the name of the input volume needs to be edited.

2. Trace contours on slices using Sterecon.

input files

sli***.dat [slice images]

output file

(e.g.) recon1.ssr [Sterecon reconstruction file, -.ssr]

An appropriate tracing strategy is essential for a good final model, so you will benefit from the advice of an experienced staff member who is familiar with your specimen. Care must be taken when assigning class names and contour numbers so that individual objects can be handled conveniently during the visualization process.

It is very important to be sure of the pixel size and slice thickness, taking into account step 1, in which the images were enlarged and the slices made thicker. This is essential if measurements are to be made, and also to allow combining the different methods of contour-processing to be used in the same final display. The questions asked by Sterecon are as follows:

Enter section or view thickness (in microns):

Enter pixels/micron:

For tomography, it is convenient to work in units of nm, so just assume the word "micron" has been replaced with "nm". For section thickness, use twice the pixel size of the tomographic volume used as input for step 1, since every two slices were added together. For pixels/micron, use one-half the pixel size, since the traced image was enlarged two times. Then take the reciprocal, since you are asked for pixels/nm, not nm/pixel.

Method 1

This method is suitable as the final display method for objects which are very small within the whole model, such as particles or filaments. Since surface-rendering is not practical for these objects, the contours themselves are used in the final display. However, note that circles entered in Sterecon using the "cir*" class can be surface-rendered as spheres after using this method.

This method is also useful for having a quick look at the contours you traced, before making a more sophisticated model. The stand-alone program "ssrtoiv" is used to directly convert Sterecon contours to Inventor format. This allows the data to be displayed using SceneViewer or Render (see step 15), providing convenient 3-D manipulation of the contour-stack model. The colors you assign in Sterecon's display function (DIS: COD) and any filling of contours (DIS: POL, FIL), will be retained in the Inventor file.

3. Convert Sterecon contours directly to Inventor format.

input file

(e.g.) recon1.ssr [Sterecon file, -.ssr]

output file

(e.g.) recon1.iv [Inventor file, -.iv]

First use the Sterecon display function (DIS: DIS) to check the selection of contours you want included in the model. The selection is done using the contour name specification. Also check that the color and filling options are correct. These options cannot be changed when making the final 3-D display. Use "ssrtoiv" as follows:

Method 2

This method is best used when the object to be modeled is complicated, branched, and/or highly-detailed. A good example is mitochondrial cristae. All five steps are usually carried out separately for each individual object. However, a class of objects can be handled at once if all the objects in the class will always be displayed at once in the final model, with the same color and transparency designation for all the objects.

This method only works with closed contours. For objects drawn with contours which do not form a continuous loop, use method 3.

4. Transfer contours of individual objects or classes to SPIDER.

input file

(e.g.) recon1.ssr [Sterecon file, -.ssr]

output files

DOC***.dat [SPIDER coordinates files]

The first step is to make a SPIDER document file for each slice containing a contour of the object to be modeled. These document files are simply a list, in ASCII form, of the x and y coordinates of all the points you entered when tracing the contours.

Use Sterecon, INTerface menu (sample responses shown in red):

INTerface menu (enter MEN to see choices).

Enter choice: doc

Enter series name: recon1

FILE OPENED: recon1.ssr

Enter doc file name template (eg. DOC***): DOC***

Enter first doc file number: 1

(Always use 1 for this application.)

Do you want to OK each contour transfer? (Y/N/MIR/CIR): n

FOR SPIDER <PP LL> USE SCALED OUTPUT

Output pixel, metric, unscaled, or scaled data? (P/M/U/S): s

(It is important to choose "s" here so all the objects will match in the final display.)

X & Y dimensions used in -PP LL- (Default: 280): 507 507

(The x-y dimensions you use here should be the original x-y dimensions of the tomographic volume you used as input for step 1.)

DEFAULT OFFSET WILL SET MINIMA TO 1.0

USE 0,0 FOR SPIDER -PP LL- INPUT

Enter X and Y offset (OR <CR> for -XMIN +1,-YMIN +1): 0 0

Enter contour name: *ommm*

(Note that the desired object is selected using the Sterecon contour name.)

(The following lines will be shown on the screen. It is very important to save the first two lines for later use by pasting them into a text file. In step 8, these lines will be pasted into the geometry file for the surface-rendered model.)

Transform { translation -247.0 -240.6 40.28

; scaleFactor 0.8633 0.8633 2.120 }

Transfer more contours? (Y/N): n

(Then you will see a list of all the contours transferred.)

NOTE: This step will fail if there are any pre-existing files with the form DOC***.dat. Therefore, it is best to go through the whole procedure, steps 4-8, for one object or class at a time, deleting the intermediate files when the final Inventor file is satisfactory.

5. Create SPIDER volumes of individual objects or classes. (b81.dat)

input files

DOC***.dat [SPIDER coordinates files]

output file

(e.g.) ommmvol.dat [SPIDER volume]

The document (DOC***.dat) files are now converted into SPIDER volumes. The procedure file does this by creating an image for each slice. The image contains the contour lines you drew. The contours are then filled in with a pixel value of 10, and the background pixel value is set to zero. All the slices are then stacked into a SPIDER volume. The filling step requires closed contours, which restricts the use of this method.

You will need to edit b81.dat according to the comments written on the appropriate lines. The size of the image, number of DOC files, and name of the output volume will have to be edited. It is very important that you use the same image size as was used in step 4, which would normally be the original x-y dimensions of the tomographic volume you used as input for step 1.

The procedure file will delete all intermediate files after the output volume is made. However, if the procedure file fails, and these files are not deleted, the (corrected) procedure file will not run again. If this happens, you must do the following:

/rm *.tmp *.tem *.stk

6. Smooth and cap individual SPIDER volumes. (b82.dat, b83.dat)

(Smoothing only)

input file

(e.g.) ommmvol.dat [original SPIDER volume]

output file

(e.g.) ommmvol_s.dat [smoothed SPIDER volume]

Surface-rendered models made from stacks of images will sometimes be rough-looking, with striations parallel to the slices. Low-pass filtering of the SPIDER volumes prior to tiling can improve the appearance of the model, but one should be careful that desired details are not obscured. The SPIDER BC command is used for smoothing. A filter box size of 5x5x5 gives about the maximum smoothing that should be used; sometimes 3x3x3 is more appropriate. Use b82.dat for smoothing. In the procedure file, you will need to edit the input and output file names and the filter box size.

(Smoothing and capping)

input file

(e.g.) ommmvol.dat [original SPIDER volume]

output file

(e.g.) ommmvol_c.dat [smoothed SPIDER volume]

The surface of any contour which exists in the top or bottom slice of the SPIDER volume will not be tiled in the model of the object unless the volume is padded with an extra pixel above and below. Without the padding, the top and bottom of the object will be open. In fact, this is often desired. If you want the model to be both smooth and capped, use b83.dat. In the procedure file, you will need to edit the input and output file names and the filter box size.

7. Tile SPIDER volumes using Explorer's "IsosurfaceLat".

input file

(e.g.) ommmvol_c.dat [smoothed SPIDER volume]

output file

(e.g.) ommm.geo [Inventor geometry file, -.geo]

This step is computationally-intensive, so it should be run on computer where the data is kept. "IsosurfaceLat" creates a tiled surface which follows outer pixels which have values at or above a set threshold. It uses an algorithm similar to "marching cubes" to do this. The number of tiles created this way is very large, and the Inventor files are large and hard to display quickly. Thus, it is advisable to reduce the number of tiles in step 13 before making the final display.

Use Iris Explorer (type "explorer") and open the map file "create.map", which you can download from here.

Maximize all three modules by clicking on the square in the upper right of each small box.

Set the min and max in "IsosurfaceLat" to 0 and 10. You might have to close and re-maximize "IsosurfaceLat" to get the min and max boxes to appear.Then set the threshold dial to about 9 o'clock. Increasing the threshold makes objects thinner. You can experiment with turning smooth on.

Make the Render window about 1/4 the size of the computer screen and click on the small box icon, the bottom one in the row along the right side. This selects for an orthographic view.

Delete the existing filename on the bottom line of "Trans3Dimage" and type in the desired SPIDER volume. "Trans3Dimage" is an Explorer module written at Albany to allow input of SPIDER volumes. The whole process will start as soon as you press return. The top of each box will turn yellow when that process is running.

When the display of the object in the Render window is finished, save the tiled Inventor file using "file>save as". Our convention is to use the extention ".geo" for these files. The Render window shows a surface-rendered version of the object as default. If you want to see the actual tiles, right-click on the black background and select "wire frame".

8. Adjust the translation and scale of tiled objects.

To make all the tiled objects fit together correctly in the final display, the two lines of text you saved in step 4 have to be pasted into the "-.geo" Inventor files. Since the Inventor format is ASCII text, the files can be opened using an editor such as "jot". You will have saved the corresponding set of lines for each Inventor file.

The beginning of a typical "-.geo" file looks like this:

#Inventor V2.1 ascii

Info {

string "Explorer Render"

}

ShapeHints {

vertexOrdering COUNTERCLOCKWISE

creaseAngle 0

}

Here are the lines you need to paste into the file:

Transform { translation -247.0 -240.6 40.28

; scaleFactor 0.8633 0.8633 2.120 }

Here is what the beginning of the file looks like after the paste. Note that you have to remove the ";" and move the new lines over one space. The presence and position of the "{" and "}" characters isvery important.

#Inventor V2.1 ascii

Info {

string "Explorer Render"

}

Transform { translation -247.0 -240.6 40.28

scaleFactor 0.8633 0.8633 2.120 }

ShapeHints {

vertexOrdering COUNTERCLOCKWISE

creaseAngle 0

}

Be sure to save the file when you exit the editor.

Steps 4-8 are best done one class at a time. This way, disk space can be saved by deleting intermediate files (DOC***.dat and the SPIDER volumes) once the tiling is satisfactory.

Method 3

This method is suitable for large, simple objects such as the bounding membrane of a cell or an organelle. A model made by this method is usually smoother than one made by method 2. In addition, the model uses a much smaller Inventor file, so the display is faster. This method can be used for complicated structures, but in that case it usually must be used interactively, which can be difficult. Special attention may also be required if the topmost or bottommost contours are highly convex.

This is the only one of the three methods which can make a model of an unclosed surface, such as a patch of membrane. This is because it can tile stacks of contours which are not closed loops.

As with method 2, the procedure should be carried out for one object at a time. If a whole class is selected, the individual objects in the class will not be able to be displayed separately, but only as a group.

9. Transfer Sterecon contour stacks to MOVIE.BYU format.

input file

(e.g.) recon1.ssr [Sterecon file, -.ssr]

output file

(e.g.) ommm.mov [MOVIE.BYU input file, -.mov]

Use Sterecon, INTerface menu (responses in red):

10. Tile objects using "tiling" or "sheet".

input file

(e.g.) ommm.mov [MOVIE.BYU input file, -.mov]

output file

(e.g.) ommm.dat [MOVIE.BYU tiled output file, -.dat]

MOVIE.BYU includes software which directly tiles stacks of contours. The applications "tiling" and "sheet" are used to connect the points on adjacent contours in depth, forming a network of triangles. The data points used are the same as those you entered when tracing contours in Sterecon. Use "tiling" for objects made from closed-loop contours and "sheet" for objects made from open contours. Both applications look and work the same. To start the application, type "tiling" or "sheet". The following dialog takes place in the small box at the lower left of the application (responses in red). Information about your data, as well as additional prompts, appears in the window from which you started the application. Sample text from that window is shown in blue in the example.

If the above procedure did not tile the object properly, the interactive method will have to be used. It is advisable to get help from an experienced staff member for this, but the method is summarized here (user responses only) for reference. Generally, method 2 should be used for objects which are difficult to tile.

read

ommm.mov

manual

auto

OK? y

(An attempt will be made to tile the first two levels. If the results look correct, type yes.)

next

(The next two levels will be displayed.)

auto

OK? n

If the results are not correct, type n. You now have several options to force a correct tiling. The following commands are usually tried, in this order: include, renumber, guide, and triangulate.

Before writing the output file, don't forget to add any necessary caps, as described above.

11. Smooth MOVIE.BYU-format objects.

input file

(e.g.) ommm.dat [MOVIE.BYU output tile file, -.dat]

output file

(e.g.) ommm_s.dat [MOVIE.BYU smoothed tile file]

Objects tiled with MOVIE-BYU sometimes have ridges parallel to the slices due to small errors in tracing contours on adjacent slices. These ridges can be smoothed somewhat using a routine provided in Sterecon's interface menu. Because the slices of tomographic volumes are close-spaced, the smoothing is not likely to obscure any "real" details, so it should normally be used.

Use Sterecon, INTerface menu (responses in red):

INTerface menu (enter MEN to see choices). smo

Inventor or MOVIE.BYU geometry input files? (I/M): m

(Be sure to use m for MOVIE.BYU format.)

Enter source geometry filename: ommm.dat

Enter smoothed geometry filename: ommm_s.dat

Smooth end contours? (N/Y): y

READ: 4367 NODES AND 8661 POLYGONS.

The routine tells you how many points and triangles there were.

12. Convert MOVIE.BYU-format objects to Inventor geometry files

input file

(e.g.) ommm_s.dat [MOVIE.BYU output tile file, -.dat]

output file

(e.g.) ommm.inv [Inventor geometry file, -.inv]

byutoexp ommm_s.dat ommm.inv

Where ommm_s.dat is an example of a smoothed, MOVIE.BYU tiled output file and ommm.inv is the corresponding Inventor geometry file. To distinguish geometry files made by this method, we use the extension -.inv.

Final steps

13. Reduce the number of tiles in Inventor geometry files.

input file

(e.g.) ommm.geo [Inventor geometry file, -.geo]

intermediate input file

(e.g.) ommm.wrl [VRML file, -.wrl]

intermediate output file

(e.g.) ommm_d.inv [decimated Inventor geometry file, -.geo]

output file

(e.g.) ommm_d.geo [scaled, decimated Inventor geometry file, -.geo]

Inventor geometry files made by method 2 have many more tiles than necessary. The final display can be manipulated much faster if the number of tiles is reduced. A stand-alone application called "decimate" (decimate literally means to reduce by 10 times) can be used to reduce the number of tiles. The application provides control of the amount of tile reduction and displays the resulting model after each trial. "Decimate" does not take Inventor files as input, so the -.geo files must be converted to VRML format first, and the output from "decimate" must be rescaled so that all objects will be placed properly in a scene. Decimate only works on individual objects, not on groups of objects.

a. Convert to VRML:

Where ommm.geo is the orginal geometry file and ommm.wrl is the VRML version.

b. Decimate decimate ommm.wrl

It may take quite a long time for the application to start and load the file. When the object appears, click on the "Decimate Surface" button. When the box appears, click on the "Fix Boundry" option. Type in the amount of decimation desired, then click on "Apply" (DO NOT CLICK ON "OK" NOW). If the model is large, this could take some time. For surfaces made using Method 2, 1-5% is usually appropriate, with the smaller value better for larger objects. After some time, the decimated surface will be shown, and the total number of triangles in the decimated surface will be indicated. Ideally, it should be not more than 2000. You can go back to the original number of triangles by typing 100%, or try any other value. When you are satisfied with the decimation, click on "OK" (if you fail to type OK, the model will not be saved in decimated form).

If desired, you can smooth the object after decimation, but before saving the decimated version. This is not suitable for all objects, since it can change the shape. Click on "Relax Surface". When the box appears, click on "Fix Boundry". Type in the number of iterations desired. Start with 5. Higher numbers make the model smoother, but can change the shape. As with "Decimate Surface", the "Apply" button is for viewing the results, and the "OK" button is for accepting the new surface.

To save the new surface, click on the "Save As" button. When the box appears, select file type "Inventor ASCII". Then type in the desired file name in the lower box, and click on "OK"

c. Scale the decimated file

You must copy the file "fixdeci.perl" to your directory from here. This step reads the scaling from your original Inventor geometry file ("ommm.geo" in the example) and applies it to the decimated file ("ommm_d.inv" in the example), creating the new file, both decimated and scaled ("ommm_d.geo" in the example).

14. Smooth Inventor geometry files

input file

Inventor geometry file [-.inv or -.geo]

output file

Smoothed Inventor geometry file [-.inv or -.geo]

This is primarily useful for smoothing Inventor geometry files that were made using method 2 and then decimated using step 13. However, at present it only works if there is only one object represented in the file. It can also be used for inventor files made with method 3 (extention -.inv). However, it uses the same method as step 11, so it is not effective if the *.inv file has already been smoothed in step 11. It has little effect on -.geo files made by method 2 without first applying step 13.

The routine is part of Sterecon. From the Sterecon INTerface menu, use the following example (responses in red):

Enter choice: smo

Inventor or MOVIE.BYU geometry input files? (I/M): i

(Be sure to use i for Inventor format.)

Enter source geometry filename: ommm.geo

Enter smoothed geometry filename: ommm_s.geo

Smooth end contours? (N/Y): y

READ: 4369 COORDINATES

READ: 8661 TRIANGLES

----------- CONNECTIVITY TABLE--------

0 : 2

4 : 4

6 : 58

8 : 86

10 : 645

12 : 2867

14 : 628

16 : 77

18 : 2

--------------------------------------

Some statistics on the data is shown on the screen.

15. Combine objects and make the final surface-rendered model.

All the Inventor files made above can be displayed using SceneViewer or Iris Explorer.

The objects can be manipulated and rendered in many ways. The following just describes the basics.

SceneViewer is simpler to use. Start by typing SceneViewer and use file>open for the first geometry file. Change to orthogonal view by clicking on the bottom box on the icon bar on the right. To add more geometry files, use file>import. To change the color or transparency of an object, use edit>pick edit or click on the arrow on the right-hand icon bar. Then click on the object you want to edit. Wait for a red box to appear around the object selected. Then use editors>material editor and/or editors>color, which will open boxes allowing you to select the rendering you like. You can rotate, translate, and zoom the model using the thumbwheels. If you want to rotate or translate the model by typing in numbers, use edit>pick all and open the transform box using editors>transform. You can save a final model, including the selected objects and color/transparency choices under file>save as.

To use Iris Explorer, type explorer, and pick and drag the modules "ReadGeom" and "Render" from the list to the map area. A ReadGeom module must be used for each geometry file. They can all be connected to Render. The map you make can be saved using file>save as with the extention ".map". Then use Render the same as SceneViewer, above.

NOTES:

Do not move the model while only one object is picked (with a red box). If you do this, only the picked object will move. Since all the objects are made to be in register with each other, it is not necessary to move them separately.

Objects made using method 1 cannot be picked properly, so the color cannot be changed. Since they can't be picked properly, they can't be deleted properly. Therefore, such objects should only be added after the rest of the display is satifactory.

16. Determine surface area from models.

Method A: Use the MOVIE.BYU display panel file, (extention -.dat, made in step 10), then use Sterecon, INTerface menu: ARE. This gives result in square micrometers or square nm, depending on the units you used when setting up the Sterecon file.

Method B: Use an Inventor geometry file and the stand-alone utility "surface". This gives a result in square pixels. When converting to actual area, be sure to use the square of the number of pixels/nm. Also, don't forget to include any size reduction or expansion relative to the original data. More information on this utility will be added later.

Method C: Use the SPIDER command IA. This is performed on a SPIDER volume, so it only applies to objects created using method 2. The file made in step 5 is needed for this operation.

17. Make presentations from models.

18. Miscellaneous tips

(Under construction)

A. Creation of "artificial" Inventor-format objects which can be added to a model.

1. Use SGI Showcase to make a simple 3-D volume, and save it as an Inventor file.

2. Use "ivquicken -a" to make the file editable.

3. Edit the "transform" parameters to get the proportions desired.

4. Add to the SceneViewer or Render model and manipulate as desired.>