FAQ

• In this FAQ/HowTo, names of variables will be written between '${' and '}'. e.g. ${number_of_cores}. Replace that variable by the necessary value.
• Wiki style (Wikipedia page)

How to compile 3D Slicer on the system

• Most instructions can be found on the 3D Slicer website in the developer's section (check here
• It is advised, if you want to be able to install extensions build on the factory to download Slicer source code from the svn repository.
• If you clone the git repository, you can build Slicer, and install extensions you build yourself against the commit you compiled.
• Use cmake >= 3.0.0
• Use the following options:
  • -DQT_QMAKE_EXECUTABLE:BOOL=/tools/Qt/Qt4.8.6/bin/qmake
  • -DSlicer_USE_PYTHONQT_WITH_OPENSSL:BOOL=ON
  • -DBUILD_TESTING:BOOL=OFF
  • -DSlicer_USE_VTK_DEBUG_LEAKS:BOOL=OFF
  • -DCMAKE_BUILD_TYPE:STRING=Release
• To package Slicer once it is compiled, run the following command from the inner build directory (i.e. your_Slicer_Superbuild_Directory/Slicer-build):

make package

Tools were working before January 2015 but are not working anymore, or have been removed

• The system had not been cleaned in a long time.
• Some tools on the system had been installed in 1999!
• The computer system has changed a lot since then: 64bits computers, linux instead of windows and solaris
• A lot of tools were not working anymore, or seem old and deprecated
• Old libraries were still on the system (ITK 3.8.0, VTK 5.2.0,...)
• To remove unused tools and to free some space on the system, everything was archived
  • Tools that were in /tools were moved to /tools/Archive
  • Tools that were in /tools/devel/linux were moved to /tools/Archive/devel/linux
  • For more information, refer to the README file in /tools
  • /tools/Archive was then copied onto the archiving system athena in /athena_archive/tools and removed from the system
• If you encounter any issue because of that (software that does not work anymore or is missing), let Martin, Francois or Joe know, so that we can unarchive what you need

Checking if some images need to be compressed

• You can use /tools/bin_linux64/checkGzip.script
• Do not interrupt the script while it is running. It will corrupt the files you are compressing
• For *.nhdr files, it will not compress the file, it will just issue a warning. This is due to the fact that a *.nhdr can point to multiple files and that this case is not handled by the script
• It will compress:
  • nrrd
  • gipl
  • nii

Software compiled/Updates

• Tools installed and compiled on the system

Which Slicer version to use

• Slicer-4.3.1-linux-amd64
• Previously
  • On local machines: Slicer-4.2.0-2013-06-06-linux-amd64
  • On servers: Slicer-4.2.0-2013-07-08-linux-amd64

Set an environment variable

• In your current terminal session:
  • e.g. tcsh: setenv PATH ${PATH}:/rodent/tools
  • e.g. bash: export PATH=${PATH}:/rodent/tools
• For all your new terminal sessions:
  • e.g. tcsh: modify the file ~/.auxcshrc and follow the instructions above. Start a new terminal or use one of the following commands (be careful, in the later case,your previous environment may create conflicts)
    • rehash
    • source ~/.auxcshrc

Servers

• list of available servers: http://pandora.ia.unc.edu/wiki/index.php/InfrastructureNotes
• Use pan for high memory jobs and pegasus for high CPU jobs (or the UNC cluster), hera or echo
• Do not use the login servers (janus, charon) to run software or do intensive computations. If you overload the login servers, nobody can connect to the lab anymore.
• Verify that the servers are not overloaded or very busy before you run your job:
  • Connect to the server: e.g. 'ssh -Y pan'
  • Check the jobs already running: 'top'
  • Check if %CPU and %MEM are high
  • Check the numbers of running jobs
  • Mem shows the amount of memory used, but is not accurate to know if the server is busy or not
• If you need to run any application that has a graphical interface on a server, or while being remotely connected to any machine:
  • Connect to the lab using the option -X or -Y (e.g. ssh -Y pan.ia.unc.edu)
  • If you don't, you will have one of the following error message:
    • "Gtk-WARNING **: cannot open display:"
    • "cannot connect to X server"

Full quota in your home directory

• Check how much space is used in your home directory: du -sh ~/
• Check what files actually use a lot of space: du -h --max-depth=1 ~/
• Remove unwanted and unnecessary files:
  • Empty trash
  • Remove/clean firefox history:
    • Open firefox, History->Clear Recent History. Select "Time range to clear: Everything". Check all the boxes. Press "Clear now".
    • or Open firefox, Edit->Preference. Advanced->Network->Cached Web Content: Clear Now, Offline WebContent and User Data: Clear Now
    • or Dangerous!!!! If you cannot open firefox because your quota is full, open a terminal and run: rm -r ~/.mozilla/firefox/*default/Cache

ImageMath

• Webpage: http://www.nitrc.org/projects/niral_utilities/
• Source code: svn --username anonymous --password anonymous checkout https://www.nitrc.org/svn/niral_utilities
• Beware: By default, ImageMath saves output images with the pixel type "short" (integers from −32,768 to 32,767). To change the output image pixel type (e.g. if your input image is an FA with values between 0 and 1), use the option "-type" (e.g. -type float) as part of your command line
• Changing image spacing:
  • e.g.1: ImageMath input_image.nrrd -changeSp .5,.5,.5 -outfile output_image.nrrd
  • e.g.2: ImageMath input_image.nrrd -changeSp reference_image.nrrd -outfile output_image.nrrd

killdevil

• Get access:
  • Connect with a browser to onyen.unc.edu
  • Click on "Subscribed to services
  • Request access to "KillDevil Cluster access"
• Usage: http://help.unc.edu/help/getting-started-on-killdevil/
• Useful commands:
  • module [ add | avail | help | list | load | initadd | unload | initrm | show ]
  • bsub | bjobs | bpeek | bkill | bqueues
• Examples: bsub -q day -M 12 -n 5 -R "span[hosts=1]" ${command}
  • -q: selection of the queue
  • -M: max amount of memory used by the software (in GB)
  • -n: numbers of cores/threads
  • -R "span[hosts=1]": run all the threads on the same host
• Image processing tools on killdevil:
  • /nas02/home/f/b/fbudin/tools
• Monitor process resource usage
  • Get node on which the process is executed: bjobs -X
  • For the job you want to monitor, the information is in EXEC_HOST: [?*]c-XXX-XX
    • c-XXX-XX is the name of the node/host on which the jobs are running
    • ? is the number of cores requested and reserved on that node (-n ? when using bsub)
    • If -R "span[hosts=1]" is not specified, EXEC_HOST might give more than one node. e.g: 1*c-195-30 1*c-194-39
  • ssh c-XXX-XX top -bHn1 -u your_onyen

killdevil - `GLIBCXX_3.4.11'/`GLIBCXX_3.4.9'

• When encountering the following error message when running a tool,
  • ANTS: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.11' not found (required by ANTS)
  • ANTS: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by ANTS)
• type the following commandline: "module initadd gcc/4.8.1" and "module add gcc/4.8.1"

killdevil - Number of threads used

• When running a software on killdevil, the user specifies the number of cores he plans to use: -n ${number_of_cores}
• Some software are by defaults multithreaded (they will try to use as many cores as they can unless otherwise speficied)
• When using a software on killdevil:
  • see if there is an option to limit the number of threads/cores used (e.g. using --help)
  • if the software you plan to use is based on ITK, set the following environment variable to the desired value: ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS (how to)

SVN - edit and commit existing source code

• A short tutorial can be found here: http://www.thegeekstuff.com/2011/04/svn-command-examples/
• When modifying an existing project that is version controlled with svn:
  • Download the source code using "svn checkout"
  • Modify the source code
  • Verify that it compiles
  • Add tests
  • Update the source code in your current source directory to verify that nobody has committed conflicting code: "svn update"
  • Resolve any possible conflict
  • Verify that your code still compiles
  • Verify that the tests still pass
  • Commit your new code (don't forget to add with "svn add" new files if necessary) from the main repository, or from just the repository that has been modified: svn commit
  • Write a useful commit message (A message that you could read in years, and still understand what was done and modified)

tcsh - No match.

• When receiving the following error message: ${CommandName} No match. (e.g. ANTS: No match.)
  • Check if your shell is tcsh:
    • echo $SHELL
    • if the output is /bin/tcsh or anything finishing with '/tcsh', then your shell is tcsh
  • If your shell is tcsh, check if your command line contains the character '[' or ']'. If so, replace them by '\[' or '\]'. The '[' and ']' characters need to be escaped. If not, they are interpreted by tcsh

ANTS or related tools

• Set your environment variable ANTSPATH (how to)
• In case of errors, read tcsh - No match.
• Main website: http://stnava.github.io/ANTs/

Use python

• Use python 2.7.5 to be able to import vtk, SimpleITK or use track: See link
• If python 2.7.5 does not work, try to run it on a different machine (pan, hera, pegasus, echo)

Use matlab

• Matlab is install on some computers in /opt/matlab/. Run '/opt/matlab/bin/matlab'
• If Matlab is not installed on your computer, connect to a server and run '/opt/matlab/bin/matlab'

Remote connection to the lab from your own computer

• From within the UNC network
  • Connect to charon: 'ssh charon.ia.unc.edu' or janus: 'ssh janus.ia.unc.edu'
  • From charon/janus, connect to any computer: e.g. 'ssh pan.ia.unc.edu'
• From outside the UNC network
  • Connect to UNC with your VPN (see link)
  • Then see above 'From within the UNC network'
• Alternatively connect to http://ondemand.rc.unc.edu/ to use longleaf desktop (VPN needs to be active) and then from longleaf use ssh to our lab servers
• Known problems with Mac OS X (10.8):
  • If you try to ssh to ares, without any Xserver application installed, you will get an error message saying "cannot connect to X server"
  • If you install qwartz as your Xserver, you will be able to connect to ares remotely with the "-X" option, but you will get error messages when running any application that needs GLX ("....GLX not found. Aborting.")
  • Solution: Download and install an older version of the XServer here from this page
  • If your program or your session crashes, you might run into the following problem: "xauthority not writeable changes will be ignored". In that case, delete the file ~/.Xauthority

Printers

• Main printer: hermes.ia,unc.edu (room 355)

Lab address

Department of Psychiatry
355 Medical School Wing C
Campus Box 7160
Chapel Hill, NC 27599-7160

• Replace 355 by your office room number

Transforming/Resampling an image

• If the transformation/registration has been performed with ANTs, it is easier to use WarpImageMultiTransform to transform your image (http://stnava.github.io/ANTs/ or run WarpImageMultiTransform -h)
• Otherwise, you can use
  • ResampleVolume2: /tools/bin_linux64/ResampleVolume2 for scalar, DWI and vector images (for more information, run ResampleVolume2 --help)
  • ResampleScalarVectorDWIVolume: /tools/Slicer4/Slicer-4.2.0-2013-06-06-linux-amd64/lib/Slicer-4.2/cli-modules/ResampleScalarVectorDWIVolume (ResampleVolume2 in Slicer4). For more information, run ResampleScalarVectorDWIVolume --help or visit http://www.slicer.org/slicerWiki/index.php/Documentation/4.1/Modules/ResampleScalarVectorDWIVolume
  • ResampleDTIlogEuclidean (/tools/bin_linux64/ResampleDTIlogEuclidean). To resample DTI. For more information, run ResampleDTIlogEuclidean --help
• When resample a label map/segmentation/mask, do not forget to select the nearest neighborhood interpolation
  • with ResampleVolume2 or ResampleScalarVectorDWIVolume: add '-i nn' to your command line
  • with WarpImageMultiTransform, add '--use-NN' to your command line

Converting an image to a different format

• convertITKformats (for 3D images) or convertITKformats4D (4D images). For usage information, run the command line with no arguments in a terminal
• Use ResampleVolume2: e.g. ResampleVolume2 input_image.nrrd output_image.nii.gz

Image Formats to use

• For DTI and DWI
  • The preferred format is NRRD (*.nrrd,*.nhdr and *.raw or *.raw.gz). Nifti (*.nii,*.nii.gz) is also possible. In that case, you may have multiple files, containing the b-values and the b-vectors.
  • To convert from NRRD to Nifti, you can use 'DWIConvert' (/tools/Slicer4/Slicer-4.2.0-2013-06-06-linux-amd64/lib/Slicer-4.2/cli-modules/DWIConvert)
• For other types of images (MRI/CT/...)
  • Many formats are available:
    • Readable with ITK e.g. NRRD (*.nhdr and *.raw or *.raw.gz,*.nrrd), Nifti (*.nii,*.nii.gz), Analyze (*.hdr and *.img), gipl (*.gipl, *.gipl.gz), meta (*.mha, *.mhd and *.raw or *.raw.gz)
    • Other formats: *.mgz
  • The preferred formats are NRRD and Nifti.
    • Advantages of using NRRD:
      • Possibility to use the tool 'unu' for many small operations
      • Possibility to visual the header of the image with any text editor
      • It is the preferred format used in the lab for DTI and DWI
    • Advantages of using Nifti:
      • It is clear if the image if compressed or not, just by looking at the file name
      • It is a standard format (used in FSL)
    • Disadvantages of using gipl: The image does not contain any information about the image orientation (space orientation and space directions)

Mac computers in the lab

• persephone: room 361
• Martin's old laptop: mostly used by Francois. Can be used for development.
• leto: room 355. Can be used for development. Development tools and libraries are in /tools_mac/

Development - Library repositories

• Libraries are compiled in /devel/linux and installed in /tools
• Any tool or library that is in /devel/linux can be erased any time. Do not rely on it.
• Installation: typically in /tools/${name_of_the_library}. Configuration process of the tool/library is done with:
  • CMake: change CMAKE_INSTALL_PREFIX to the directory in which the tools/library needs to be installed
  • ./configure: run ./configure -prefix ${installation_directory}

ccmake -NOTFOUND

• When find_package( ${LIBRARY_NAME} ) is specified in a CMakeLists.txt file, cmake searches a file call ${LIBRARY_NAME}Config.cmake or ${LIBRARY_NAME}-config.cmake
  • If the package is compiled but not installed, this file is typically in the build directory
  • If the package is installed, this file is typically in ${INSTALL_PATH}/lib/cmake/${package_name}/ (ITKv4 and VTK6) or in ${INSTALL_PATH}/lib/${package_name}/ (ITK3 and VTK5)

ITK

• Installed in /tools/ITK/
• Static and dynamic versions compiled and installed
  • e.g.:/NIRAL/tools/ITK/ITKv4.4.1/ITKv4.4.1_THL64_stat_ITK3compatible (give /NIRAL/tools/ITK/ITKv4.4.1/ITKv4.4.1_THL64_stat_ITK3compatible/lib/cmake/ITK-4.4 in ccmake)
  • e.g.:/NIRAL/tools/ITK/ITKv4.4.1/ITKv4.4.1_THL64_dyn (give /NIRAL/tools/ITK/ITKv4.4.1/ITKv4.4.1_THL64_dyn/lib/cmake/ITK-4.4 in ccmake)
• Compiled in /devel/linux/ITK_3.X/ and /devel/linux/ITKv4/

VTK

• Installed in /tools/VTK
• Static and dynamic versions compiled and installed. e.g.: /NIRAL/tools/VTK/VTK_5.10.1/VTK_5.10.1_X64_dyn_release_Qt4.8.2_Python (give /NIRAL/tools/VTK/VTK_5.10.1/VTK_5.10.1_X64_dyn_release_Qt4.8.2_Python/lib/vtk-5.10/ to ccmake)
• Different versions compiled in /NIRAL/devel/linux/VTK/

VTK and Qt

• VTK5: /NIRAL/tools/VTK/VTK_5.10.1/VTK_5.10.1_X64_dyn_release_Qt4.8.2_Python
• VTK6: /NIRAL/tools/VTK/VTK_6.0.0/VTK_6.0.0_linux64_dyn_release_Qt4.8.2_Python2.7.5
• VTK6 cannot be build against Qt5 yet (expected to be working for VTK6.1)

SlicerExecutionModel (GenerateCLP)

• SlicerExecutionModel cannot be installed (more precisely, it can but the installation is incomplete and useless)
• Give the build directory to ccmake. e.g.: /NIRAL/devel/linux/SlicerExecutionModel/SlicerExecutionModel-build_ITKv4.3_dyn
• SlicerExecutionModel is compiled against ITK. When specifying a directory for SlicerExecutionModel and a version of ITK, make sure that it is the same ITK used in both cases
• To avoid the problem mentioned above, in your CMakeLists.txt, write "find_package(SlicerExecutionModel)" above "find_package(ITK)"
• GenerateCLP is part of SlicerExecutionModel. It is in ${SlicerExecutionModel-build}/GenerateCLP/
• Help to write an xml file for SlicerExecutionModel: http://www.slicer.org/slicerWiki/index.php/Slicer3:Execution_Model_Documentation#Slicer3_GUI_Generation
• To simplify your CMakeLists.txt, you can use the CMake macro "SEMMacroBuildCLI" (example)

Qt3

• Installed in /NIRAL/tools/Qt3/Qt3-Release
• Used for old tools

Qt4

• How to compile Qt4
• Installed in /tools/Qt/
• Multiple versions:
  • Qt4.7.4:
    • dynamic compilation: Qt4.7.4-linux64_THL (QT_QMAKE_EXECUTABLE=/NIRAL/tools/Qt/Qt4.7.4-linux64_THL/bin/qmake)
    • static compilation: Qt4.7.4_stat (QT_QMAKE_EXECUTABLE=/NIRAL/tools/Qt/Qt4.7.4_stat/bin/qmake)
  • dynamic compilation: Qt4.8.2 (QT_QMAKE_EXECUTABLE=/NIRAL/tools/Qt/Qt4.8.2/Qt4.8.2_linux64_THL/bin/qmake)
  • static compilation: Qt4.8.4_stat (QT_QMAKE_EXECUTABLE=/NIRAL/tools/Qt/Qt4.8.4_stat/bin/qmake)

Qt5

• Qt5.1: QT_QMAKE_EXECUTABLE=/NIRAL/tools/Qt/Qt5/Qt5.1-dyn/bin/qmake
• VTK6.0.0 cannot be build with Qt5.1

CMake tricks

• Typical commands when using cmake to configure a project:

mkdir projectdir-build
cd projectdir-build
ccmake ../projectdir

• When modifying or updating a project, if one is in the binary directory, one may use

ccmake .

• A better way of doing the same thing, and make sure that the same version of cmake is used, is to type

make edit_cache

• When one has modified the CMakeLists.txt file but doesn't need to modify any CMake variable, one may use:

make rebuild_cache

Compile a tool statically

• To compile a tool statically, it needs to be linked against libraries that are compiled statically (see static and dynamic FAQ)
• To verify the libraries an executable depends on, use:
  • on linux: ldd ${executable_with_full_path}
  • on Mac:: otool -L ${executable}
• The executable will very likely still depend on some system library, but shouldn't depend on other libraries such as ITK, VTK,...

Static and dynamic libraries

• Dynamic libraries (*.so on linux) are loaded when a tool that needs them is started. The library has to be on the system at runtime
• Static libraries (*.a on linux) are copied in the tool that needs them when it is compiled. The library doesn't have to be on the system at runtime, but the executable compiled with static libraries is larger
• The type of compilation is selected when configuring the library:
  • with CMake, typically called "BUILD_SHARED_LIBS" set to ON (dynamic) or OFF (static)
  • with configure, typically -shared (dynamic) or -static (static)
• Typical installation directory on the system
  • Dynamic libraries: ${LIBRARY_NAME}_linux64 or ${LIBRARY_NAME}_linux64_dyn
  • Static libraries: ${LIBRARY_NAME}_stat

Tools on the system

• Image Analysis Tools Manuals
• DTI tractography pipeline

Using the Windows computer (Ratland office)

• Log in using your onyen (user must be registered as belonging to the med school)
  • If it doesn't work, contact the computer support for psychiatry:
    • 966-1826
    • psych_css@unc.edu
• Remote connection
  • ask an administrator to allow your account to remotely connect
    • Search 'remote'. Select 'Allow remote access to your computer', '->Select Users->Add'
  • Try connecting to the machine
    • On a Windows machine, open the start menu and type "remote". An application called "Remote Desktop" should be available (It is by default available in 'Accessories').
    • On a Mac computer, install 'Remote Desktop Connection' (Microsoft) http://www.microsoft.com/en-us/download/details.aspx?id=18140
    • On a Linux Machine, install 'rdesktop'
    • Run the install software
    • Computer name: CSS-W02jblocher361.ad.unc.edu
    • Username: AD\UserName
    • On Mac, 'Domain=AD', 'Username=your_username'
    • On Linux, in a terminal, type 'rdesktop -u AD\\your_user_name CSS-W02jblocher361.ad.unc.edu'
  • From outside campus, use the vpn connection
  • Only one person can be using the windows machine at a time
    • If somebody is trying to connect when another user is connected, it will
    • Ask the new user if they actually want to connect
    • If so, a message will be send to the current user
    • The current user has the choice to:
      • Let the new user connect (Beware: the current user will be disconnected): OK
      • Refuse the connection of the new user: Cancel

Using ITKTransformTools to concatenate ANTS output transform files

• Registering images with ANTS will give 2 output transform files for the deformation field:
  • A displacement field (OutputWarp.nii.gz)
  • An affine transform (OutputAffine.txt)
• You can use ITKTransformTools to concatenate those 2 transform files into 1 displacement field. The order in which the transforms have to be specified is the following:

ITKTransformTools concatenate outputTransform.nrrd -r ReferenceImage.nrrd OutputWarp.nii.gz displacement OutputAffine.txt

Upload large files on NITRC

• NITRC allows to upload large files. However, to do so you have to connect through ftp.
• To connect, use filezilla or sftp (examples)

sftp nitrc_username@www.nitrc.org
cd /home/groups/dtiprocess

• For more information, refer to the nitrc webpage

VTK file formats

• http://www.vtk.org/VTK/img/file-formats.pdf

Compressing VTK files

• Typically, VTK files are saved as ASCII, uncompressed files.
• It is easy to look at their content with a text editor, but it takes a lot of space on the system.
• To save space, VTK files can be compressed using PolyDataCompression.
• Example of USAGE: PolyDataCompression -e binary -f /Data/*.vtk

Anonymizing DICOM files

• Use dcmodify (/tools/bin_linux64/dcmodify)
• You need to modify all dcm files!
• To find how to use dcmodify, just type 'dcmodify' in a terminal. This will print the help.
• Find the tags you want to modify using dcmdump

dcmdump /one/dicom/file.dcm

• Or use Slicer: import the DICOM with 'File->DICOM' and click on 'Import'. Once the DICOM is imported, click on 'Metadata'
• Finally, modify all your DICOM files:

dcmodify *.dcm --modify-tag "(0040,0244)=00000000"

Using DTIAtlasBuilder on killdevil

• In the "Grid Processing" tab, check "Enable Grid Processing"
• In the same tab, under "General Command", write:

bsub -q day -M 4 -n 1

• In the same tab, under "Atlas Building Command", write:

bsub -q day -M 48 -n 12 -R "span[hosts=1]"

• Adjust requested memory amount (-M XX) as necessary.

Copying large files into mass storage on killdevil using Globus Connect

• Copying files >100GB using the cp command is problematic, because the connection to the server will break (Write failed: broken pipe) after several hours.
• Try using Globus Connect. Follow the instructions on http://help.unc.edu/?s=globus
• You will need to create an account and password.

Connection Servers (available through eduroam/VPN)

• charon.ia.unc.edu 152.19.8.150
• janus.ia.unc.edu 152.19.10.22
  • Connect to these machines from off campus with VPN or Campus wireless before accessing other lab resources

Workstations with TITAN Video Cards for Deep Learning

• sirius.ia.unc.edu TITAN X
• castor.ia.unc.edu TITAN Xp
• homer.ia.unc.edu TITAN Xp
• pluto.ia.unc.edu TITAN V
• theseus.ia.unc.edu GeForce RTX 2080

Deep Learning Server with 4 TITAN RTX Video Cards

• theia.ia.unc.edu (IP = 152.19.10.80)