Mesquite Project HowTo Guide  (by Boyd Tidwell)

Mesquite is part of the Trilinos system so a good starting place for general info is:

      https://software.sandia.gov/trilinos/developer/index.html

  This pages has many links to useful things such as Bugzilla, SSH, and git/eg.


1. Installing and Learning Git

   Windows:
     TortoiseGit is a good choice.  Binds with Windows menu system to allow non-command line operation.  If command line control is needed/wanted, provides a Git Bash "terminal" for a Linux-like experience. Git requires SSH which is not directly supplied by Windows. PuTTY is a free implementation that works well.

   Linux: The Trilinos system seems to prefer Easy Git (eg), a single-file wrapper script for git.  If you are new to git it can provide a safety net by catching some of the common errors made by beginners.

   Some git commands I found useful:

     Delete extra, not versioned files in a git repository: 
           git clean -d -x (-n)

     Abandon committed but not pushed changes:
           git reset HEAD <file name>
           git checkout <file name>

     Complete reset to head, get rid of work in progress
           git reset --hard HEAD

     Merge branch into master branch
           # on your branch
           git rebase master
           git checkout master
           git merge mesquite_dev
           git push
           git branch -d mesquite_dev   # delete local branch

           git branch origin :mesquite_dev  # drop remote branch (not allowed for Trilinos)


2. Get Mesquite source code:

   a. Checkout fresh copy of Trilinos (and Mesquite):
                     eg clone software.sandia.gov:/space/git/Trilinos

     This is the version used for development purposes (i.e, adding features and fixing bugs).

   b. Get Current Development Version from: http://software.sandia.gov/~bktidwe/mesquite.html

3.  Building Mesquite

     Linux:
       (from the mesquite directory)
        ./mkconfigure
        ./configure  (use --help to see all possible options)  
                      or configure with CMake.
        Make

   Note: If the ./mkconfigure command doesn't appear to work correctly on software.sandia.gov, it's probably due to their old versions of the GNU autotools. An easy solution is to copy the autotool files from /home/bktidwe/bin to your own ~/bin directory and adjust your $PATH variable in .bash_profile so that your local bin directory is searched before /usr/bin.

    CMake is used to configure Mesquite on Windows and can be used on Linux if the iMesh and/or iMeshP interfaces are not needed (See below).  On Windows MS C++ Visual Studio is used to compile and link. 

  A "normal" build of Mesquite is called Serial because it is single threaded and only uses one CPU. To build Mesquite for Parallel execution it must be built with a version of MPI (most people use openmpi).  This works well on Linux and Mac but apparently is not really supported on Windows.  To build an MPI version:

       - build and install a version of openmpi.
       - from the Mesquite top level directory:
            ./configure --with-mpi=(MPI_DIR)   
                 (where MPI_DIR is the path where openmpi 
                  was installed.)
            make

    example for second step above: ./configure --with-mpi=/usr/local/openmpi
       

   IMesh and IMeshP are ITAPS common interfaces that support the abstract data model.  Go to "http://www.itaps.org" for more information.

    To build Serial iMesh:
        -build Serial FMDB using instructions at: http://www.scorec.rpi.edu/FMDB/build.html
        -from Mesquite top-level directory enter:
             ./mkconfigure
             ./configure --with-imesh=
                          /<path_to_fmdbSerial>/lib/iMesh-Defs.inc
              make
    
       Real-life example of building Serial iMesh:

        (from your home directory)

      $ export ENABLE_ITAPS=1
      $ mkdir fmdbSerial
      $ cd fmdbSerial
      $ ~/Downloads/build_Serial_FMSB_DB_GMI.sh
      $ cd ~/Trilinos/packages/mesquite
      $ ./mkconfigure
      $ ./configure --with-imesh=/home/bktidwell/fmdbSerial/fmdbSerial/lib/iMesh-Defs.inc
      $ make
      $ make check



    To build Parallel iMeshP
        - build Parallel FMDB using instructions at: 
         http://www.scorec.rpi.edu/FMDB/build.html    

          It seems to work better if you delete the Serial FMDB installation 
          before building Parallel FMDB. Make sure MPIHOME, CXX, and CC point 
          to the correct mpi location and compilers and the environmental
          variable ENABLE_ITAPS=1 is set.
         - manually copy SCOREC_Software/FMDB-1.3.7/imesh/IMESH_FCDefs.h to 
           fmdbParallel/include. This must be done because of a bug in the FMDB 
           build scripts.  It has been reported.

         - from Mesquite top-level directory enter:
            $ ./mkconfigure  (just to be safe)
                     /<path_to_fmdbParallell>/lib/iMeshP-Defs.inc 
                     --with-mpi=<path_to_mpi>
            $ make

       Real-life example of building Parallell iMeshP:

        (from your home directory)

      $ export MPIHOME=/usr/local/openmpi
      $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MPIHOME/lib
      $ export CXX=$MPIHOME/bin/mpicxx
      $ export CC=$MPIHOME/bin/mpicc
      $ export ENABLE_ITAPS=1
      $ mkdir FMDB-1.3.8_Paralell
      $ cd FMDB-1.3.8_Paralell
      $ ~/Downloads/build_Parallel_FMSB_DB_GMI.sh
      $ cp /home/bktidwell/fmdbParallel/SCOREC_Software/FMDB-1.3.8/imesh/IMESH-FCDEFs.h
          /home/btidwell/FMDB-1.3.8_Paralell/fmdbParallel/include  
      $ ./mkconfigure
      $ ./configure --with-imesh=/home/bktidwell/FMDB-1.3.8_Paralell/fmdbParallel/lib/iMeshP-Defs.inc
          --with-imeshp=/home/bktidwell/FMDB-1.3.8_Paralell/fmdbParallel/lib/iMeshP-Defs.inc
          --with-mpi=/usr/local/openmpi/
      $ make clean
      $ make
      $ make check

  Note: The ITAPS interface files (iGeom.h, iBase.h, iMesh.h, and iMeshP.h) needed by Mesquite for the Serial and Parallel builds are not located in the Mesquite source.  They are provided by the application that supplies the actual support code, in this case FMDB.  When a new version of ITAPS is released, a new version of FMDB must be obtained that contains the new ITAPS headers.  For additional detail, see the note at the end of this document under the heading "Miscellaneous Useful and/or Interesting Information".


4.  Testing Mesquite

  The testSuite for Mesquite contains two types of tests; individual tests that are found in their own descriptive directory with a main.cpp file, and unit tests that require a special utility called CppUnit. Each type is described below. The tests are located in the directory mesquite/testSuite.

4.1  Linux

4.1.1 Using GNU Autotools

   On Linux, after configuring and building Mesquite, at the mesquite directory enter:
             make check
    
  This will build and run all the tests in the testSuite (and even build Mesquite if that hasn't been done). Be sure to watch as the tests go by because there is no results summation at the end.  It should list any errors at the end though.

If there is a problem locating the cppunit libraries, make sure cppunit has be installed.  Refer to the Mesquite Users Guide on how to obtain and install cppunit.

  From the top-level mesquite directory:     
          
     $ ./configure --with-cppunit=<cppunit_install_dir>
     $ make
     $ make check

4.1.2 Using CMake

  CMake can also be used to configure testing on Linux.  The easiest was is to run the command "cmake-gui" and, when configuring Mesquite, make sure the option "Mesquite_ENABLE_TESTS" is set to "ON".  If you are building Mesquite within the Trilinos framework, "Trilinos_ENABLE_TEST" should be set to "ON" as well.  

   Configure cppunit as described in the Windows section below; (CPPUNIT_LIBRARY will be set to <cppunit_install_dir>/lib/libcppunit.so'). 

After configuring Mesquite, exit CMake and cd to the build directory. (Hopefully you are doing an outside of source build).  

From that directory, enter:

  make
  ctest

4.2 Windows

   On Windows, the best way to run the unit tests is using CMake and CppUnit.  The tests come in two forms.  Many are stand-alone executables named "main.cpp" and can be run individually be making them the default project and running them from the "Debug" menu.  A lot of the tests are executed through the "msq_test" project.  These tests are written using test harness software known as cppunit. See below for instructions on setting up cppunit and the unit tests.  Though cppunit makes writing and running tests very easy, it makes figuring out which tests are running very difficult.  There are no clues in the output to what order the tests are run and the only way I could figure it out is by opening various test programs in Visual Studio, putting a breakpoint at the beginning, and then run the tests in debug mode to see when the break was hit. 

To configure the running of the unit tests you need a 3rd-party open source unit testing framework called CppUnit that can be found at:
         http://sourceforge.net/projects/cppunit/files/cppunit/1.12.1/cppunit-1.12.1.tar.gz/download

  Cppunit needs to be built and installed before it can be used.  Follow the instructions included with the download.  When I built Cppunit using VS 2008 Express, only three of the six projects built cleanly but all the libraries needed by Mesquite were built so I didn't worry about the three that failed to build.

Cppunit on Windows:
  The only way to configure the tests on Windows is via CMake.  In CMake, search for 'cppunit' and set CPPUNIT_INCLUDES to '<cppunit_install_dir>/include', set CPPUNIT_LIBRARY to '<cppunit_install_dir>/lib/cppunit.lib'. Set Mesquite_ENABLE_TEST to ON, and if building a Trilinos version, set Trilinos_ENABLE_TEST" to "ON". Press 'Configure' then 'Generate' to create the make files.  Exit CMake and build Mesquite using Visual Studio C++.  When done building, in the Source Explorer window right click on the 'msq_test' project and select 'Set as StartUp Project'.  From the Debug menu Start the project (with or without Debugging) and the unit tests should execute.

  Note that it will not work to build the unit tests on Windows if Mesquite is built as a DLL.  It must be built as a .lib on Windows because the unit tests reference Mesquite internals that aren't exported from the DLL.  This restriction dates back quite a while and may no longer be true.  If you are bored, you could try building Mesquite with shared libraries and cppunit.  If it works, inform someone on the Mesquite Support Team so they can change these words.
      
4.2 Trilinos Checkin Test

  Before actually checking in changes, it is a good idea to run the Trilinos Checkin script.  All the gory details on doing this are at:

      https://software.sandia.gov/trilinos/developer/checkin-test.help.out

  Note that the checkin procedure is not available on Windows. The instructions are pretty straight-forward but here are a few things that could save you some headaches:
   - Create the CHECKIN directory outside the Trilinos top level directory to 
     prevent the "git clean" performed as part of the CHECKIN testing 
     from wiping out your CHECKIN configuration info.
   - cd to the Mesquite top level directory within the Trilinos directory 
     structure.
   - After committing your changes (but not pushing them), do a 'git status' to
     see what files may have been missed.  Usually there will be quite a number  
     of Untracked files. Scan the list for any new files that should be added to 
     the repository or files that should be copied to a location outside the 
     Trilinos directory structure. All untracked files will be deleted by the 
     next step.
   - If desired, list all files that will be removed by doing a dry-run clean:  
         'git clean -d -x -n'
   - Remove all untracked files by entering:  'git clean -d -x -f'
   - Update all Trilinos files:  'git pull'
   - Assumming the CHECKIN dir is on the same level as the Trilinos directory, 
     run the checkin tests:
              $ cd to CHECKIN dir                     
              $ ../Trilinos/checkin-test.py --do-all --no-eg-git-version-check

   - To run only MPI_DEBUG: 
              $ ../Trilinos/checkin-test.py --default-builds=MPI_DEBUG              
  
   - After the checkin test is complete, you will have to do the './mkconfigure' 
     and ./configure commands from the top level Mesquite directory because the 
     clean will have wiped out all the files mkconfigure and configure creates.

5. Setting up cron jobs to create distro tarballs

You'll need to move the Mesquite downloads page and scripts for updating those nightly from the current account on software.sandia.gov to new account of your choosing.  Moving the existing data is simple enough:
   a) Copy the entire contents of "/space/home/<cur_user>/public_html" to some  
      web-accessible directory of  your choosing.  You should probably use 'tar'
      to do the copying so that symbolic links are still symbolic links rather 
      than copies of the file.  For example:
              tar -C ~bktidwe -cvf - public_html | tar -C ~pknupp -xvf -
   b) Update the 1400 download link appropriately.  The current link is    \
        http://software.sandia.gov/~bktidwe/mesquite.html .  The last time it 
        was updated, Karen Devine was the person to contact about that.

The second part of this process, which may not be necessary if you think there will be no ongoing Mesquite development, is to set up a script to update the tarball of the repository trunk nightly.  The script checks if anything has changed in the trunk and if not, does nothing.  If the trunk has changed, it runs "make distcheck", which generates a tarball and then builds Mesquite and runs the test suite from the source contained in the tarball.  If that succeeds then it updates the tarball at the download location and the corresponding generated documentation.

The script itself is /space/home/bktidwe/git.cron.  Copy that script into your home directory and then run "crontab -e" and add lines something like the following to run the script each night:

PATH=$HOME/bin:/usr/local/bin:/usr/kerberos/bin:/usr/bin:/bin:/usr/sbin
#min	hour	DoM	mon	DoW	command
10	1	*	*	*	./git.cron nightly/master $HOME/public_html
40	1	*	*	*     ./git.cron nightly/MESQUITE_2_1 $HOME/public_html

Next, check out the copies of Mesquite that are to be built.  From your home directory, do:
  # Edit cron lines above if you want to change this directory name
  mkdir nighlty
  cd nightly
  # Edit cron lines above if you want to change this directory name
  mkdir master
  cd master
  # Edit git.cron *script*  if you want to change this directory name
  git clone /space/git/Trilinos src
  cd ..
  # Edit cron lines above if you want to change this directory name
  mkdir MESQUITE_2_1
  cd MESQUITE_2_1
  # Edit git.cron *script*  if you want to change this directory name
  git clone /space/git/Trilinos src
  cd src
  git checkout MESQUITE_2_1

If everything is correct, you should now have the following files:
ls ${HOME}/nightly/master/src/packages/mesquite/src/include/Mesquite.hpp
ls ${HOME}/nightly/MESQUITE_2_1/src/packages/mesquite/src/include/Mesquite.hpp

To verify that everything is working correctly, set the timestamp on the
tarballs in your public_html directory such that they look out of date:
  touch -c ~/public_html/mesquite-2.99.tar.gz -t 04010000
  touch -c ~/public_html/mesquite-2.1.tar.gz -t 04010000

The choice of 04010000 is arbitrary (midnight April 1st.)

And then, from your home directory, run the script:

 $ cd ~
 $ ./git.cron nightly/master /home/bktidwe/public_html pknupp@sandia.gov
 $ ./git.cron nightly/MESQUITE_2_1 /home/bktidwe/public_html pknupp@sandia.gov

And finally, verify that the tarballs got updated:
 $ ls -l public_html/mesquite-2.99.tar.gz
 $ ls -l public_html/mesquite-2.1.tar.gz

If something went wrong, you should see an email, and there should also be
log files in  ~/nightly/master/src and ~/nightly/MESQUITE_2_1/src.


6.  Releases

-How to do a Release

Full Release (assuming version 2.2.0, substitute as appropriate):

  1) Update file containing list of changes for each Mesquite version
       - Filename: ReleaseNotes.txt It is the top level of the Mesquite source. 
  2) Check in that change to release history
  3) Create git branch for new mesquite release series, named something like
       MESQUITE_2_2 (be consistent with whatever naming scheme was used before)
  4) Check out branch
  5) Modify version number in configure.ac appropriately and check into new
      branch
  6) Tag the actual release in the release series branch as MESQUITE_2_2_0 
       (or whatever syntax is consistent with past releases)
  7) Perform the steps outlined in the Section 7 below: "How to Update Mesquite
     Benchmark Tests Results File" to create a current version of the benchmark 
     tests results.
  8) Get nightly tarball generation set up for new branch following instruction 
     in Section 5 above.
  9) Run the same command you added to crontab manually to verify that it works,
      and to generate the first tarball
 10) Update download page for new release version 
 11) Send message to mesquite-announce@software.sandia.gov


  Bug fix release (e.g. 2.1.5):

  The same as above, except that you should begin by checking out the MESQUITE_2_1 release series branch and make all changes to that branch (so no step 3, and 1 and 2 happen after 4).


7. How to Update Mesquite Benchmark Tests Results File

  The benchmark tests are found in the mesquite/testSuite/benchmark_tests directory.  The tests provide a way to monitor the execution times of various mesquite optimizers and solvers, watching for excessive jumps.  Ideally, a single computer would be used to run the tests on the latest release to add the current results to the file contains the cumulative results: /testSuite/benchmark_tests_results.txt.  But, because computers break and change, it's probably not reasonable to expect the same computer with the same configuration to be available over time.  These instructions explain how to create an accurate results file for all previous releases on a single machine.  Then should be preformed as the last step before building a release so that the file created by the process will include the current release and will be included in the release tarball. Because the benchmark tests automatically include the version of the release they are running against, the changes should have been made to update the "mesquite_version.h" file to the current release version.

These instructions are for a Linux system.  You could probably do the same thing on a Windows or Mac but it would be different.  Especially on Windows.

The source file name in the benchmark_tests directory is main.cpp.  To build the main binary if not present, enter "make check-TESTS".  

The benchmark tests were added in release 2.2.1 but it is possible to run them in version 2.2.0 with a little farbling that will be explained later.  The test cannot be run against versions before 2.2.0 because the wrappers is uses did not exist before that release.

First thing to do is obtain and build all releases of Mesquite since 2.2.0.  These can be found in the mesquite_releases directory of the current maintainer of Mesquite on the software.sandia.gov server.  It is currently "/home/bktidwe/mesquite_releases".  Create a temporary directory and copy all the tarballs for all the releases to be tested to the directory.  Unzip each of them and build each one by cd'ing to each unzipped directory and entering:

  ./mkconfigure
  ./congiure
  make

 Then, to make sure that everything build correctly, cd to testSuite/benchmark_tests and enter:

  ./main

 The benchmark tests should then run.

Once the above steps have been completed, delete the file "benchmark_test_results.txt" from the testSuite/benchmark_tests in your current working Mesquite repository.  Then, starting with the oldest release that was built, cd to each of the "mesquite-X.Y.Z/testSuite/benchmark_tests" directories and enter:

  ./main >> <path_to_mesquite>/mesquite/testSuite/benchmark_tests/benchmark_test_results.txt

Finally, cd to the testSuite/benchmark_tests directory for the current code being released and enter the command:

   ./main >> benchmark_test_results.txt 

Check the benchmark_test_results.txt file into the Mesquite repository and you are ready to release the current version.



8. OS Platforms

  The Mesquite environment seems to be geared towards Linux.  It's the only platform of the three (Windows, Mac, Linux) where everything can be done easily and completely. 

 Windows doesn't have a make system and I haven't been able to get the make files to work using Cygwin. CMake is used on Windows to build the basic libraries but I had lots of trouble trying to build the IMeshP version using CMake within the Trilinos directory structure.  As far as I can tell, no one use the Parallel features on Windows so it's not a problem.

 Mac has a usable make system that can be used to build the Mesquite libraries but apparently, according to Pat Knupp, it's very difficult to build the users guide on Mac.  

9. Users-guide and related documentation

  How to create Mesquite users guide
      (starting in the mesquite directory)
         ./mkconfigure
         ./configure  --enable-user-guide=HTML,PDF,PS  or ALL
         cd doc/user
         make pdf
         # or: make ps
  
   To create api docs:
        ./configure --enable-api-doc=HTML,PDF,PS or =ALL                                 



10. Miscellaneous Useful and/or Interesting Information

  Taken from a commit log:

Author: Jason Kraftcheck <kraftche@cae.wisc.edu>
Date:   Wed Feb 10 16:02:16 2010 -0600

    Changes to configuration and build of ITAPS support:

    1) Mesquite previously compiled support for all ITAPS APIs w/out needing
    an implementation of each API to be installed by keeping its own copy
    of all of the ITAPS headers.  This changes the build system such that an
    implementation of each API must be present for Mesquite to enable support
    for that API, and Mesquite now uses the implementations' versions of the
    ITAPS API headers rather than its own copy.  This change is
    necessary because of the ITAPS Fortran name-mangling mess.  I'm tired of
    tracking down issues such as the case where a user builds MOAB, then at a
    later date installs a Fortran compiler and at a yet later date builds
    Mesquite. This scenario results in a MOAB and Mesquite being unable to
    link together because Mesquite did Fortran name-mangling for the iMesh
    API and MOAB did not.

    2) As a part of implementing the above change, change the configure options
    from --enable-imesh, etc. to --with-imesh=<path>, where <path> must be the
    iMesh-Defs.inc file that came with the iMesh implementation.  Also, honor
    the Fathom group convention of using environmental variables such as
    IMESH_DEFS to specify the name of that file. If IMESH_DEFS is defined, then
    configure will default to enabling iMesh support using the specified file.
    This behavior can be overridden by specifying --without-imesh.

    3) Build the Mesquite support for ITAPS APIs as separate libraries     
    (libmsqimesh, ibmsqigeom, etc.)  This way when Mesquite is built as a shared 
    library, the user need only link ITAPS API implementations (e.g. MOAB, 
    CGM, etc.) if they want to use the ITAPS APIs (the core Mesquite library 
    doesn't depend on the ITAPS API implementations.)

