1. 24 Jul, 2018 1 commit
  2. 06 Feb, 2018 1 commit
    • Mohammad Akhlaghi's avatar
      Pipeline ready for public release · f97e8fff
      Mohammad Akhlaghi authored
      Some cosmetic changes were made in the pipeline to prepare it for the
      public release. In particular the following:
      
       - The `README' file was updated to be more clear and follow the same
         format as that of the photometry and astrometry pipeline (with the
         necessary corrections).
      
       - A `configure' script was added to greatly simply the processing of
         setting the input directories. The version controlled
         `DIRECTORIES.mk.in' file is now the base file. After running
         `./configure', a copy of it is put into `DIRECTORIES.mk' and an
         editor is opened so the user can set them easily.
      
       - The acceptable version of Gnuastro was changed from the unofficial
         and not-released version `0.2.52-f620' to `0.2.51-bc56-bugfix'. The
         only necessary correction was in a single file
         (`bin/mkcatalog/upperlimit.c'), so that file is now included in
         this pipeline to allow easy correction to the main Gnuastro branch
         if it is necessary to fetch this snapshot from Gnuastro's history
         directly. The instructions to do this are available in `README'.
      
       - In the version of the pipeline that was used to publish the paper,
         we had 13 extra objects that were not specificly detected by
         ORIGIN, making a total of 173 objects. However, only 160 objects
         have an ORIGIN-only flag in the final catalog (the extra objects
         were not used). So to keep things clean, the 13 extra objects are
         removed from the input table and the whole input table is now
         available in the repository. The file
         `reproduce/input/input-positions.txt' also describes how to
         generate this table from the released catalog. Since the input
         catalog is no longer a user-specified parameter, the file and
         variable names were changed to an all-small-caps format.
      
       - A small Perl `flock' script is added to the pipeline so if it is
         not available on the system, it can easily be copied from here.
      
       - One of the objects (MUSE ID 6512) that was mistakenly classified as
         `RP' (existing/present in Rafelski catalog) is now correctly
         classified as `RC'. The reason was an visual mistake: there was
         another brighter source near the large galaxy, so I had thought
         that is the main target. But zooming into the FITS file, I noticed
         that the main object of interest is actually part of the brigher
         galaxy.
      
       - A typo of `Ns' (instead of `NS') was corrected in the
         classification table. To avoid such mistakes the checks of the
         codes are now in an `if-else' conditional, not independent `if's.
      
       - To allow more clarity in the percentages of the classification,
         they are now reported to two decimals in the internal description.
      
       - The demonstration cutout codes were corrected for the new list
         (that is now only 160 objects, instead of the old 173).
      
       - Removed the extra `subsection.tex' file since the the contents of
         the main PDF weren't included in the main paper either.
      
       - Removed the version number from the final output files, so they
         aren't need-lessly created on every change in the Git repo. The
         version is alrady present within the files.
      
       - Added section describing the small differences between the
         classsifications of this pipeline and those reported in the paper.
      
       - Corrected the copyright information and my email.
      f97e8fff
  3. 30 Jun, 2017 2 commits
    • Mohammad Akhlaghi's avatar
      Corrected version in README · 7080708f
      Mohammad Akhlaghi authored
      From the last commit, the version of Gnuastro has changed from 0.2.51
      to 0.2.52. This change was described in the previous commit.
      
      But the README file was left out! This commit corrects that.
      7080708f
    • Mohammad Akhlaghi's avatar
      Upper-limit magnitude changed to 3 sigma · 5d88fd9c
      Mohammad Akhlaghi authored
      The upper limit magnitude reported for the objects was changed from 1
      sigma to 3 sigma as suggested by Hanae (motivated by the referee of
      her paper). Also, to allow upper limit measurements of large objects
      (where no blank area with that size can be found), we needed to make a
      small correction to Gnuastro after the 0.2.51 commit. Objects like
      this should get a NaN upper limit magnitude (because its impossible to
      calculate it). But Gnuastro 0.2.51 hadn't accounted for this.
      
      This issue has been subsequently fixed in Gnuastro, but in higher
      commits that would possible change the behavior. So to keep things
      reproducible, we just created a branch from that version and used that
      for this reproduction pipeline.
      5d88fd9c
  4. 03 Jun, 2017 1 commit
    • Mohammad Akhlaghi's avatar
      Corrected merging of multi-depth IR filters · b003cbdf
      Mohammad Akhlaghi authored
      In the Infra-red filters, were have multiple depths. The rule that was
      in charge of merging the results of multiple depths into one catalog
      using an uninitiated `i'. So effectively, only the gradient results
      were used. Not the deep and shallow results. This has been corrected.
      b003cbdf
  5. 18 May, 2017 2 commits
    • Mohammad Akhlaghi's avatar
      Correction in only-aperture catalogs · 9d6df47b
      Mohammad Akhlaghi authored
      In the previous commit, filters with a NaN S/N value would get a 0.0
      magnitude and S/N which was not consistent with the other cases (where
      a -1.0 should be used for a no-value). So now, we check the magnitude,
      and if the magnitdue is NaN, a `-1.0' will be put in the magnitude and
      S/N. This corrected that problem.
      
      Also, after some tests, we saw that the raw HST images have been
      over-sky-subtracted, so the magnitudes were not reasonable. So now, we
      are using the Sky values from NoiseChisel instead of the zero valued
      array.
      9d6df47b
    • Mohammad Akhlaghi's avatar
      Magnitudes and S/N values only with apertures · 6953cf4c
      Mohammad Akhlaghi authored
      As suggested by Roland, a secondary catalog has been made that only
      uses an aperture of diameter 0.4 arcseconds. It assumes a Sky value of
      zero and uses the weight maps to derive the standard deviation. In
      order to do this, the pipeline will now download the `_wht' images of
      the input filters also.
      6953cf4c
  6. 17 May, 2017 1 commit
    • Mohammad Akhlaghi's avatar
      Fixed versions in output names and Gnuastro · 78a2d3de
      Mohammad Akhlaghi authored
      The output files are now prefixed with the version of the pipeline
      that has created them. Until now, this version was written into the
      file, but not in the file name. Also, Gnuastro's programs are
      configured to only work with Gnuastro version 0.2.51.
      
      Also, the base name for the output files were changed to
      `hst-raf-comp-cat' and `hst-raf-comp-seg', for "HST Rafelski
      Complement" catalog and segmentation map.
      78a2d3de
  7. 16 May, 2017 2 commits
  8. 03 Mar, 2017 1 commit
    • Mohammad Akhlaghi's avatar
      Sigma-clipped difference in a bin instead of plot · 6ade6d53
      Mohammad Akhlaghi authored
      Previously we had intended to include the full 2D histogram of the
      magnitude difference between NoiseChisel and Rafelski+2015. However,
      the paper already has too many figures making it hard to read, so we
      decided to remove the 2D histogram and just put a short statistic
      (sigma-clipped RMS in one bin) in text instead.
      
      This commit does the calculation and adds the few lines describing it
      which can be directly input into the paper.
      6ade6d53
  9. 06 Jan, 2017 2 commits
  10. 05 Jan, 2017 1 commit
    • Mohammad Akhlaghi's avatar
      Separate smaller subsection to go in final paper · 76efc9ee
      Mohammad Akhlaghi authored
      The written description was too long, so it was decided to write a
      summary subsection to go into the final paper. So I wrote a summary to
      fit within the lengths discussed.
      
      Also to make plotting in the paper easier, the magnitude differnece
      plot is now saved as a FITS binary table and included in the
      `reproduce/output' directory (for files that are to be shared.
      
      In order to make the FITS binary table, I had to use an
      under-development version of Gnuastro, so the README file as updated
      to explain what people should do until Gnuastro 0.3 is released.
      76efc9ee
  11. 11 Dec, 2016 1 commit
    • Mohammad Akhlaghi's avatar
      Build symlink now dependency of final pdf · bc241b0f
      Mohammad Akhlaghi authored
      The build directory symbolic link was not created by default! The
      reason was that the actual build directory was a prerequisite of the
      final PDF. Now the symbolic link has replaced it, since the build
      directory is a prerequisite of the symbolic link, it will be built
      anyway.
      
      Also, the total fraction of objects not detected by NoiseChisel was
      added to the output LaTeX macros.
      bc241b0f
  12. 30 Nov, 2016 1 commit
    • Mohammad Akhlaghi's avatar
      Changed visual classification for new catalog · 9b1690e7
      Mohammad Akhlaghi authored
      Roland sent me a new catalog for analysis, so the visual
      classification was re-done.
      
      Also, in the comments of `aperture.mk', the size of the aperture
      was also incorrectly written (the value was correct!).
      9b1690e7
  13. 07 Nov, 2016 1 commit
    • Mohammad Akhlaghi's avatar
      Final segmentation map now prerequisite of PDF · 60c20a94
      Mohammad Akhlaghi authored
      The final segmentation map is now a prerequisite of `description.pdf',
      so it will be created. Until now it was necessary to explicitly add
      it. This is not the segmentation map used internally, this the merged
      segmentation map of this pipeline with Rafelski+2015's segmentation
      map.
      60c20a94
  14. 03 Nov, 2016 5 commits
  15. 02 Nov, 2016 2 commits
    • Mohammad Akhlaghi's avatar
      General explanations of full procedure complete · 90487be5
      Mohammad Akhlaghi authored
      A general outline of the full procedure is now present in the
      description PDF, all the way to the final outliers.
      90487be5
    • Mohammad Akhlaghi's avatar
      New classification in description's figures and text · d7196e6f
      Mohammad Akhlaghi authored
      The new classification was adopted in the description's figures and
      text. In particular the old cutouts figure is now broken up into two
      figures showing the two different types of objects that we are
      discussing.
      
      Some other minor corrections:
      
       - Also, there is now no `DEFAULT_GOAL', everything is a prerequisite
         of the final PDF file.
      
       - Two of the classifications were corrected.
      
       - The flux range in the display PDFs was made slightly narrower to
         display better.
      d7196e6f
  16. 01 Nov, 2016 3 commits
  17. 28 Oct, 2016 2 commits
    • Mohammad Akhlaghi's avatar
      Versions and date now printed under authors · fec05a7b
      Mohammad Akhlaghi authored
      Until now this information was printed as a footnote to the title. But
      with this commit, this information is now printed under the
      authors. This was partly motivated by the fact that with the adoption
      of `\twocolumns', the title's footnotes are not visible any more.
      fec05a7b
    • Mohammad Akhlaghi's avatar
      Using \twocolumns instead of the multicols package · 3938dcfd
      Mohammad Akhlaghi authored
      The multicols package had problems with displaying floating figures on
      one column, so we moved to LaTeX's twocolumn environment. With this it
      is easy to have figure in one or over both columns.
      
      Also the Make rules to create cutouts were moved to another file to
      make the `description.tex' file only dedicated to TeX related rules.
      
      Figure captions are now also managed using the caption package and the
      2D histogram has been polished to be more clear.
      3938dcfd
  18. 26 Oct, 2016 1 commit
    • Mohammad Akhlaghi's avatar
      Small corrections · 0bedad3e
      Mohammad Akhlaghi authored
      Some small corrections were done to fix prerequisites in
      multi-threaded builds and in various parts of the pipeline.
      0bedad3e
  19. 25 Oct, 2016 1 commit
  20. 13 Oct, 2016 1 commit
    • Mohammad Akhlaghi's avatar
      Magnitude difference (with R15) plot added in paper · d553f99b
      Mohammad Akhlaghi authored
      The magnitude difference plot has been added to the paper. The
      initial, simple Python implementation was removed. Because it involved
      the plotting of a very large number of points. The points also fell
      over each other so the true distribution was not easily visible. So a
      new 2D histogram derivation program was written in AWK and PGFPlots
      was used to draw the 2D histogram in LaTeX.
      d553f99b
  21. 12 Oct, 2016 1 commit
    • Mohammad Akhlaghi's avatar
      Why NoiseChisel and its features explained · 3d96181e
      Mohammad Akhlaghi authored
      The discussion on why NoiseChisel was used is now explained in the
      text.
      
      The cutout filter is now set in its own separate Makefile
      `r15prob-cutout-filter.mk' so the result will change if it changes.
      
      Also some more parameters are now read from the pipeline and included
      in the LaTeX source. Since the final catalog is necessary for one of
      them, the final catalog is now removed from the prerequisites of the
      `all' target.
      3d96181e
  22. 11 Oct, 2016 6 commits
    • Mohammad Akhlaghi's avatar
      Cutout images of some of the objects now in paper · 2d0f8ac9
      Mohammad Akhlaghi authored
      A classification of all the objects was done and they were divided
      into three classes: deblending problems (d), visible to the eye(v),
      and un-visible to the eye (u). Some of the objects were chosen and put
      into the paper along with a short discussion.
      
      Also the LaTeX macros produced during the rules were moved into a
      separate directory to keep them clean. Since the other LaTeX files are
      also kept in that directory.
      2d0f8ac9
    • Mohammad Akhlaghi's avatar
      Demonstration cutout of all objects · 624f3648
      Mohammad Akhlaghi authored
      A few rules were added to `description.mk' to show a cutout of all the
      objects in the F775W filter. These rules are mainly intended for
      deciding which cutouts to use in the final LaTeX file but it was
      necessary to have an over-all look at them.
      624f3648
    • Mohammad Akhlaghi's avatar
      Some notes on reproducibility for later · 93a43cae
      Mohammad Akhlaghi authored
      Some notes were added to suggest uploading the pipeline to arXiv. Also
      `tex/pipeline.tex' was removed as a prerequisite to `description.bbl'.
      93a43cae
    • Mohammad Akhlaghi's avatar
      Input catalog is now a configuration variable · 36be1ab7
      Mohammad Akhlaghi authored
      The input catalog was previously hard-coded in the Makefiles (and this
      reproduction pipeline), but now it is a configuration variable and the
      file can be located anywhere.
      
      This was mainly done so we can use `git clean -fxd' easily to cleanup
      the working directory. In the previous versions, the input catalog
      (which was here but not version controlled) would also get deleted.
      
      This change is also useful in the spirit of keeping the inputs
      separate from the reproduction pipeline which can help in their
      management (like the intermediate files that are kept in `$BDIR').
      
      Also, all the configuration variables are now read before the separate
      Makefiles in one command. Previously they were only read when
      needed. But this greatly simplifies the process of freely using their
      values anywhere.
      36be1ab7
    • Mohammad Akhlaghi's avatar
      Top level Makefile broken into multiple Makefiles · 4ec553e4
      Mohammad Akhlaghi authored
      The top level Makefile was growing too much and becoming hard to
      manage. So it was broken up into multiple Makefiles. We did not adopt
      the sub-Make file structure though. They are all just included in the
      top Makefile as Make is reading it. So ultimately Make sees all these
      Makefiles as one Makefile. This greatly helps in managing the
      prerequisites between the different Makefiles.
      4ec553e4
    • Mohammad Akhlaghi's avatar
      LaTeX macros built in separate rules, text extended · 729cec2d
      Mohammad Akhlaghi authored
      Until now, the LaTeX macros were built all at once in one rule (except
      for versions). But now, after each step, the necessary macros are
      built separately and the final rule only concatenates them together
      into one file for LaTeX.
      
      The text was also made into a two column text for a more similar
      experience with the final paper. The download rules for the
      CANDELS-Deep GOODS-South, GOODS-South and 3D HST images were also
      removed since they are not needed.
      729cec2d
  23. 10 Oct, 2016 1 commit