GitLab

Gnuastro 0.2.51 only builds with CFITSIO 3.41 or earlier. But this was
not described in the README file. With this commit, a note has been
added regarding the CFITSIO version.

This issue was raised by Alejandro Serrano Borlaff.

A link to the page where I have described reproducible science has
been added to the README to help people unfamiliar with the concept
understand the purpose of this reproduction pipeline.

The link describing where to get the necessary version of Gnuastro was
changed to point directly to the file in Zenodo. Also, the README file
was slightly edited.

The copyright information on the files were not in a unified
format. They are now all in a single format and mention the years that
work has been done on them (2016-2018).

The main input dataset (the MUSE pseudo-broad-band images) and the
whole pipeline in general (along with the HST images and throughputs,
and also the necessary version of Gnuastro) will be uploaded to Zenodo
for easy access in the future.

With this commit, if the MUSE images are not in the given directory,
they will also be downloaded and unpacked for use (similar to the HST
images). The README file is also updated to account for this
change. The contents of this commit will also be uploaded to Zenodo.

This commit does not affect the processing.

Some final touches were made to the pipeline to make it easier and
more stable for reproduction. The most important changes are:

 - The configuration files now have the specific version of Gnuastro
   in them so the pipeline doesn't run with any other version.

 - A README file was added to explain how to run the pipeline and also
   the dependencies.

 - Since the `flock' program might not be available on some systems, a
   Perl script was put in `reproduce/scripts/flock' so people can use
   it instead.

 - Some corrections were made in the downloaded file names.

 - Until now, the pipeline used a separate version of Gnuastro for its
   Table program which could write output into a FITS table (version
   0.2.51-bc56 which is used in the pipeline can't do that). But the
   format of the output is irrelevant, for the purpose here, the
   contents are important. So this step was removed and the final
   tarball output of the pipeline is now fully reproducible with the
   single 0.2.51-bc56 version of Gnuastro.

 - The final tarball is placed in the `reproduce/output' directory,
   because it may be necessary later (when the intermediate files are
   deleted).

The `data-products' target was defined under the comments for the
BibLaTeX target. So to make it more clear, some comments were
specifically written for it and it was placed separtely in that file.

This change has no effect on the results, only helps in readability.

The comments on the final tables were corrected to fit the new
Gnuastro plain text table format. So using the Table program from new
Gnuastro (currently under development) datastruct branch, the plain
text tables were converted to FITS tables. They are then packed into a
`.tar.gz' file to be easily communicated.

The plots for the difference in RA and Dec as a function of magnitude
for the mosaic and the UDF10 have been created in the final PDF.

Until now NoiseChisel was ran on the HST images and the segmentation
map was used for the comparisons of both magitude and
astrometry. While this is accurate for the former, it is not so good
for the latter, especially as the targets become fainter. The shape of
the segmentation map (from HST) will bias the shallower MUSE
astrometry. So for astrometry, we done the opposite: doing detection
on MUSE and using its segmentation map on both MUSE and HST. The final
comparison catalogs are now made with this commit and we can go onto
the statistics.

The difference in distance plots are now also plotted in a similar way
to the differences in magnitude. To do this generically and easily,
the $(two-d-hists) rule in `statistics.mk' was slightly modified, so
all the two-d histograms are generated with separate invokations of a
single AWK command. This greatly helps later corrections/additions.

The source files have been separated: with the Make (`.mk') files now
in `reproduce/make' and the scripts (for the time being only AWK) in
`reproduce/scripts'. Also work has started on deriving the statistics
of each field, but is not yet complete (it isn't a prerequisite of the
final PDF for now in this commit).

Until now, NoiseChisel was run on the degraded HST images only for the
segmentation map. However, because of the bad Sky subtraction in the
HST images (as shown in the previous commit), the F814W result was
strongly affected. So we now run NoiseChisel on the degraded HST
images first, save the segments, and then subtract the Sky value from
the degraded image. The result in the F814W image is now reasonable.

A first draft of a description has been written to explain the
process.  Since the F814W results are so significantly
offset/different, a cutout was also included as PDF in the paper to
display the possible cause.

Until now, the segmentation maps were created from the original
resolution HST image, then warped to the grid of MUSE. However, the
mixed pixels that were created would bias the position measurements of
the objects, so now, we are using NoiseChisel on the degraded HST
image. But since there is practically no noise left in those images,
we use very lax NoiseChisel parameters. Also, the Sky value of the HST
images is subtracted before convolution.

The ranges of the 2D histograms were corrected to more
reasonable/descriptive values for better comparison.

PDF plots of the magnitude differences are now made and included in
the description. Currently, the description doesn't have any text, but
tha will be added when everything is thoroughly checked.

Catalogs with HST and MUSE magnitudes and positions are now created
for the full UDF Mosaic and the UDF-10 field. The comments above the
rules are descriptive enough.

Due to very small WCS differences, we might be having one column or
row difference between the warped HST images and the MUSE images. To
generate the catalog, the two images need to be the same size, so this
was a problem.

To solve it, the MUSE images are now cutout from the input image in a
new directory (`muse-cutouts'). Then, after the HST images are warped,
the MUSE and HST-warped image sizes are compared and if they are
different, the MUSE image corrected (by one row or column of pixels
along the respective dimension). Then it is put into the old `cutouts'
directory. If the sizes are the same, then a copy is just put there.

Magnitude and position catalogs are now generated for both the HST
images and the MUSE images. Using GNU Make's Secondary expansion, we
don't need the `-h' or `-m' suffixes for the segmentation maps any
more. Since the instrument name will be removed from the output
filename during secondary expansion.

The Sky and Sky standard deviation images are now also created. Note
that since the purpose of this study is calibration, they are set to
all 0 and all 1 valued images. Finally, using the filters and
instrument the zeropoint magnitudes are calculated and used for
MakeCatalog.

Currently the catalogs are only used for similarly sized original
cutouts. But the cutouts can be differently sized (due to sub-pixel
differences in the WCS). So when the images have a different size, it
currently stops.

NoiseChisel is now run on the original HST images and the object
labeled image is then warped to the MUSE resolution and prepared to
derive the segmentation maps. To make the prerequisite management of
the final catalog much more easier, the final cutouts of both MUSE and
HST are now in the same directory, only differing with a `-m.fits' or
`-h.fits' suffix. The segmentation maps also follow the same naming
convention, however, since the segmentation map is identical, the
`-m.fits' file is actually a symbolic link to the `-h.fits' file.

The model MUSE PSFs were incorrectly calculated until now! I was
mistakenly assuming that the `FSF01FWA' was the FWHM. However, the
FWHM must be calculated from `b*lambda+a' where `b' is the value to
`FSF01FWB' keyword and `a' is the value of `FSF01FWA'. The effective
`lambda' (or wavelength) to use for each broad-band filter is the
throughput-weighted average wavelength of the filter.

So, the pipeline now uses the HST filter throughputs to find `lambda'
and then uses the values of these keywords to generate the proper FWHM
for the Moffat function. The throughputs will be downloaded if not
already available. The FWHMs decreased from around 14 pixels to around
10 pixels.

After implementing this correction, a test run of the deconvolution
gave a better result. The reason was that the image crop to feed into
the deconvolution algorithm was larger and so lower frequencies could
be sampled. After seeing this trend, I increased the truncation radius
of the model MUSE PSFs and also the deconvolution image size and saw
that the deconvolution output kernel is becoming better and better. I
finally stopped at 6 times the FWHM since it was apparently not making
much more change.

So, now the proper kernel to convolve with the HST images is created
and we can directly compare the results with the MUSE images without
worrying about a larger PSF. After convolution, the HST images are now
also sampled to the same pixel size as MUSE.

The directories directly under the build directory are now built by
one rule in the top level `Makefile' to avoid the confusion and Make
errors that can be caused by doing it every time. This can also
encourage building more of these directories for easy checking.

The size of the kernel was also increased because in the MUSE PSF, 21
pixels was too small. The problem with the deconvolution is certainly
more fundamental than that and must be solved separately. So for the
time being we are ignoring the steps in generating the HST PSF.

The aligned UDF (not UDF10) images were in the same directory as the
cutouts of each subfield. They are now saved in a separate directory
of the top build directory to make the cutouts directory more easily
understandable/managable. Especially since the HST images will be
coming in too soon.

The MUSE PSFs were given as parameters to a Moffat function fit. So
they were created using Gnuastro's MakeProfiles. The HST PSF was also
generated from one of the bright stars in the UDF. It isn't the
brightest, but so far we haven't checked if it was saturated or
not. However, apparently, Gnuastro's convolve had problems in finding
the kernel to convolve with the HST image to get to the MUSE
resolution.

So for now, we have kept the rules (to work on them later), but they
are currently ignored in the pipeline (they aren't a prerequisite to
anything). Since the MUSE PSF is much wider than the HST, for now, we
will just convolve the HST image with the MUSE PSF. Later, when the
problem above is solved, we will find the proper kernel more
accurately.

The UDF1 to UDF9 regions of the MUSE UDF-Mosiac image have been
defined and cropped from the main input image. However, the input
image was not aligned with the celestial coordinates (which ImageCrop
currently needs), so it was first aligned and then the regions were
cut.

The UDF10 region cutouts are now made.