Analysis Challenge #4

Welcome to the fourth ngEHT Analysis Challenge! The fourth Challenge differs from the first three in that the primary focus is not on image reconstruction, but on black hole parameter estimation. We have generated synthetic ngEHT datasets from three different M87-inspired models: a geometric m-ring model, a time-averaged GRMHD image, and a GRMHD snapshot image. We invite participants to submit measurements (preferably with 1-sigma errors) of a set of parameters defined below. If you want to use an imaging algorithm but don't have access to a parameter estimation algorithm, you may also upload image reconstructions, from which we will extract relevant (geometric) parameters.

Please note that unless otherwise specified, all source models and data products should be kept proprietary among those currently invited to participate in the challenge, which includes all EHT Collaboration members, and any members of ngEHT science and technical working groups.

In parallel to Challenge 4, we are also running the ngEHT Forecasting Tournament. Participation in this tournament is highly encouraged for everyone! Please fill out the questionnaire here.

To get you extra motivated we now have a grand prize for the best submission, which will be a dinner for two! Submissions can come in the form of image reconstructions or results from a model fit. We are especially interested in the recovery of both geometric and physical source parameters. Submissions will be judged based on (a combination of) image reconstruction quality, number of measured parameters, and accuracy and precision of the parameter measurements.

Table of Contents

Communicating with the organizers

The primary way to talk to the challenge organizers is the private analysis-challenge-4 channel on the ngEHT Slack. If you need an invite to the Slack or to the channel, please contact Greg at lindahl ZAT pbm.com. We're also happy to help people with software installation advice.

Schedule

Downloads

Please note that unless otherwise specified, all source models and data products should be kept proprietary among those currently invited to participate in the challenge, which includes all EHT Collaboration members, and any members of ngEHT science and technical working groups.

Downloads are password protected: the username is challenge1 (yes, it's still challenge1 for the fourth challenge!) and the secret password is available if you ask the organizers, or on the analysis-challenge-4 channel on the ngEHT Slack.

Challenge 4 Downloads

Submit your results

If you have any problems with this upload, please contact Greg Lindahl on slack or email at lindahl ZAT pbm.com




Submissions

Parameter measurements

We are especially interested in your measurements (including 1-sigma errors, see below for format instructions) of the following parameters. This list is non-exhaustive, and we'd like to hear about anything else you learn from your analysis! You do not need to submit a measurement for each parameter or each dataset.

Geometric ring (n=0 and/or n=1) parameters (for all datasets):

Physical parameters (for GRMHD datasets):

With your parameter measurements, please submit a short explanation on how you got them (what analysis method did you use, with what kind of parameters, etc.). If you have additional information in the form of plots, a slide deck, etc., that is very welcome as well.

Images

If you'd like to submit images, please submit them as FITS or hdf5 files bundled in a zip file as specified below. The images can be reconstructed with any field of view or pixel resolution, as long as this is clear from the metadata. If you can, submit an image for each source model, frequency, and array. You may submit multiple images reconstructed using different methods; please follow the filename conventions as specified below. We will be using eht-imaging to load and evaluate the images, so it may be worth checking if your image loads properly in eht-imaging.

Evaluation

It would be helpful but not required to add a txt file summarizing your experience with this challenge, and any feedback on how we can improve things for a next challenge.

Format and file name conventions

Please submit your main parameter measurements in a txt file.

In your txt file, make 4 columns with parameter name, measured value, 1sigma error (or N/A), and unit (or N/A).

For example:

n=1_detection yes N/A N/A
meandiameter_n=0 42.2 1.5 uas
axis_ratio_n=1 0.95 0.02 N/A
majoraxis_pa_n=1 1.25 0.2 rad
width_n=1 1.2 0.1 uas
fractionalflux_n=1 0.05 0.003 N/A
total_circpol_fraction_n=0 0.02 0.01 N/A
beta1_stokesi_amp_n=0 0.1 0.2 N/A
beta2_linpol_phase_n=0 1.25 0.2 rad
beta2_linpol_phase_n=1 0.25 0.2 rad
mass 6.5e9 0.5e9 Msun
spin -0.7 0.2 N/A
magnetization MAD N/A N/A
rhigh 60 40 N/A

With the file name following the format below:

ch4_[model]_[array]_[frequency]GHz_[syntheticdatapipeline]_[analysismethodname]_[firstnamelastname]_measurements.txt

You can provide additional explanations in a seperate (text) file.

Please upload any images (.fits or .hdf5) using the same file name format as above.

Source models

Double m-ring model

We use a double m-ring model, which is the sum of a thicker and a thinner m-ring, representing n=0 and n=1 emission, respectively (similar to Tiede et al. 2022). The single m-ring definition is given in the paper snippet below, taken from EHT Collaboration et al., 2022d (Sgr A* Paper IV):

mring

The definition in linear and circular polarization is equivalent, except that in linear polarization there is no conjugation symmetry in the betas (the linear polarization image is complex), and in circular polarization the image is not confined to be positive. Note that we do not add a Gaussian floor component like in Sgr A* Paper IV.

In addition, we add an ellipticity to the m-rings by squishing it on one axis (or, equivalently, stretching it on the orthogonal axis from a smaller diameter). This operation consists of a scaling (axis ratio) and a position angle.

Our double m-ring model is frequency-independent apart from an overall total flux scaling.

We have used the implementation in eht-imaging to generate the m-ring model. An example double m-ring generation script has been added to the Challenge 4 release.

GRMHD time-average

This is a time-averaged image generated from the frames from a GRMHD movie at 86, 230, and 345 GHz, parameterized by a black hole mass, spin, magnetization (MAD/SANE), and electron temperature distribution parameter (Rhigh, from the prescription by Moscibrodzka et al. 2016).

GRMHD snapshot

This is a single snapshot (frame) from a GRMHD movie at 86, 230, and 345 GHz. The GRMHD snapshot may or may not come from the same model as the time average above.

Array and data synthesis

The synthetic datasets were generated with either ngehtsim or SYMBA, using the latest versions of the ngEHT Phase 1 and Phase 2 arrays, at 86, 230, at 345 GHz. The figure below shows the Phase 2 array, where we have included all sites (including the yellow sites, which are not ngEHT sites but may observe with the ngEHT) in the synthetic datasets. The SPM site is called BAJA and the LCO site is LAS in the synthetic datasets. The Phase 1 array is identical to the Phase 2 array, but does not contain the stations BOL, KILI, SGO, and SPX. Not all stations observe at all frequencies (simultaneously). Thermal noise with contributions from the receiver and atmosphere was added to the data, and the effect of frequency phase transfer (FPT) was included in setting detection limits. No amplitude gain errors and no polarimetric leakage effects were added to the ngehtsim datasets. See the data generation script released with the datasets for more specific details.

In addition to the ngehtsim datasets, we included datasets generated with SYMBA (Roelofs et al. 2020), using the same array and weather parameters. The effect of frequency phase transfer is not included here, and the datasets contain additional systematics from antenna pointing offsets and imperfect fringe fitting and a-priori amplitude calibration. No polarimetric leakage effects were added. See the SYMBA input files released with the datasets for specific settings.

array

Image: ngEHT Phase 2 reference array, taken from Doeleman et al. (2023).

array array

Image: Example uv-coverage of the ngEHT Phase 1 and Phase 2 arrays from our synthetic datasets.