Prospects for Wideband VLBI Correlation in the Cloud

The angular resolution of a VLBI radio interferometer is $\theta \simeq \lambda \,/{B}_{\max }$ in radians or $\theta \simeq 2\times {10}^{5}\lambda \,/{B}_{\max }$ in arcseconds, where λ is the observation wavelength, and B_max is the length of the maximum projected baseline. At a wavelength of 2 cm with intercontinental baselines of 10,000 km, an angular resolution of 0.4 mas can be achieved, whereas at the shorter wavelength of 1.3 mm with baselines of 10,000 km, an angular resolution of 20 μas is possible (Event Horizon Telescope Collaboration et al. 2019c), a resolution that is unparalleled in Earth-based diffraction limited observations.

The massive compute and storage resources provided by the cloud could handle the increased data rates and data volumes resulting from wider bandwidth VLBI observations. The scientific opportunities enabled by expanding the bandwidth of VLBI have been considered in various studies (Walker et al. 2007; Fish et al. 2013). This includes microarcsecond astrometry of high mass star-forming regions within the Milky Way (Reid et al. 2009), stellar mass black holes (Miller-Jones et al. 2009; Reid et al. 2011), pulsars and tests of general relativity (Deller et al. 2007), and satellites within the Local Group. Increasing VLBI bandwidths can also enhance the number of detectable targets, transients, and astrometric reference sources, because the number of detectable sources scales with bandwidth as N  ∝ B^3/4 (Reid & Honma 2014). Improvements in imaging gravitational lens systems to study the properties of the relativistic jets in distant Active Galactic Nuclei (Koopmans & Treu 2002) and small scale structure near the cores of lensing galaxies can be expected (Mao et al. 2001; Winn et al. 2004; Zackrisson & Riehm 2010). More sensitive observations could also allow constraining the properties of supernovae (Bartel et al. 2000) and gamma-ray bursts (Taylor et al. 1998; Bietenholz et al. 2014; Abbott et al. 2017) by resolving their afterglows as well as constraining the energetics of the explosions and the nature of the circumburst environment.

At short wavelengths (λ ≤ 1.3 mm), VLBI can resolve and image the emission near the event horizon of nearby supermassive black holes, presenting an opportunity to study general relativity in a strong gravity regime (Doeleman et al. 2009). Recent VLBI observations at 1.3 mm by the EHT have captured the first image of the 6.5 × 10⁹ M_⊙ supermassive black hole at the center of the giant elliptical galaxy M87 (Event Horizon Telescope Collaboration et al. 2019a). Previous observations have detected ∼5 Schwarzschild radii structure at the base of the relativistic jet produced at the core of M87 (Doeleman et al. 2012; Akiyama et al. 2015). Observations at 1.3 mm have also resolved structures on the scale of a few Schwarzschild radii in the 4 × 10⁶ M_⊙ black hole at the center of the Milky Way, Sagittarius A* (Doeleman et al. 2008; Fish et al. 2011; Johnson et al. 2015; Lu et al. 2018).

Geodesy is the study of the geometric shape, rotation and orientation in space of Earth and its gravitational field. Because VLBI observations can be used to determine antenna positions with millimeter accuracy, the technique is also well suited for geodetic studies. VLBI observations for geodesy are performed by the International VLBI Service for Geodesy and Astrometry (Schlüter & Behrend 2007), together with the satellite range network and the Global Navigation Satellite System antenna network (Whitney et al. 2013).

VLBI for geodesy is carried out by observing many extragalactic broadband radio sources that are uniformly distributed across the sky (Schuh & Behrend 2012). Because fast slewing antennas are required to quickly observe different sources around the sky, geodetic VLBI antennas typically have smaller diameters. To compensate for the decrease in sensitivity due to the smaller antenna diameters, higher bandwidths are needed to reach an equivalent sensitivity. To observe a uniform source distribution, it is often necessary in geodesy to observe weaker sources, which in turn also requires higher bandwidths.

Under the current VLBI Global Observing System (VGOS) operation, VLBI correlation is distributed to multiple computer clusters, where each cluster is constrained by the number of available playback units and the percentage of time allocated for geodesy.⁹ In contrast, shifting the correlation of geodetic VLBI to the cloud could provide greater user flexibility by leveraging the massive I/O and compute resources for a short period of time.

To estimate the optimal number of vCPUs per VM to use for computation, we used the n1-highmem-16, n1-highmem-32, n1-highmem-64, and n1-highmem-96 VMs, with the number of vCPUs varied from 16, 32, 64, and 96, respectively, as well as the number of stations varied from 2 to 10 stations. The 96-vCPU maximum was set by the ready availability of VMs up to that size on the GCP.

The results of computational time and cost with number of stations for different VM vCPUs are shown in Figure 2. The 16 and 32-vCPU benchmarks were only done up to 5 and 6 stations, respectively, as the computation for even more stations in those cases would have been too time-consuming. Figure 3 shows the computational time with the number of vCPUs for a five-station data set, suggesting an approximately inverse relationship between computational time and the number of vCPUs.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

To estimate the uncertainty in the measurements, the 96-vCPU, 10-station measurement was performed twice. The difference in correlation time was less than 1%. For the 10-station test data set, these results suggest that using 96 vCPUs per VM is the most efficient for data processing.

To study the scaling of correlation time and thus computational cost with the number of stations, N, we used the 96-vCPU, n1-megamem-96, 1433.6 GB VM at the us-west1 data center in Oregon. We varied the number of stations from 2 to 20. The results are shown in Figure 4. To estimate the uncertainty in the measurements, the 20-station measurement was performed twice, and the difference in correlation time between the two measurements was less than 0.3%. The measured computational time in seconds as a function of the number of stations was fit with a quadratic with a R² of 0.998.

$$\begin{eqnarray}&&t\simeq 1003{\left(\displaystyle \frac{N}{10}\right)}^{2}+1060\left(\displaystyle \frac{N}{10}\right)+285.3.\end{eqnarray} \tag{ 1 }$$

Zoom In Zoom Out Reset image size

The processing time is expected to exhibit a linear dependence on station number because many processing steps in DiFX-2.5.2 are performed once per station. A quadratic term in station number is also expected because the channel-by-channel cross multiplications that follow the FFT are done for each baseline. The nonlinear term begins to dominate at large N with a crossover point at N ∼ 11. The number of full-polarization correlation products is 4N², including auto-correlations.

The deviation from purely quadratic dependence at large N could potentially be due to each Datastream and the FXManager processes taking a significant fraction of one vCPU's capacity, effectively limiting the number of total vCPUs performing the core processing. The set-up time, which is the time from when DiFX-2.5.2 is instantiated until the first bits begin to get correlated, and the tear down time, which is the time from when the last bits are correlated to when all the processes are terminated, could be substantial for the relatively short 20 s subscans used for the benchmarks. This effect is almost independent of the number of stations. The slowdown at large N could also be due to the extra memory required because of CPU cache misses described in Deller et al. (2011). The performance at large N could potentially be improved by further optimizing the DiFX-2.5.2 correlation parameters.

We transferred 10-station VLBI data consisting of a single 20 s subscan as 20 VLBI Data Interchange Format (VDIF) files each with a size of 20 GB making a total size of 400 GB, from the GCS to n1-highmem-96, 624 GB VM to measure the data transfer rate. We then transferred the same data to two different VMs in parallel to study its effect on the data transfer rate. The results are shown in Figure 5.

Zoom In Zoom Out Reset image size

First, the aggregate data transfer rate to a single VM increases as the number of files increases until about 12 files (240 GB), after which it saturates at ∼6 Gbps. Second, the data transfer rate is not affected when transferring the 10-station data to two VMs in parallel, as shown in the right panel of Figure 5. These results suggest that it is not unreasonable to expect that VLBI data separated by scans can be independently transferred from the GCS to several VMs without a decrease in the data transfer rate, as the cloud fabric is designed to support independent resource scaling while simultaneously catering a large collection of users at any given time.

Based on the benchmark results, we describe an example cloud correlation configuration in Figure 6. In all sections that follow, we consider the processing of a 10-station dual-polarization VLBI observation consisting of a total of 200, 5-minute individual scans, constituting a total observation time of ∼16.7 hr and a total data size of 1.2 PB recorded at a bandwidth of 2 GHz per polarization. Each 5-minute scan or 20 s subscan referred to in Figure 6 already contains data from all 10 stations.

Zoom In Zoom Out Reset image size

The first step consists of staging the data from the GTAs onto the GCS. Each 5-minute scan from all 10 stations consists of 6 TB of data volume, which is too large to transfer onto any individual VM.

The second step involves preprocessing. The correlation parameters are initially setup, after which the data are split into smaller time segments or multiple frequency channels or transformed into the frequency domain for faster correlation. For instance, each 5-minute scan could be divided into smaller time segments that are small enough to load onto individual VMs. The 5-minute scans can be split into 15, 20 s subscans, each with a size of 400 GB, making a total of 3000 subscans for the entire data set. These subscans can then be transferred from the GCS to 3000 independent n1-highmem-96, 624 GB VMs in parallel. The data transfer benchmark results in Figure 5 suggest that it is not unreasonable to expect that the parallel data transfer rate will not be affected when transferring to multiple VMs with the assumption that bottlenecks are not encountered. The transfer of 3000 scans should take ∼530 s per VM. Future optimization could allow the transfer of data as it being processed, as further discussed in Section 6.3.

The third step consists of the data processing. To make the computation more manageable, the correlation could be performed using six consecutive instances of simultaneously using 500 subscans or 500 VMs in parallel. From Figure 2, the expected DiFX-2.5.2 correlation time of a 10-station, 20 s subscan with a size of 400 GB subscan is ∼2400 s per VM. The total expected correlation time for all 3000 subscans is ∼4 hr (∼25% of total exposure time).

The total time for which the data is stored in the cloud will be dominated by the time required to set up the optimal correlation parameters in the preprocessing step and the scrutinization of the correlated products afterwards. It is sometimes necessary to perform multiple trial correlations to determine the optimal correlation parameters, troubleshoot bugs in the software correlator, and fix errors in the recorded data. An advantage of cloud correlation is the faster turn-around time on running trial correlations, which could make the process of determining the optimal correlation parameters much more efficient for the correlator operator. The additional time for setting up the correlation parameters and scrutinizing the correlated data products will generally be greater for part-time arrays compared with dedicated full-time arrays like the Very Long Baseline Array and geodetic arrays such as VGOS.

Here, we consider recording directly onto to the GTAs at the telescopes with the assumption that either the current or the next-generation GTAs are reliable enough to record data at high altitudes. The alternative is to first record the data onto existing VLBI recorders and then transfer the data to the GTAs. Recording the 1.2 PB data would require twelve 100 TB GTAs, each with a usage fee of $300 with 10 free days of on-site usage. The total cost would be 12 × $300 = $3600. The two-way shipping cost of the GTAs for a 16 Gbps observation is estimated at $100 per site, or ∼$1000.

Data storage dominates the cost of the cloud correlation architecture. On the GCS, there are four types of storage classes¹⁵ : regional, multi-regional, nearline, and coldline.

Regional storage consists of data storage in only one specific region with no redundancy. If storage of data only in a single region with no redundancy is acceptable for the experiment, which can be expected to be the case for most VLBI experiments, then regional storage could be used. Regional storage costs $0.02 per GB per month. The storage cost model on the GCS is not monthly based, and per-day pricing is also available.

Multi-regional storage is designed for frequently accessed data, is georedundant (stored in at least two different geographic locations), and ensures maximum data availability, but it is more expensive than regional storage. It costs $0.026 per GB per month.

Nearline and coldline storage are low-cost storage options for infrequently accessed data, once a month for nearline and once a year for coldline storage. The cost is $0.01 per GB per month for nearline storage and $0.007 per GB per month for coldline storage. However, both nearline and coldline storage have a retrieval cost of $0.01 per GB and $0.05 per GB, respectively.

Because many radio telescopes are located at remote places such as the South Pole and on top of mountains, it is not unreasonable to expect delays in acquiring the GTAs from some stations in the VLBI array. Therefore, the cheaper nearline or coldline storage could be used for longer term storage until the GTAs from all stations have been acquired, with the caveat of retrieval costs. We consider the storage of 1.2 PB in regional storage for one month for a cost estimate of $25,166. This represents a best-case lower bound, and assumes no major delays from telescopes at remote locations. Where such delays exist, costs for GTA rental and cloud storage may be substantially higher, and nearline or coldline storage needs to be specified at a minimum.

In our benchmark operational model, correlation of data does not begin until raw data from all antennas has been transferred in full to the VMs. The data transfer of 3000 scans to the VMs in parallel should take ∼530 s per VM, which would cost $0.18 per VM with a total cost of $0.18 × 3000 = $540 for all 3000 subscans, considering a preemptible price of $1.20 per hour for the n1-highmem-96, 624 GB VM. With relatively minor changes to the underlying software and data model, however, it would be possible instead to stream data as it is being processed and remove any separate cost for the data transfer time. The DiFX-2.5.2 software correlator supports a streaming mode in which the data can be accepted via a network socket. Because data transfer bandwidth exceeds the rate of correlation, this would not impact correlation run time. Because we did not use a streaming data model for the model, however, we include the costs of transfer time in Table 2.

The DiFX-2.5.2 correlation time of a 400 GB subscan is expected to take ∼2400 s, which would cost $0.80 per VM with a total cost of $0.80 × 3000 = $2400 for all 3000 subscans, considering a preemptible price of $1.20 per hour for the n1-highmem-96, 624 GB VM. Here, we do not consider some up front computational cost of running trial correlations in the estimate.

Preemptible VMs are affordable, short-lived VMs, which last up to 24 hr. The disadvantage of using a preemptible VM is that it could be terminated at any time by the GCP if other higher priority compute instances require that particular VM. However, as the results suggest that the correlation time is ∼4 hr, and the benchmarks were performed using preemptible VMs, we expect that preemptible VMs would be appropriate for cloud correlation. Non-preemptible VMs could also be used if necessary. The correlation cost for a 400 GB subscan would be $3.79 per VM with a total cost of $3.79 × 3000 = $11,370 for all 3000 subscans, considering a non-preemptible price of $5.68 per hour for the n1-highmem-96, 624 GB VM.

We estimate the total cost for the cloud correlation of the full example observation in Table 2. For the 16 Gbps recorded benchmark data set, using preemptible VMs, the total cost is $32,706, whereas using non-preemptible VMs, the total cost is $43,646. A typical EHT experiment, however, is recorded at 64 Gbps, which quadruples the 16 Gbps costs to $130,824 and $174,584 for preemptible and non-preemptible VMs, respectively.

Equipment for this hypothetical cluster includes: 25 high-performance rack mount Linux machines, each with a dual-node with each node based on an Intel Xeon Processor with 20 cores; fast Ethernet switches and cables to tie the nodes together; 10 Mark6 playback machines each capable of delivering data at a rate of 16 Gbps; and physical infrastructure such as racking, uninterruptable power supplies (UPS) and network and power distribution cables. We have summed quoted costs for this equipment and arrive at an estimated capital investment of about $500,000. Assuming a 5-year lifetime for the cluster, the cost is amortized linearly at $100,000 per year.

In the cluster case, GTAs are not rented, so 1.2 PB of hard disk drive space needs to be purchased. We round up slightly given that the Mark6 disks have to be installed in eight-pack disk modules. At the time of writing, 10 terabyte enterprise quality Helium disk drives are selling for $330, so this scales to $42,240. Allowing for eight-pack disk modules for the Mark6, 16 modules at $500 each, the media cost is estimated at $50,000. This is likewise linearly amortized over five years or $10,000 per year.

The power consumption of such a cluster is estimated as 30 kW, and assuming an energy cost of $0.20 per kWh, the power to run the cluster 24/7/365 amounts to $50,000 per year. A widely quoted rule-of-thumb for data centers suggests roughly equal power is needed to cool the cluster. With cooling, the total annual electric bill is $100,000 per year. We assume 0.2 FTE per year of a computer systems technician ("IT" support) to assist with the assembly and maintenance of the cluster, and estimate $40,000 for this service.

Under these assumptions, the annualized cost of running the cluster sums to $250,000 assuming cost of media sufficient for a 16 Gbps experiment. Again, a typical EHT VLBI experiment is run at a bandwidth of 64 Gbps. Although scaling from 16 to 64 Gbps in the cloud quadruples the cost as per Table 2, the cluster would be more heavily utilized for 64 Gbps, but at no greater marginal cost. This is with the exception of media, for which an annualized cost of $40,000 to record four times the amount of data must be assumed. This brings the annual cluster cost to $280,000 for a 64 Gbps experiment. Thus, the annualized cost of the cluster is more than double the ∼$131,000 cost for a 64 Gbps cloud correlation.

The cluster cost estimates assume that it is paid for and run for the full year. However, it is available to be utilized on other experiments and computation in general when not running VLBI correlations. Making the assumption of perfect utilization at other times—which assumes excellent operations management—it is appropriate to prorate the annual cost of the cluster for the time it is actually used during the year.

We estimate that a 64 Gbps, 10-station, 4.8 PB experiment would require about 11 days of continuous run time for correlation on the hypothetical 1000-core cluster. Allowing for setup and disk changes, as the cluster only reads and correlates 16 Gbps per run, this might reasonably translate into about a month of wall clock time or about 8% of the year. If cluster costs are prorated for 8% of the year—though not prorating the costs of media—the cluster is then the more economical choice at $59,200.

We also consider the possibility of prorating media, which can be recycled through multiple observations, as is common in VLBI. However, the 8% factor used to prorate the cluster for a month of use is not appropriate in the case of media. The Mark6 disk packs, for instance, are shipped to the sites well ahead of the observation and are conditioned with read-write cycles there, which is a time-consuming process. After the observation, they are packed and shipped back to the correlator for processing. The data continues to reside on the Mark6 media modules until the correlation job is completed, for which we have assumed one month of clock time.

We estimate two additional months on each side of the observation, to pre-check, pack and ship the disks to the site, including clearing customs; to condition them and then run the observation; and then pack, ship, unpack and stage back at the correlator. In principle, the cluster can be used for other processing jobs during the observation and shipping time, but the disks are not available to recycle into other observations. With the one month of processing in addition to the two months for shipping and staging, we prorate disks to three months, or 25% of annual. Of course, other observation opportunities are available at exactly the right times for the media to be reused four times a year. Under this assumption, the cluster cost is lower still than the cloud for a 64 Gbps observation.

Table 3 provides a summary of the three types of scaling relations used for the cluster cost estimates in the prior discussion:

• 1.  
  amortization: the capital investment considered over an assumed five-year lifetime of the equipment.
• 2.  
  bandwidth scaling: our benchmarks are for a 16 Gbps recording while a typical EHT wideband experiment is recorded at 64 Gbps.
• 3.  
  prorating: the sharing of capital investment to the benefit of multiple projects.

Table 4 provides a cloud and cluster cost comparison for the correlation of 4.8 GB data set recorded at 64 Gbps, where the cluster cost is considered under three cases: no prorating, 8% cluster prorating, and 8% cluster and 25% media prorating. We note that the primary benefit of the cloud is not the cost, but rather the ability to deploy much larger computational resources than practically possible on a cluster to run the computation much faster and thereby shorten the time-to-science. This in turn implicitly improves the effective utilization of the entire astronomical instrument, not merely the correlator. Deploying more cores for a shorter time in the cloud does not significantly affect the cost of the computation. To the contrary, running the computation faster with a greater number of cores reduces the costs of regional storage, which are the dominant cloud costs. However, it is not unreasonable to expect the storage costs on the cloud to decrease with time.

Prorating the cluster and the disk packs is only valid if the rest of the year the equipment is effectively utilized on other compute jobs (cluster) and observations (media). In practice, the prorating is almost certainly overly optimistic, especially in the case of media, where observations have to become available on precisely a three-month cadence for the 25% prorating to be achieved. The prorating calculation is useful as a guide to what is possible in principle. Further, the cluster estimates presented in this subsection are intended only as a rough basis for comparison, with a proper cost comparison being highly assumption dependent, rather complex to put together, and dependent on cost figures which vary greatly over time.

Viewing utilization actually achieved as an economic parameter, there exists a crossover point between the cloud and cluster cost. For low duty cycle wideband observations where substantial cloud resources are strategically deployed, the dominant storage costs for cloud are reduced. We note that when not utilized, cloud resources are simply released back to the provider, whereas for the cluster, efficient utilization is a significant logistical problem for the operating institution.

In practice, corporate cloud service providers may provide services at a significant discount, or even gratis, to scientific research groups and non-profits. This practical factor could dramatically affect the comparative retail costs assumed here, substantially affecting the cost crossover point between the cloud and the cluster.