Novo-G# Large-Scale Reconfigurable Computing with Direct and Programmable Interconnects

Novo-G#: Large-Scale Reconfigurable Computing with Direct and Programmable Interconnects Abstract—While High-Performance Computing is ever more pervasive and effective, computing capability is currently only a small fraction of what is needed. Three fundamental issues limiting performance are computational efficiency, power density, and communication latency. All of these issues are being addressed through increased heterogeneity, but the last in particular by integrating communication into the accelerator. This integration enables direct and programmable communication among compute components. Novo-G# is a large-scale FPGA-centric cluster being built to investigate and develop architectures, system and tool infrastructure, and applications for this model. In this report we discuss the motivation behind and particular objectives of Novo-G#, the work completed so far, the products of that work, and their potential impact. We end with a description of and an invitation to join the Novo-G# Forum, the project users group. I. I NTRODUCTION The overall goal of the Novo-G# project (pronounced NovoGee sharp) is to develop and demonstrate innovations that give the high-performance computing (HPC) user community the capability to conduct transformative research via scalable, cost effective, high-performance, general-purpose systems built from off-the-shelf components. The particular objective is to build a compute cluster and related infrastructure that facilitates computational science research and advances supporting computer systems. Specifically, Novo-G# is an enhancement of Novo-G, an accelerator-based reconfigurable compute cluster developed and housed at the University of Florida. A key aspect of the project is addressing the problem of strong scaling in high-performance computing through acceleratorcentric clusters, that is, clusters where the accelerators themselves are the central components and communicate directly with one another. The specific accelerator technology we are investigating is the FPGA: they are currently the only COTS component that combines innate communication support, highcompute capability, low power, and an installed application base. This work was supported in part by the National Science Foundation through Awards CNS-1405695, #CCF-1618303/7960, EEC-0642422, and IIP1161022; by a grant from Microsoft by Altera through donated FPGAs, tools, and IP; by Gidel through discounted FPGA boards and daughter cards; and through tools provided by Mirabilis Design. This article is a progress report and outreach activity of the Novo-G# Community Resource project. All intellectual contributions have appeared elsewhere as cited in the text. The construction, demonstration, and dissemination of Novo-G#, together with building and organizing the associated communities, are the primary activities of this project. Novo-G# is designed to support research projects in various disciplines. These areas can broadly be classified into applications research (molecular dynamics, graph processing, bioinformatics, and artificial neural networks), and systems research (Exascale emulation, reconfigurable networks, and application-specific routing). Our goal is to reduce the barrier to entry of multi-FPGA design and enable researchers to use Novo-G# and similar systems as a platform for accelerating their research. In this report, we will discuss the motivation and particular objectives of the project, the work completed so far, the products of that work, and their potential impact. We end with a description of and an invitation to join the Novo-G# Forum, the project users group. II. C ONTEXT A. Motivation and project overview Together with theory and experiment, computer simulation now constitutes the third pillar of scientific inquiry, enabling researchers to build and test models of complex phenomena that either cannot be replicated or would be prohibitively expensive to be replicated in the laboratory. Applications range from the practical, such as designing more efficient aircraft and effective drugs, to basic research in understanding the molecular basis of diseases such as Alzheimer’s. Yet computing capability is currently only a small fraction of what is needed: e.g., detailed biological simulations are limited to small numbers of macro-molecules; additional factors of millions are needed to simulate cells and far more than that for larger structures. Three fundamental issues limiting performance are computational efficiency, power density, and communication latency. All of these issues are being addressed through increased heterogeneity, but the last in particular by integrating communication into the accelerator. This integration enables direct and programmable communication among compute components. Direct links enable the bypassing of CPU, network interface, and even device memory. Programmable communication enables data transfers to proceed with high efficiency even under substantial loads. The Novo-G# infrastructure consists of the physical system and hardware, but also software and configurations, existing and under development, to enhance both general usability and the enabled research projects. Another aspect of this infrastructure, as with the Novo-G, is the community of collaborators (the Novo-G# forum) who are contributing applications, tools, evaluation, and feedback. The end-goal of this project is to advance the capabilities of scientific computing. The nearer-term goal over the next few years is to provide a system testbed for transformative research in a variety of areas in Computer Science and Engineering including programmable network components, processor/network interfaces especially for accelerators, FPGA-based systems, applications in reconfigurable computing, architecture of clusters with direct and programmable communication, and libraries and tools to support such clusters. B. Previous and current work As of 2016 we are in an era where large FPGA clusters are not yet commonplace—outside niche areas such as circuit simulation, high-frequency trading, and security—but have been well-studied with perhaps dozens of examples. The following is a by-no-means exhaustive list; we have somewhat arbitrarily begun this survey from 2004. An early FPGA cluster with 96 FPGAs was the Heterogeneous HPC computer (HHPC) at the Air Force Research Laboratory/ Information Directorate (AFRL/ID) Distributed Center [1]. Large research systems were also built at the University of Edinburgh (Maxwell) [2], UNCC by the Sass Group (RCC) [3], and Imperial College (Cube) [4] and (AXEL) [5]. COPACABANA is a large cluster of FPGAs built and commercialized for cryptanalysis [6]. The RAMP series of projects used FPGA clusters, e.g., for circuit simulation [7]. IBM has also been using FPGA clusters for circuit simulation [8]. Compared to the above, Novo-G# uses high-end Stratix V FPGAs and a 3D interconnect that can provide an aggregate bandwidth of 240 Gbps to and from each accelerator. Furthermore, we aim to upgrade Novo-G# in the near future with a new generation of FPGAs that will also feature hardware floating-point units. Some FPGA cluster projects have had an emphasis on exploring issues with direct FPGA-FPGA communication, including RCC [3], and by a group at UCSD [9]. Bluehive is an FPGA cluster with direct communication built for modeling ensembles of neurons [10]. The Grape series of ASIC- and FPGA-based clusters has targeted N-Body and Molecular Dynamics most recently with the MDGrape4 [11]. Other projects have investigated issues in FPGAbased communication including work at Toronto (TMD-MPI) [12] and Los Alamos (PetaFlops Router) [13]. In general, the use of FPGA in commercial routers is well-known [14] with Arista recently giving access to the internal FPGA [15]. NetFPGA is an educational platform for Gigabit switching and routing [16]. Novo-G# shares many of the advantages of these systems, but is also designed to be flexible at all levels of the communication and application stacks. With Novo-G# we aim to explore existing communication IP, network architectures, and application-aware communication protocols, therefore we have designed the system to be as reconfigurable as possible. More recently, the Catapult system [17], deployed in their Cloud by Microsoft, encompasses 1632 servers with one midsized FPGA per server. FPGAs are interconnected directly through their transceivers in 6x8 tori. Catapult has been demonstrated to be cost-effective in a non-traditional application for FPGAs, page ranking, and also for security and deep learning. Extensive work was done to demonstrate not only performance, but holistic cost-effectiveness: connectivity, resilience, form factor, power, and cooling. Being an experimental system, Novo-G# is better suited for exploring cluster architecture as it is not wedded to any particular application or system integration. As a major goal of the Novo-G# project is integrated communication and computation, we very briefly mention some of the prior work in that area. Certainly prior to the microprocessor, especially in the SIMD arrays of the 1980s, there was little distinction between the two. Dally’s J-Machine (and its successors) was premised on communication latency of just a few cycles [18]. Physicists have long been building such systems for QCD, including the QCDOC [19], which was the basis of the first BlueGene [20]. Anton (now Anton 2) integrates communication and computation for Molecular Dynamics [21]. III. S YSTEM A. Background and overview Novo-G [22], [23] began in 2009 as an effort to create a research cluster using high-density FPGA boards to accelerate scientific applications. The original machine began with a head node and 24 Linux servers, each featuring a quad-FPGA board from Gidel [24], for a total of 96 Altera Stratix III E260 FPGAs. Over subsequent years, the machine has been upgraded annually and now stands at 192 Stratix III E260 FPGAs in 24 servers, 192 Stratix IV E530s in 12 servers, 64 Stratix V GSMD8s in 16 servers, and a second set of 64 Stratix V GSMD8s in 16 servers under construction. Each server features dual Intel Xeon multicore processors. Server connectivity is provided by gigabit Ethernet and DDR/QDR InfiniBand within the system, and a 10 Gb/s connection to the Florida LambdaRail. At CHREC (NSF Center for High-Performance Reconfigurable Computing), Novo-G has been used for a variety of application acceleration projects from the domains of bioinformatics [25], image processing [26], and financial computing [27]. The common factor among the above applications is that they are embarrassingly parallel and can therefore scale almost linearly with the available hardware resources. A greater challenge is accelerating communicationintensive applications like Molecular Dynamics. Traditionally, such communication makes use of centralized networks such as Ethernet or InfiniBand, and entails multiple interactions between the FPGA and the host. The increased latency, and communication bottleckneck in such applications emphasize the need for a better solution. The Novo-G# system is part of our effort to create an FPGA cluster that can handle communication-intensive applications. In keeping with that theme, the system features ProceV boards, which are PCIe-based accelerator boards from Gidel populated with Stratix V GSMD8 FPGAs from Altera. The GS-series devices are optimized for high performance, high bandwidth applications with support for up to 36 on-chip transceivers that can operate up to 12.5 Gbaud. Each FPGA is connected to two 8GB DDR3 SODIMM and two 36-Mbit SRAM memory banks and communicates with the host CPU via PCIe v3. The FPGAs are housed in a 4U chassis with two Xeon E5-2620V2 (Ivy Bridge) processors per server. The servers themselves are interconnected via Gigabit Ethernet and QDR InfiniBand. Our FPGA platform vendor Gidel has provided invaluable assistance by designing a custom daughterboard that allows external access to 24 high-speed transceivers. The transceivers are grouped into six bidirectional links, each link consisting of four parallel channels, enabling the construction of a 3D torus of arbitrary size. Physical connectivity between the boards is provided by a COTS CXP-3QSFP+ split cable that enables each FPGA to be connected in six different directions. Initial deployment of the Novo-G# system was completed in the second half of 2014 with 32 Stratix V boards housed in eight chassis and supporting up to a 2 × 4 × 4 torus, and an upgrade to 64 nodes (4 × 4 × 4 torus) was completed in August 2015. C. Protocol stack A part of the hardware resources on each FPGA is used to implement a network stack that services the 3D torus network. The network stack is responsible for accepting data from the application logic, packetizing the data, routing packets across the 3D torus network by the shortest route, and delivering the data to the application logic at its destination. These functions represent a subset of the services provided by the lowest three layers of the OSI reference model (i.e. physical, data link, and network layers). Figure 1 depicts our initial implementation of the 3D torus network stack on Novo-G#, and the network services associated with each component. The IP cores used for the transceiver interfaces are primarily supplied by Altera and interface directly with hardware resources attached to each transceiver channel. The remaining blocks are implemented as RTL code and therefore do comprise a resource overhead for each FPGA. Data generated by the application logic is Trans- -X ceiver 256 Ext. Rx Trans- -Y ceiver Ext. Rx Trans- -Z ceiver Ext. Rx Trans- +X ceiver Ext. Rx Trans- +Y ceiver Ext. Rx Trans- +Z ceiver Ext. Rx FPGA app_clk domain router_clk domain Int. Tx. xcvr_clk domain Ext. Tx Internal Router -X Transceiver 256 256 Ext. Tx -Y Transceiver Ext. Tx -Z Transceiver Ext. Tx +X Transceiver Ext. Tx +Y Transceiver Ext. Tx +Z Transceiver 256 Ext. Rx.: External Receiver Int. Rx.: Internal Receiver Ext. Tx.: External Transmitter Int. Tx.: Internal Transmitter 10 Gbps Int. Rx. xcvr_clk domain Negative Router Positive Router Services Provided B. Architecture App logic 10 Gbps The Novo-G# system [28], which has been under development for the last two years, features high-density Stratix V FPGAs connected by a 3D torus network that connects the FPGAs, enabling direct, low latency, and high-speed communication among the FPGAs. The hardware is supported by an easy-to-use, but efficient, protocol stack that packetizes, routes and buffers data, allowing coomunication-intensive, multi-FPGA applications to be developed rapidly. We have also recently added support for OpenCL-based, multi-FPGA apps that can also utilize the inter-FPGA communicaiton links. Network layer Data-link layer Physical layer Dimension-order routing Source data buffering Physical addressing Packet switching Data framing Clock recovery Line coding Multi-lane sync. Fig. 1. Novo-G# node architecture detailing the 3D torus network stack. Services provided to the user through RTL or third-party IP are also shown. packetized by the internal receiver block and stored in a FIFO. Similarly, the external receiver blocks accept packets or streaming data from the transceiver IP. One or more routers are used to route packets from the receiver to the transmitter blocks, which transmit the data further along the network or to the application at the destination node. Traditionally, the router block is the bottleneck in such network architectures and much research has been done on optimizing on-chip network architectures. In our previous work on Novo-G# performance prediction [29] we observed that the internal bandwidth provided and latency incurred by the network architecture has the greatest effect on execution time. Our approach uses a centralized single-level router network that can be customized based on hardware resources available and expected utilization of the 3D torus network. Figure 1 shows one such configuration with three routers, each servicing a different subset of output ports. In this manner more routers can be instantiated (up to the number of output ports) to increase the internal bandwidth available without increasing latency, but at the expense of hardware resources. Further optimization of the router and network design will be explored in future work. The transceiver blocks in Figure 1 are low-level IP blocks provided by Altera that instantiate various portions of the hardwired transceiver channel. In our aim to make Novo-G# as flexible and reconfigurable as possible, we provide support for the use of the Low Latency, Custom, and Interlaken PHYs from Altera, which can be substituted without affecting the rest of the network architecture. The Low Latency PHY IP provides the lowest inter-FPGA latency, but has no word alignment or DC balancing features, making it difficult to use without additional hardware. Conversely, the Interlaken PHY IP does automatic word alignment, 64b/67b encoding, scrambling, and multi-lane synchronization, but the interFPGA latency is the highest, and varies due to the addition of framing words into the datastream. The Custom PHY provides word alignment through 8b/10b encoding and considerably lower inter-FPGA latency than the Interlaken PHY, but the 25% overhead of 8b/10b encoding only makes this PHY useful for small packets. D. Alternative routing support +x -x +z Input Handler Input Handler 7 -z -y +y …... Input Handler 7 …... Output handler 7 …... (a) Local Processing Unit …... We now describe another of our routing mechanisms. Our goal is to create general infrastructure that supports a variety of routing modalities. For example, for a specific application, its decomposition and communication is often known a priori (see, e.g., [30]). Therefore the routes and timing of each packet could be optimized prior to execution, i.e., statically or offline. We adopt table-based routing to support offline routing [31], [32] including collectives [33]; we show this and the internals of the switches. 1) Routing Mechanism: Table-based routing can be implemented in two ways: source routing and node-table routing [34]. Since source routing needs to carry the table indexes along with the packets, which consumes extra bandwidth, we adopt node-table routing. The routing table preserves a table entry for each incoming packet (see Figure 2). Node B sends it to the Y- port of Node D, at which point it is ejected. Similarly for the second fan out, the packet is routed to the Y- port of Node C, where it is ejected. Table-based routing enables implementation of any oblivious routing scheme by simply entering values in the routing table. Examples include simple patterns, such as dimensionorder routing, and complex patterns, such as certain optimal routes that achieve congestion-free communication even under heavy loads [35]. 2) Switch Architecture: Our switch architecture is based on the classic four-stage pipelined Virtual-Channel switch [36]. By adding support for multicast and reduction, the four-stage pipeline is extended to seven-stage pipeline. Its architecture is illustrated in Figure 3 …... Output handler …... …... Output handler +x -x +z -z -y +y Node A ; Node D ; Unicast Table on Local Port Type Exit Index 0 Unicast -X 4 1 Unicast +X 1 2 Unicast LOC 0 Processing unit Processing unit Fig. 2. The table-based routing scheme In this example, node A dispatches a multicast packet that carries three fields: packet type, table index, and payload. The router routes the packet to either a unicast, multicast, or reduction table based on the packet type. In the corresponding table, multicast in this example, the router looks up the table entry based on the index field in the packet. The multicast table entry has slots for each of the six possible fan outs. In this example, the multicast packet has two fan outs. For the first fan out, the table entry shows that it is a unicast packet, that it should be routed to the X+ port, and that its table index for next node is 2. The router then generates a unicast type packet and routes it to the X+ port, after which it goes to Xport on node B. On node B, since it is a unicast packet, the router looks up in the unicast rather than the multicast table. The table entry shows that this packet should be routed to the Y+ port of node B. In the same manner, the router on the If reduction packet Reduction unit 7 Mux Index: 0 Payload Index: 0 Payload (b) 7 VC + _ Node C Input buffer PriorityIf multicast based packet Packet Assembler Comparison Mulitcast unit 7 + _ < < Unicast Table on YPort Type Exit Index 0 Unicast X3 1 Unicast LOC 0 2 Unicast Z+ 1 + _ Unicast Index: 2 Payload VC VC + _ < Unicast Idx:1 Payload VC VC + _ ; VC VC + _ ; Unicast Idx:2 Payload 7 Routing table …… < Node B Unicast Table on XPort Type Exit Index 0 Unicast Y+ 7 1 Unicast X+ 3 2 Unicast Y+ 2 Output handler …… Multicast Idx:1 Payload Multicast Table on Local Port Processing Type Exit Index unit Unicast X+ 2 1 Unicast Y+ 1 ` Input Handler Mux (c) Fig. 3. Switch architecture: (a) The switch is connected by seven input handlers and seven output handlers. (b) The input handler has four stages: input buffer consumption, routing table lookup, multicast table lookup, and virtual channel allocation. (c) The output handler has three stages: switch allocation, reduction table lookup, and reduction table write-back. In the classic switch, there are four stages: routing computation (RC), virtual channel allocation (VA), switch allocation (SA), and switch traversal (ST). The first change is that we divide RC into three stages: input buffer consumption, routing table lookup, and multicast table lookup. When a packet is injected into the switch input buffer, we spend one cycle to fetch it from the buffer. By examining its header, we get its routing table index so in the routing table lookup stage, we find its routing table entry. We then know whether it is a multicast packet or not. If it is, then in the next stage we look up its multicast table entry, based on the index from the routing table. Our switch also contains a VA stage, in which we allocate VCs to all the multicast children generated in the previous stage. If VA fails, then the router generates backpressure to stall the pipeline. In the SA stage, there might be multiple packets (up to 7) requesting the same output port. The packet with the highest priority wins the contention. The priority field is attached to each packet at the routing table lookup stage; our priority scheme is typically farthest-first. Another change from the classic switch is that we divide ST into two stages: reduction table lookup and reduction table write-back. If the packet is not a reduction packet, it spends two cycles going through the two stages and leaving the switch. If the packet is a reduction packet, it is routed to the reduction unit, which contains a reduction table. We allocate one entry in the table for each reduction operation. In the reduction table lookup stage, the reduction packet finds its corresponding entry. It then checks whether the expected number of downstream packets has arrived or not. If not, then the reduction operation is executed, and the reduction table entry is updated. If all of the expected downstream packets have arrived, then the reduction unit dispatches a new packet and injects it into the upstream link. IV. S AMPLE A PPLIC ATION dispatching FFT data to available IP blocks, and consolidating 1D FFT outputs into packets for transmission to other FPGAs. The design has been implemented on 8 FPGAs connected in a 2×2×2 network configuration and the performance measured and validated using the UF Novo-G# performance model. On scaling up the system size to 64 nodes, the model predicts a speedup of 6-19 times that of a BlueGene/Q system of the same size, for FFT sizes ranging from 32 × 32 × 32 to 128 × 128 × 128 elements. V. W ORK IN PROGRESS In this Section we give an overview of a selection of the many projects currently under way under the direction of the authors and collaborators. A. OpenCL The push for better HLS tools for FPGAs in the past few years indicates that programmability and usability have become important issues for the RC community to address. Altera OpenCL (AOCL) has presented an end-to-end solution that tries to address both issues. This tool allows GPU-based OpenCL code to be readily ported to FPGAs, while also simplifying the FPGA design process and reducing turnaround time. While Altera OpenCL currently only provides support for single-FPGA design, we are in the process to extending it to support our 3D torus network, and thereby enable productive use of the Novo-G# network. This extension would also allow Novo-G# users to leverage the existing OpenCL framework and tools to verify, debug and profile their designs. By extending the existing capabilities of AOCL on the ProceV boards, we have created a multi-FPGA OpenCL framework that allows AOCL kernels to communicate with each other within the OpenCL model. We have also established and tested an MPI-based framework to run AOCL programs on multiple OpenCL devices. Finally, we have benchmarked and evaluated the framework through three varied case studies and devised the most effective manner in which to make use of the channels efficiently, achieving a measured data rate of 24 Gbps on every inter-FPGA channel. There are a number of improvements that can be made to this framework in the future. Adding the ability to tune the transceiver parameters at runtime could potentially allow the inter-FPGA channels to operate closer to the 40 Gbps line rate. Additionally, integrating the network-layer router from [28] would allow for transparent packet routing within the 3D torus network. Both of these improvements, however, are unlikely to fit within the established AOCL model or the OpenCL 2.0 specification and would therefore require significant changes to the AOCL hardware and runtime to support them. The 3D FFT is a core application across diverse scientific and engineering domains. Of particular interest to the NovoG# project, the 3D FFT can be on the critical path for long timescale Molecular Dynamics (LTMD) simulations on large clusters [37]. Since MD has previously been shown to be highly amenable to acceleration with single FPGAs [38]–[40], demonstrating the efficiency of the 3D FFT on FPGA-centric clusters is a critical step in implementing LTMD on the NovoG#. The difficulty with accelerating the relatively small 3D FFTs used in LTMD, typically 323 to 1283, is scalability. Few published results show much speed-up beyond a small number of processors; one exception is the ASIC-based Anton processor family, which was designed with this application in mind [37]. We show that the Novo-G# can perform comparably to Anton, and an order of magnitude better than the best published results for non-ASIC-based clusters. Our previous work proposed a generalized mapping method for the 3D FFT of an arbitrary size onto FPGA clusters with arbitrary numbers of nodes. By knowing the routing information, it is possible to schedule each packet to minimize the congestion in the network. The simulation results, and the comparisons with other technologies, are shown in Table I. The results demonstrate that even with conservative simulation settings (see [31], [32] for details), the Novo-G# 3D FFT is able to achieve the same order of magnitude latency as the ASIC implementation [37]. In the UF version, the IP cores are fed data by a Nios II soft-core processor. The processor is also connected to the internal transmitter and receiver nodes, and is responsible B. Middleware integration Another approach is to provide middleware support for for aggregating FFT inputs coming in from other FPGAs, FPGA developers. Again, FPGAs offer attractive power consumption, flexibility, and performance compared with CPUs and GPUs and much lower cost than ASICs. But programmability and portability are still obstacles: it is often challenging TABLE I R ESULTS FOR VARIOUS TECHNOLOGIES AND PROBLEM SIZES Model Blue Gene/L Intel Nehalem NVIDIA Tesla Altera StratixIII DE Shaw Anton Intel Sandy Bridge NVIDIA Kepler Xilinx Virtex-7 Altera StratixV Implementation Technology Parallelism Date Code 2005 era technology 512 or 4096 04/Q4 FFTW 2008 era technology 4 cores 09/Q1 MKL 240 SPs 08/Q3 CUFFT single FPGA 08/Q2 report 512 PEs 08/Q3 report 2012 era technology 8 cores 12/Q1 MKL 2688 SPXs 12/Q4 CUFFT single FPGA 12/Q2 report 64 or 512 FPGAs 12/Q3 report 163 Perf. in μs 323 643 NA 100 200 38 54 4.5 NA 116 66 NA 4 983 257 NA 13 22 25 3.6 1.63 55 29 21 2.24 288 92 216 6.72 Anton is fixed point, other results are for single-precision floating point. All times are in microseconds. Release dates are from corporate announcements of availability in quantity. Stratix V times are our simulation results calibrated with our prototype cluster. Anton results are from [37]. Blue Gene/L results are from [41]. All others are from our previous work [32], [42]. to get a complex application working on a single FPGA, let alone a large FPGA cluster. The LEAP environment from Intel [43] is one of several active projects that builds an FPGA OS that allows users more focus on algorithm level while retaining full flexibility in the design. LEAP builds on the idea of latency-insensitive channels (LI) as the primary communication primitive to abstract the communication inside one FPGA and among multiple FPGAs. Current work by one of the authors (Sheng) has the goal of providing a similar interface that hides the low-level details in inter-FPGA communication, but includes flexibility in modes of synchronization, timing, and handshaking. The link should be as transparent as a UNIX-style pipe to users. Our current progress, described in Section 3, has already facilitated part of this goal. The difference is that the latency is not insensitive; for HPC applications, we do care the latency and so we would like the latency to be manageable by users. One solution is to provide several communication link abstractions that provide different latencies and QoS. For example, a heavyweight abstraction ensures lowest error probability, but with longest latency, while a lightweight abstraction provides the lowest latency but the QoS is not 100% reliable. In the latter case, some error correction at a higher level would be necessary. Another project focuses on portability by integrating the higher level routing mechanism (e.g., the global communication described above) with LEAP or other layer that facilitates abstract pipes. VI. O UTREACH : T HE N OVO -G# F ORUM Given recent advances, such as the introduction of FPGAs into the cloud [17], we are entering a time where clusters such as Novo-G# will be ubiquitous. We have organized the NovoG# Forum based on the highly successful Novo-G Forum model, which had over 20 members across four continents, as a mechanism to foster participation and collaboration. The Novo-G# Forum is comprised of an international group of academic researchers and technology providers working collaboratively on applications and tools to establish and showcase advantages of reconfigurable supercomputing at scale with unprecedented levels of performance, productivity, and sustainability. Faculty and students in each academic research team can contribute innovative applications and/or tools research on the Novo-G# machine based on their unique interests. This commitment will consist of four steps. 1) One or more FPGA boards of the type in Novo-G# will be procured for local research use via help from Altera (donated or discounted devices) and Gidel (discounted boards). 2) One or more promising applications and/or tools activities will be completed and optimized for maximum speedup (versus a common CPU reference) on the local board(s). 3) The preceding achievements will then be moved and scaled onto multiple nodes in the Novo-G# machine. 4) Finally, applications successfully optimized for Novo-G# will be measured (in terms of performance, productivity, and sustainability) and compared versus conventional supercomputers to showcase overall achievement. Each technology provider will provide equipment and/tools (as donation or at discounted cost) to the academic research teams for this effort, including a reasonable level of technical support for their products, and directly participate in forum activities and discussions as they choose.