\begin{figure}
\begin{center}
\begin{lstlisting}[xleftmargin=0.35cm, xrightmargin=0.25cm]
boolean isSafe = true;
if(M >= row.length) isSafe = false;
if(isSafe) {
  for(int id = 0; id < M; id++) {
    int row_begin = row[id];
    int row_end = row[id+1];
    for(int i =0 ; i < (row_end-row_begin); i++) {
      if(row_begin + i < 0 || row_begin + i >= val.length ||
         row_begin + i < 0 || row_begin + i >= col.length ||
         col[row_begin+i] < 0 || col[row_begin+i] >= x.length) {
         isSafe = false;
         break;
      }
    }
    if(!isSafe) break;
  }
}
\end{lstlisting}
\end{center}
\caption{Array Bounds Checking Code for SparseMatmult}
\label{fig:sparsematcheck}
\end{figure}

This section presents experimental results for HJ-OpenCL on two platforms.

The first platform is an AMD \textit{A10-5800K} APU. This APU includes an
AMD Radeon HD 7660D GPU with 6 Streaming Multiprocessors(SMs). The CPU of the \textit{A10-5800K}
includes 4 cores, 16KB of L1 cache per core, and 32MB of L2 cache.
Each SM in the GPU has exclusive access to 32
KB of local scratchpad memory. The
CPU and GPU can each directly access system memory, but share
bandwidth. While physical memory is shared, it is partitioned between
devices such that the CPU has ~6GB and the GPU has ~2GB.  We conducted
all experiments on this system using the Java SE Runtime
Environment (build 1.6.0\_21-b06) with Java HotSpot 64-Bit Server VM
(build 17.0-b16, mixed mode).

The second platform has
two hexacore Intel X5660 CPUs and two NVIDIA Tesla M2050
discrete GPUs connected over PCIe. There are 4GB of RAM per CPU core,
with a total 48GB of ran inside a single node. Each GPU also has
approximately 2.5GB of global memory. Only 1 of the 2 available GPUs was used at a time
to evaluate this work.  In this platform, we used the Java SE Runtime Environment
(build 1.6.0\_25-b06) with Java HotSpot 64-Bit Server VM (build
20.0-b11, mixed mode).

The ten benchmarks shown in Table~\ref{tab:benchmark} were used in our experiments. 
Four of these benchmarks use array views to perform multi-dimensional array accesses, 
and two of these benchmarks use the \nextOp{} construct for synchronization.

\noindent
Each benchmark was tested in the following execution modes: 
sequential Java, sequential HJ, and parallel HJ.  The
sequential HJ version is equivalent to the Java version except for
the use of
array views for multi-dimensional array accesses. This is intended as a test of any
overhead from the HJ environment. The parallel HJ version
employs the \hjforall{} construct to mark parallel loops which
can be run in the HJ runtime or on OpenCL devices.
%
The parallel HJ version was executed on three different platforms: 
1) on HJ's parallel work-sharing runtime in the JVM using HJ's standard code generation for SMP multicore, 
2) on OpenCL CPUs using HJ-OpenCL's code generation and runtime, and 
3) on OpenCL GPUs using HJ-OpenCL's code generation and runtime. 
In the following sections, these three
variants are referred to as HJ parallel, HJ OpenCL CPU, and HJ OpenCL GPU,
respectively. 

Performance is measured by retrieving elapsed nanoseconds from the start of a parallel
loop to the completion of all iterations of that loop. The Java system call \textit{System.nanoTime()}
was used. This measurement includes the overhead of task spawning and joining for parallel HJ. For OpenCL
execution, this includes the overhead of evaluating the \safe{} condition and any overhead from
JNI or OpenCL API calls, such as communicating data to and from the device. 
Thus, any performance difference between HJ OpenCL execution and a completely native 
C++/OpenCL implementation would be from overheads incurred to pass through JNI, which are negligible.
Our measurements do not include OpenCL initialization time for creation of OpenCL contexts and command queues.

\subsection{Performance on A10-5800K}\label{subsec:per-a10}
\begin{figure*}[t]
\begin{center}
\includegraphics[width=7in]{fig/result_fornax.pdf}
\end{center}
\caption{Performance improvements over sequential Java on \textit{A10-5800K}}
\label{fig:result-fornax}
\end{figure*}

\begin{table*}
{\small
\begin{center}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
\hline
{\bf Platform} & {\bf Blacksholes} & {\bf Crypt} & {\bf MatMult} & {\bf Doitgen} & {\bf MRIQ} & {\bf Syrk} & {\bf Jacobi} & {\bf SparseMatmult} & {\bf Spectral-Norm} & {\bf SOR} \\
\hline
HJ OpenCL CPU & 0.99 & 0.99 & 1.00 & 1.04 & 1.03 & 0.99 & 1.00 & 0.94 & 0.98 & 0.98\\
\hline
HJ OpenCL GPU & 1.02 & 0.99 & 1.00 & 1.00 & 1.00 & 1.00 & 0.97 & 0.91 & 1.00 & 1.00\\
\hline
\end{tabular}
\caption{Slowdown for exception checking on the \textit{A10-5800K}}
\label{tab:slow-a10}
\end{center}
}
\end{table*}
\begin{table*}
{\small
\begin{center}
\begin{tabular}{|c|c|c|}
\hline
{\bf Platform} & {\bf Sectral-Norm} & {\bf SOR} \\
\hline
HJ OpenCL CPU & 0.10 & 0.08 \\
\hline
HJ OpenCL GPU & 0.19 & 0.29 \\
\hline
\end{tabular}
\caption{Slowdown for without \nextOp{} construct on the \textit{A10-5800K}}
\label{tab:slow-next-a10}
\end{center}
}
\end{table*}

Figure~\ref{fig:result-fornax} shows the speedup numbers on the
AMD \textit{A10-5800K} APU relative to the sequential Java version.
%
On the AMD APU system, Blackscholes, Crypt, MRIQ and Jacobi performed as expected across all versions and platforms.
By expected, we mean that we see performance improvements for these data-parallel 
benchmarks from sequential, to parallel HJ, to native OpenCL CPU, to native OpenCL GPU.
The result shows speedup of up to 36.71 $\times$ for HJ OpenCL CPU and 55.01 $\times$ for HJ OpenCL GPU for Jacobi, relative to sequential Java\footnote{Some result shows super linear speedups because running natively in OpenCL incurs lower overhead than executing inside the JVM.}.
Doitgen, Syrk, Spectral-norm, and SOR each experience slowdown on HJ OpenCL GPU relative to sequential Java and sequential HJ.
%
Due to runtime overhead of array views sequential HJ is
slower than sequential Java for MatMult and Doitgen, 
both of which contain multi-dimensional array accesses.
%
Table~\ref{tab:slow-a10} shows the performance of 
each benchmark with exception checking using \safe{} relative to without checking (not using \safe{}) on the \textit{A10-5800K}.
Some benchmarks shows performance difference between with checking and without checking due to the overhead of evaluating the \safe{} condition.
For example, SparseMatMult demonstrates the largest overheads from performing exception checking 
on both HJ OpenCL CPU (0.94$\times$) and HJ OpenCL GPU (0.91$\times$) due to indirect array accesses in its kernel. 
Indirect accesses require checking all index values contained in arrays to guarantee no out-of-bounds array accesses. 
To illustrate this point, Figure~\ref{fig:sparsematcheck} shows the programmer-written array bounds 
checking code for SparseMatmult.

\begin{figure}
\begin{small}
\begin{lstlisting}[xleftmargin=0.35cm, xrightmargin=0.25cm]            
for (int j = 0; j < iter; j++) {
  forall (point [i]:[0:n-1]) {
    method1(i, j);
  } 
  // synchronization point 1    
  forall (point [i]:[0:n-1]) {
    method2(i, j);
  }
  // synchronozation point 2
}
\end{lstlisting}
\end{small}
\caption{HJ-OpenCL Barrier Example without \nextOp{} construct}
\label{fig:barrier-example2}
\end{figure}
\begin{figure*}
\begin{center}
\includegraphics[width=7in]{fig/result_davinci.pdf}
\end{center}
\caption{Performance improvements over sequential Java on \textit{Westmere}}
\label{fig:result-davinci}
\end{figure*}
\begin{table*}
{\small
\begin{center}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
\hline
{\bf Platform} & {\bf Blacksholes} & {\bf Crypt} & {\bf MatMult} & {\bf Doitgen} & {\bf MRIQ} & {\bf Syrk} & {\bf Jacobi} & {\bf SparseMatmult} & {\bf Sectral-norm} & {\bf SOR} \\
\hline
HJ OpenCL CPU & 0.98 & 0.98 & 0.98 & 0.99 & 1.00 & 1.00 & 1.00 & 0.97 & 1.00 & 1.02\\
\hline
HJ OpenCL GPU & 0.95 & 0.94 & 0.99 & 1.00 & 0.98 & 1.00 & 0.99 & 0.68 & 0.99 & 1.00\\
\hline
\end{tabular}
\caption{Slowdown for exception checking on the \textit{Westmere}}
\label{tab:slow-westmere}
\end{center}
}
\end{table*}
\begin{table*}
{\small
\begin{center}
\begin{tabular}{|c|c|c|}
\hline
{\bf Platform} & {\bf Sectral-Norm} & {\bf SOR} \\
\hline
HJ OpenCL CPU & 0.14 & 0.06 \\
\hline
HJ OpenCL GPU & 0.35 & 0.19 \\
\hline
\end{tabular}
\caption{Slowdown for without \nextOp{} construct on the \textit{Westmere}}
\label{tab:slow-next-westmere}
\end{center}
}
\end{table*}

Spectral-norm and SOR both make use of \nextOp{} constructs. Both benchmarks were written
in the style shown in Figure~\ref{fig:barrier-example}, a \hjforall{} loop containing barriers.
However, an alternative to using all-to-all barrier synchronizations in this style would be to
use multiple \hjforall{} loops and have the implicit barrier at the end of each take the place of \nextOp{}, as shown in
Figure~\ref{fig:barrier-example2}. However,
this approach leads to high overhead when using HJ-OpenCL.
Multiple \hjforall{} loops would also require multiple calls into JNI.
Each call to JNI includes compilation of the OpenCL kernel, transfer of data to and from the device, and execution of the kernel.
On the other hand, using \nextOp{} rather than multiple \hjforall{} loops only incurs the overhead
of additional kernel launches.
As part of our investigation, we implemented Spectral-norm and SOR in this alternative style.
Table~\ref{tab:slow-next-a10} shows the relative performance of Spectral-norm and SOR 
without the \nextOp{} relative to with the \nextOp{} on the \textit{A10-5800K}.
Both benchmarks without \nextOp{} are significantly slower (3.4 $\times$ - 12.5 $\times$) than with \nextOp{}.
 
\subsection{Performance on Westmere}\label{subsec:per-westmere}

Figure~\ref{fig:result-davinci} shows the speedups on the \textit{Westmere} platform
with two NVIDIA Tesla GPUs and two hexacore Intel CPUs.
%
For our evaluation on the Intel-NVIDIA system,
Blackscholes, Crypt, MatMult, MRIQ, Jacobi and Spectral-norm show expected performance across all versions and platforms.
The results show speedup of up to 35.68 $\times$ for HJ OpenCL CPU on Jacobi and 324.22 $\times$ for HJ OpenCL GPU on MRIQ relative to sequential Java.
For the benchmarks Doitgen, Syrk, and SOR HJ OpenCL GPU is slower than HJ OpenCL CPU due to insufficient parallelism.

Table~\ref{tab:slow-westmere} shows the performance of 
each benchmark with exception checking with the \safe{} construct relative to without checking on \textit{Westmere}.
Like the previous results on the AMD APU, all benchmarks except SparseMatMult demonstrate low overhead from exception checking using \safe{}.

Table~\ref{tab:slow-next-westmere} shows the performance of Spectral-norm and SOR 
without the \nextOp{} construct relative to with the \nextOp{} construct on the \textit{Westmere}.
Like the previous results on the AMD APU, both benchmarks demonstrate high overhead when not using \nextOp{}.

One obvious conclusion from these results is that benchmarks show very different performance
across different platforms due to the architectural features of those platforms.
As an example, consider the results on the AMD APU system.
Of the ten benchmarks, two perform best on parallel HJ, four perform best on HJ OpenCL CPU,
and four perform best on HJ OpenCL GPU. Parallel HJ provides a load-balancing runtime and avoids
added overheads incurred when running in OpenCL. HJ OpenCL CPU has the advantage of native execution
and the flexibility of the CPUs architecture. HJ OpenCL GPU also provides native execution,
as well as the massive parallelism of GPUs. It is important to note that the HJ application code requires
no modifications to switch between these platforms. This enables rapid prototyping and testing 
across a wider variety of platforms than existing programming models support.
