Kevin,
Cray MPI on their XC systems sets env var ALPS_APP_PE to the rank, unique for each rank, from 0 to N-1 for N ranks. They do not use the same env vars as MPICH, Intel MPI or OpenMPI to pass rank information down to applications.
advixe-cl run under MPI needs to open a results dir for each rank. I believe it is looking at MPICH, iMPI and OpenMPI env vars to find the rank number to use for the results directories. I am pretty sure it is not looking for Cray's env var ALPS_APP_PE. What I'm seeing is if I launch a Cray MPI job thusly:
aprun -n 1 advixe-cl --results-dir=/foodir ./a.out this works. When I run more than 1 rank I get a file open error on the results dir.
As background, Vtune used to have this problem also. They modified their collector to look for an MPI job's rank via the env vars of MPICH, OpenMPI and Cray's ALPS_APP_PE. I think advixe-cl needs this similar mod, to look for env var ALPS_APP_PE to flag an MPI job and to fetch the rank to use in the results dir name.
Could you confirm that the collector is not looking for ALPS_APP_PE to indicate an MPI job and to fetch the rank? If not, consider this a feature request to get advixe-cl to work under Cray's ALP MPI environment.
As a workaround, I aprun a wrapper script that launces multiple collectors with results dir set to <results dir>.$ALP_APP_PE. This works around the issue.
thanks
Ron