DANSE: Distributed Data Analysis for Neutron Scattering Experiments

DANSE was an NSF-funded project to build new software for neutron scattering science. It started with a design effort on Aug. 3, 2004. The construction phase began June 1, 2006 and ran through Nov. 30, 2011 (5 years plus no-cost extension).

DANSE was a software construction project, with Brent Fultz as P.I. As a project, DANSE had a fixed lifetime and produced deliverables, namely software for neutron scattering experiments. Later, both the software and people in the project moved on to other software development projects. This web site summarizes some of the work done under the DANSE project, and gives directions some of today's projects that had roots in DANSE.

Organization of DANSE: Project Thrusts

The DANSE project had a central group at Caltech for administration and services for software development. Most of the work done within DANSE was organized like that of the neutron scattering community, which is centered around experimental methods. The four successful subprojects were:

Paul Butler was P.I. for small-angle scattering (SANS). At the start of the DANSE project in 2004, SANS had a user base that wanted to see model fits to experimental data. A number of candidate models can be tested, and their parameters optimized, in a short amount of time with visual interactivity on a laptop computer. At the time there was a need for the analysis of two-dimensional data. Methods to work with 2-D data and orientation distributions were developed in the DANSE project, and migrated to other application software at NCNR, for example. The work goes on under SasView. All the various repos are on github under the umbrella of
SasView The main page for the project is sasview.org

Paul Kienzle was P.I. for reflectometry software. It emphasized model fitting with 1-D models, although some basic codes were developed for off-specular scattering. Different optimization algorithms for data fitting were developed and tested. It was found necessary to give the user some control and flexibility to change the optimization algorithms at different stages of fitting, or for different datasets.
Today the github repos from the reflectometry subproject include:
pkienzle/periodictable -- Scattering length density and neutron activation calculator (see also resources/activation)
bumps/bumps -- Bayesian fitting and uncertainty modeling
reflectometry/refl1d -- 1D nuclear and magnetic reflectometry fitting
reflectometry/direfl -- Direct inversion reflectometry

Simon Billinge was P.I. for the diffraction software effort, which was facilitated by prior software for pair distribution function analysis. The early versions of this code were not flexible in implementation and were not user-friendly, often requiring some changes to the source code and recompiling before analyzing a new dataset. After a successful prototype application was released with a user-friendly interface, the diffractometry subgroup developed a modular analysis toolkit that is especially powerful for advanced users.
Two main products were PDFgui and Diffpy-CMI, available on the website diffpy.org
The code, which has evolved since DANSE, is here diffpy

Brent Fultz led the inelastic scattering subgroup, which developed the data reduction software used on the ARCS spectrometer. This proved important for getting ARCS off to a productive start as operations began at the SNS. The larger effort of the inelastic scattering group was to develop software to bring computational materials science methods into use for the calculation of energy and momentum transfer between neutrons and a sample. Some of these codes are ab initio density functional theory planewave codes for electronic structure calculations, and codes for molecular dynamics. A difficulty was that fully-integrated applications were too big and complicated for users to install on their own hardware. The solution to this deployment problem was to build the virtual neutron facility (vnf) as a web service. (Web services have, of course, become prevalent in recent years.)
danse-inelastic -- This is the "github organization" that contains github repositories for quite a few danse software packages, including pyre and histogram
sns-chops/multiphonon -- This is a rewrite of the multiphonon/multiple-scattering code.
MCViNE is a Monte Carlo neutron ray-tracing program for simulating instruments or some aspects of samples in neutron scattering experiments.

Organization of DANSE: Meetings and Reviews

Nearly every Monday morning for five years, the DANSE project held developers' meetings using telephone conference calls and web-based video sharing software (vnc). The meetings, led by Mike McKerns with Michael Aivazis, developed a style where a developer from one of the subgroups gave a 40-50 minute presentation on capabilities of prototype or release candidate software, interspersed with questions from the different subgroups. Overall, the developers did an excellent job in explaining what they had done, and often posed technical questions for the group that impacted the efforts of a number of developers. After some presentations, approximately five or so, there was a followup review meeting, attended by a subset of attendees who reviewed the code and its design. More code reviews would have been helpful for the project, but generally the bugs were found by scientists using the code, typically the developers themselves, or graduate students who were making use of the code for their own research.
Mike McKerns maintains an activity in uncertainty quantification uncertainty quantification, and
Michael Aivazis keeps an activity on pyre.

There were many in-person developers' meetings and reviews with staff from NSF and the SNS. There were 18 Workshops and Developers Meetings from 2003-2011 usually at Caltech, sometimes at ORNL. Some were appended to reviews for the NSF, typically near Washington D.C. Group dinners were memorable events at these meetings, but all were funded by discretionary money, and never by the NSF.

Legacy of DANSE

DANSE had impact on both the technology and the community of neutron scattering research. The technical contributions go beyond the subfields of neutron scattering. Some links above include new software packages that were developed and tested in the context of neneutron scattering research, such as optimizers for fitting functions to data. Another contribution was a framework to set up high performance computing services on the web. Some of these tools were prototypes, and some were controversial at the time, such as web services. (Today, of course, web services are well accepted for much of scientific computing.)

In a broader context, DANSE was one of the earlier software efforts of a scientific community. The neutron scattering community came together to create a modern cyberinfrastructure. Developing new capabilities for scientific computing requires efforts from teams of scientists who can identify the types of software and hardware tools needed for their research. DANSE was an early example of such an effort. Organizing it by scientific subfields was effective for focusing technical expertise on specific problems, and for training new experts in these fields. A number of young persons in the project have carried this style, and their expertise, to new positions where they can contribute to scattering research, and find new directions well beyond the scope of the DANSE project itself.