Data and Methods

In order to provide our stakeholder, the Army Research Institute, with pertinent insight regarding the feasibility of predicting behavioral performance using existing data sources we must first identify existing authoritative behavioral performance items from academic, industry, government and military sources. In essence, the collection of this corpus of documents will serve as the basis from which a consensus regarding themes and topics relating to behavioral performance will be extracted. Thus our first goal was:

To identify documents from academic, industry, government, and military sources which contain assessment scale items measuring various manifestations and/or notions of performance.

Having achieved this, we now needed to extract the individual assessment items (i.e. "questions") from each of these documents so that the terms explicitly used in the assessment items (and, presumably, the latent variables they correspond to) could be subjected to analysis. Furthermore, in order to maximally facilitate secondary analysis, additional characteristics were manually ascribed to the assessment items, based on a number of criteria. Thus, our second goal was:

For each of the documents identified in the previous step: extract each assessment item present in the document and characterize it based on (1) provenance; (2) general theme, content, and target; (3) Koopmans et al. framework features and characteristics; and (4) quality/quantity of assessment scale items.

With the assessment items extracted, we then needed to determine what common topics and themes they shared. Thus our third goal was:

Perform a text analysis on the extracted items, and attempt to characterize the topic groups and their relations

These preliminary topic modeling results can be seen under the 'Findings Tab' on our website. Due to the limitations of the size of the corpus gathered by the team this summer, we only set out to establish topic modeling as a viable technique for future work.

With the literature's themes available, we would next be able to map these back on to the data collection tools used by the military. Thus our fourth and final goal was:

Time permitting, compare these groups to assessment items and tools used by the military in order to assess overlap and determine suitability of those items for potential application to military personnel.

With a larger corpus of performance scales, the code developed by the team this summer could be adapted to meet this goal.