Methodology – Justice Navigator

Understanding CPE Methodologies

What do Justice Navigator assessments show?

Justice Navigator assessments support CPE’s mission to reduce racial disparities in policing through data. By combining policing data with demographic and crime data, we assess evidence of racial disparities in recorded police contact with members of the public. We use data on pedestrian stops, vehicle stops, use of force incidents, calls for service, and officer-initiated activities to identify disparities and draw insights on factors that may be associated with any disparities observed.

What does CPE mean by “racial disparities”?

Our findings describe two potential types of racial disparities: racial disparities in rates of contact and racial disparities in the outcomes of this contact. Racial disparities in contact exist when members of one racial group experience proportionally more police contact than members of other groups. Racial disparities in outcomes exist when the likelihood of a police encounter resulting in a given outcome (for example, a vehicle stop resulting in an arrest) differs across racial groups.

What does it mean if an assessment finds evidence of racial disparities?

Findings of racial disparities mean that groups of people in a community are having different experiences with policing, and some communities are subject to greater burden and harm than others. Identifying these disparities through data is critical to acknowledging and addressing the negative experiences people in vulnerable communities have had with police for generations that have too often been ignored. Establishing an understanding that racial disparities are present is an important first step to collaborative change. Additionally, analysis of racial disparities can shed light on specific sources of frustration and risk of harmful outcomes in communities, which is key to identifying effective solutions. As described below, we examine potential sources of racial disparities that include not only departmental policies and officer behavior, but also factors outside of a department’s direct control, such as poverty and crime rates in a neighborhood.

What factors might be associated with observed racial disparities?

The Justice Navigator Analysis Plan—a framework for collecting and interpreting the data used in our analyses—aims to distinguish among five sets of issues that tend to be related to racial disparities in policing:

1 Individual characteristics or behaviors.

Attributes or behaviors of individual community members, such as mental health challenges, experiencing homelessness, or participation in activity that is criminalized, may lead to a greater risk of interaction with law enforcement.

2 Community characteristics.

Neighborhood conditions, such as poverty or high crime rates, may result in higher rates of interaction with law enforcement.

3 Individual officer characteristics or behaviors.

Some officers may view members of certain communities with a higher level of suspicion, resulting in a disproportionate rate of stops or more punitive outcome of the stop for these people.

4 Departmental culture, law, or policy.

Institutional policies, practices, or norms may increase law enforcement contact with some members of the population more than others. For example, officers may be deployed to patrol some communities more intensively than others, or federal, state, or local laws may contribute to disparate interactions with people and communities.

5 Relationships between communities and police.

Mistrust of law enforcement can reduce community members’ willingness to cooperate with police. Similarly, a sense that communities do not trust or respect police may cause officers to feel unsafe or defensive in encounters with members of those communities.

We recognize that these five factors are closely interlinked, and the whole story behind any observed racial disparity likely incorporates some of each of these factors. The findings presented in an assessment use police administrative data provided by the department and census data (from the Census Bureau’s American Community Survey five-year estimates) on the resident population of a jurisdiction to examine the role that some individual behaviors (explanation 1), community characteristics (explanation 2), and police factors (explanations 3 and 4) may play in any observed racial disparities. While it is not possible to completely isolate the role of each of these factors in contributing to disparities, as described below, our analyses use the best available science to inform our approach to producing actionable findings for police departments and communities.

What specific research questions do assessments attempt to answer?

Using the above analytic framework, we produce a series of analyses in each Justice Navigator assessment that broadly aim to answer the following questions:

1 Are there racial disparities in who is subjected to police force?

After accounting for certain factors, including poverty levels and crime rates, what level of observed disparities in use of force incidents remains?
How much of the frequency of use of force can be explained by these factors?
Are certain force types more commonly used against people in different racial groups?
Do specific work units contribute differently to observed racial disparities in use of force incidents?

2 Are there racial disparities in who is searched at vehicle stops?

Are there differences in how likely drivers in different racial groups are to be searched or for those searches to produce contraband?
Are there disparities in the reason for stopping drivers of different racial groups?
Once stopped, are stop outcomes different for drivers of different racial groups?
Do specific work units contribute differently to the racial composition of vehicle stops?

3 Are there racial disparities in who is stopped while on foot?

After accounting for certain factors, including poverty levels and crime rates, what level of observed disparities in pedestrian stops remains?
How much of stop frequency can be explained by these factors?
Once stopped, are stop outcomes different for pedestrians of different racial groups?
Are there differences in how likely pedestrians in different racial groups are to be searched or for those searches to produce contraband?
Do specific work units or a small number of officers contribute differently to observed racial disparities in pedestrian stops?

4 What types of activities do officers initiate, and how does this align with what the community is requesting through calls for service?

How much of recorded police activity is made up of events involving reports of bodily harm, property harm, or threats versus all other events? What percentage of these events are initiated by officers (“officer-initiated activities”) versus a call for service (“calls for service”)?
- When data do not distinguish between officer-initiated activities and police responses to calls for service, our analyses will primarily examine events by event type.
What types of calls for service do officers respond to? Does this align with the types of enforcement police are initiating themselves?
How often do officers initiate or respond to events recorded as mental health crises?
Do police generally enforce more low-level offenses in neighborhoods with a higher proportion of Black residents and/or higher levels of poverty? Where in the jurisdiction do police most often record this type of enforcement?
Do certain areas of the jurisdiction have higher levels of officer-initiated activity relative to the number of calls for service? If so, where?
Do call response times vary by officer assignment or neighborhood poverty rates? Where in the jurisdiction do police take more time on average to respond to calls for service?

Why does CPE focus on those questions?

These basic principles guide our selection of research questions:

1 An understanding that criminal justice data are imperfect.

There are limitations to using police data to analyze racial disparities. For example, crime data do not capture crimes not reported to, or seen by, the police. Similarly, data cannot be collected on the racial group of every person seen by an officer on patrol, so we can’t determine exactly how much the officer’s stop or use of force decisions reflect the racial makeup of the local community. Additionally, data rely on the perception of officers, who may only provide partial records or misidentify individuals with whom they interact. Working within these limitations, we aim to produce the most accurate and useful findings about racial disparities in policing, erring on the side of caution where there is room for interpretation in our conclusions.

2 Alignment with evolving science.

The way we produce analyses in these assessments changes based on the best available evidence from the fields of social science, criminal justice, and policing. To this end, we have a dedicated Science team and an external Science Advisory Board, both made up of experts in social science research, who help shape the way we analyze data.

3 Creating consistent and high-quality policing data standards.

Many departments simply do not collect the data required to analyze racial disparities or may collect relevant data in thousands of different ways. We advise departments on practices for data collection and analysis that are consistent and beneficial to a large number of departments. In many cases, they become empowered to measure racial inequity within their departments for the first time.

4 Prioritizing actionable findings.

There are many ways to analyze data to produce new or interesting insights into racial disparities. Our analyses are specifically designed to clearly identify racial disparities that may support solutions by the department or other stakeholders.

What types of analyses do assessments not include?

Generally speaking, we do not produce analyses that rely on theories, methodologies, or other assumptions that are not accepted science. More specifically, we do not perform analyses that have one or more of the following characteristics:

1 Uses data that are not easily accessible.

Inaccessible data are those that rely on data sources outside of the department and are not publicly available, such as personnel records filed with a city or charging decisions made by prosecutors (such data also usually fall outside of our analytical focus on direct interactions between community members and police). We also exclude analyses that rely on data collected by only a few departments around the country, to ensure that the findings we produce are relevant and useful for as many participants as possible. For example, estimates of access to mental health services and prevalence of homelessness are recorded by some larger cities, but not universally, and there are no national estimates for these data.

2 Uses data that are unreliably reported or are not good measures of what they seek to capture.

Scientific research has found that officer reports on community members’ resistance in the context of use of force incidents may be inconsistent or influenced by conscious or unconscious bias. Accordingly, we require additional, more objective data to confirm an officer’s report (such as information on whether the community member in question was found to be in possession of a weapon or found to be intoxicated). Even if collected reliably, data may not effectively measure what we want to study. For example, if a search dataset includes only searches resulting in the discovery of contraband, these data cannot be used to analyze equity in the frequency of searches.

3 Uses a dataset that includes many missing observations.

When a large, systemic number of data observations are missing (meaning the dataset lacks information on some members of the group), the resulting analysis is likely to be biased and the findings may be misleading. For example, we would not conduct a neighborhood-level analysis if data collected from certain neighborhoods had many more missing observations on race than data collected from other neighborhoods.

4 Would violate CPE's data protection and confidentiality rules.

We take careful measures to protect departmental confidentiality and avoid releasing information that could be linked to any single person. Therefore, we do not analyze data that would compromise this confidentiality, such as a specific combination of racial group and gender that would be held by only a few officers.

How does CPE use population benchmarking?

There are many approaches to estimating racial disparities in policing activity. Our approach combines population benchmarking—used to assess whether disparities exist and how large they are—with regression analyses and other strategies that estimate the portion of disparities in policing activity that may be attributable to law enforcement behavior.

We believe that population benchmarking, when used with other statistical methods, is an important way to establish an initial measurement of disparities, which law enforcement agencies and communities can use to measure future progress. Population benchmarking works by taking into account the number of people of each racial group that live in the community served by a given department as a baseline for estimating what equitable treatment should look like (for example, if 10% of a community’s residents are Latinx, one would expect that 10% of pedestrian stops are of Latinx people). From there, the difference between the size of the population and the share of police contact is used to calculate the disparity. To measure the local population, we use the U.S. Census Bureau’s American Community Survey’s 5-year estimates, which are available across the U.S. on an annual basis. This allows us to perform standardized analyses across law enforcement agencies. This approach provides insight on how likely people of different racial groups are to have contact with law enforcement, which can illuminate community pain points and disparities in the relative burden of police enforcement.

Some scientists and practitioners use other approaches to benchmarking, such as comparing police behavior to people who are arrested or to crime rates, to measure disparity. Unlike population benchmarking, these approaches compare police behavior to groups that may have already been subject to bias from police and other systems. For example, greater police presence in neighborhoods with a majority of Black residents can result in higher arrest rates for Black people than for White people engaging in the same behaviors. These approaches therefore carry the risk of underestimating the size of true disparities by not accounting for any bias that impacts who is included in the comparison group. Like all other approaches population benchmarking is not perfect, and cannot capture the exact population with which officers interact. For example, it cannot account for out-of-town visitors—though it is not known whether any disparity observed would look bigger or smaller if that population was fully accounted for. However, estimating disparities using population benchmarking provides meaningful information about the experiences of people interacting with a police department, even if some or many of the people who are stopped or who are subjected to force may have come from out of town. Unlike the analyses conducted on use of force incidents and pedestrian stops, CPE does not use population benchmarking to analyze vehicle stops (for more explanation, see “More information” in our Sample City Assessment).

Population benchmarking is useful for showing whether the rate of police contact is equitable across different racial groups and to approximate the scope of any inequitable contact. However, it is important to understand that population benchmarking cannot tell us the source of disparity or the outcome of inequitable contact, which is why we also use other statistical methods in combination with population benchmarking. We use other statistical approaches to investigate disparities in outcomes of police behavior (such as comparing the racial composition of search rates or force type) in order to shed light on patterns of treatment in the outcomes of police activity beyond any patterns in the likelihood of experiencing contact with police. We also use other advanced statistical approaches, such as multilevel regression analysis, to examine the extent to which disparities in some police behaviors, including pedestrian stops and use of force incidents, are explained by neighborhood characteristics such as poverty levels and rates of crime. This information helps estimate the portion of disparities that law enforcement can most directly address. Taken together, this suite of analyses can provide reliable estimates of the scale of disparities in police contact and can direct community members, law enforcement, and advocates to areas that warrant further investigation—and immediate action.