How We Work

Kokosing began with a belief that young people often notice changes in their own lives before institutions know how to study them. Our researchers are close to the questions we work on because they are experiencing many of them themselves: the arrival of generative AI in school, changing relationships with social media, new forms of online gambling and financial risk, and uncertainty about what work will look like in the years ahead.

Such proximity is useful, but it is not evidence. Once we decide a question is worth studying, we try to approach it with the same skepticism we would want from anyone evaluating our work. Our researchers learn to distinguish what they have observed from what the data can actually establish, to pay attention to results that run against their expectations, and to narrow a conclusion when the evidence does not justify the larger one.

This page provides an overview of that process. Our full Research Process and Methodology explains our standards in greater detail, including survey construction, sampling, data handling, statistical analysis, ethics, publication, and reproducibility. We revise that document as our work develops and as each round of research teaches us something we should do differently.

Developing the question

Most projects begin with something a member of the research team has noticed and wants to understand more carefully. That first observation is usually broad. Students may feel that AI is changing how their peers approach school, for example, but that does not immediately tell us what to measure. Are we interested in how often students use AI, what they use it for, whether it changes how much they learn, how they think about academic integrity, or whether some students have access to tools and guidance that others do not?

The early stages of a project are spent breaking broad concerns into questions that can actually be investigated. We look at existing research, discuss what we think is missing from the public conversation, and decide which parts of the problem our own data can reasonably speak to.

Student researchers play a particularly important role here. They often know when language that sounds natural to an adult would be interpreted differently by their peers, or when two behaviors that look similar from the outside are meaningfully different in practice. We use that knowledge to improve the research question, not to decide the answer in advance.

Designing the research

For our survey work, we spend considerable time on the instrument before it reaches respondents. Questions are reviewed for ambiguity, leading language, unclear scales, missing response options, and assumptions built into the wording. At least two people who did not write the survey complete it before fielding and flag anything they find confusing.

We learned some of these practices through mistakes in our first survey wave. Conditional questions were not always programmed correctly, selection limits were stated but not technically enforced, and language around anonymity was not as precise as it should have been. We documented those problems rather than quietly treating them as if they had never happened, and they became concrete requirements for later work. Our methodology is meant to describe what we actually do, including what we have had to improve, rather than an idealized version of the process.

We also preserve the original survey definition so that there is a record of exactly what respondents saw: the wording, response options, their order, and any conditional logic. This becomes part of the research record alongside the eventual dataset.

Understanding the sample

Kokosing surveys are recruited through student networks, schools, organizations, and peer outreach. They are convenience samples, not probability samples, and we treat that distinction seriously.

This affects what we are willing to say about the results. A sample concentrated among students at particular Bay Area schools cannot simply become “American teenagers” when we write the report. We describe the population we actually reached, report its composition, and repeat important qualifications when a finding might otherwise be read more broadly than the evidence allows.

We do not apply weighting when we do not have a defensible population frame, and we do not present conventional margins of error as though our respondents were selected randomly. Sometimes the composition of a sample also creates useful opportunities for comparison, but those analyses still have to be described within the limits of the data we collected.

This can make our conclusions narrower than we initially hoped. We are comfortable with that. The goal is to understand the group we studied well, rather than make a larger claim simply because it sounds more consequential.

Preparing and checking the data

The original survey export is preserved without modification. Analysis is conducted on a separate copy, and the changes made between the raw data and the analysis file are documented.

Before beginning substantive analysis, we create a codebook that records the original question wording, variable type, response options, coding decisions, scale direction, missing-value rules, and any uncertainty about how an item should be interpreted. If the meaning of a variable cannot be established from the original instrument, we flag it rather than guess.

We also run data-quality checks before treating the responses as analytical evidence. These include looking for likely duplicate submissions, unusually fast completions, straightlining across rating questions, responses produced by incorrectly required questions, and violations of stated selection limits. Where a problem can be handled through a clear rule, the rule is written down and applied consistently. Where it introduces uncertainty that cannot be removed, that limitation stays with the analysis.

Because many of our respondents are minors and some of our questions concern sensitive subjects, identifying information is separated from the working dataset. We assess combinations of demographic variables for re-identification risk and generalize them when necessary before data is shared beyond the research team. Our full methodology describes these procedures in detail.

Analyzing the results

Our analysis usually begins with the simplest question: what does the sample actually show?

We examine distributions, group differences, relationships between variables, and open-ended responses where appropriate. The statistical methods depend on the type of data rather than on which test produces the most interesting result. For example, much of our survey data uses ordinal rating scales and relatively small subgroups, so our methodology specifies nonparametric tests and Fisher's exact test in circumstances where stronger assumptions would not be justified.

The interpretation matters just as much as the test itself. We look at the size of a difference, the number of respondents behind it, whether a finding survives reasonable sensitivity checks, and whether there are plausible alternative explanations. Because our surveys are generally cross-sectional, we do not turn associations into claims of causation.

We also conduct many exploratory comparisons, which creates a real possibility of finding apparently significant relationships by chance. Rather than ignore that problem, we disclose the number of comparisons involved and use more demanding standards for findings that become central to a report. Findings that are weaker but potentially meaningful are described more cautiously.

Results that do not fit the original hypothesis remain part of the research. In some cases, a null result can be more informative than the relationship we expected to find. What matters is whether the evidence changes our understanding of the question, not whether it produces the argument we imagined at the beginning.

Interpreting the work together

Student researchers are involved in the process of making sense of the results, not only in collecting responses or appearing on a byline.

We review tables and figures together, discuss competing explanations, compare findings with what previous research suggests, and challenge claims that seem to run ahead of the evidence. Drafts often change substantially during this stage. Sometimes the strongest version of an argument becomes more precise. Sometimes an interesting result becomes a footnote. Sometimes an argument disappears entirely.

That experience is central to what Kokosing is trying to teach. Research requires enough confidence to make an argument, but also enough distance from that argument to change it when the evidence points somewhere else. Giving young researchers practice doing that is more valuable to us than giving them a predetermined conclusion to defend.

Writing and publishing

Our reports are organized around questions and arguments rather than as summaries of every survey item. We include the evidence necessary to support the argument, preserve relevant findings that complicate it, and make methodological limitations visible where readers will actually encounter them.

Figures are generated directly from the analysis data rather than transcribed by hand. Reported differences include the underlying percentages or means and the size of the groups being compared. Findings based on small groups are identified as such. Where there is an important alternative explanation or a result could easily be misunderstood, we address that alongside the finding rather than relying on a methodological appendix to correct the impression later.

The same standards apply when research leaves the report itself. Statistics used in op-eds or other public writing must be traceable to the underlying analysis, and the qualifications that materially affect their interpretation travel with them. Our aim is to make the work accessible without making it more certain than it is.

Ethics and responsibility

Research involving young people carries responsibilities that go beyond getting the statistics right. Our respondents may be sharing information about mental health, finances, gambling, political views, academic conduct, or other parts of their lives that they would not expect to become identifiable.

We therefore try to collect only information that serves a defined analytical purpose, limit access to raw responses, separate contact information from survey data, and publish findings in aggregate. We do not report groups small enough to make individual respondents identifiable.

We have also become more formal about review as the Institute has developed. Our current standard is that surveys involving minors and sensitive subjects should receive review from an adult with relevant expertise before they are fielded, and that our publications should state whether that review occurred. Our first survey wave did not receive that review, and our methodology says so directly.

That kind of disclosure is important to us. Credibility does not come from pretending a young organization has always gotten everything right. It comes from being specific about the standards we use now, the places where earlier work fell short of them, and what we changed as a result.

Making the work reproducible

For each Kokosing report, we preserve the materials necessary to understand how the published findings were produced, including the codebook, data-quality documentation, de-identified analysis data where appropriate, the archived survey definition, and the code used to generate figures.

The basic expectation is that someone with those materials should be able to follow the path from the original responses to the numbers in the final report. Our full Research Process and Methodology sets out those standards and is revised as our practices continue to develop.

For us, this process is as important as any individual finding. Kokosing exists both to produce useful research and to give young researchers experience doing the less visible work that makes research trustworthy: defining questions carefully, documenting decisions, confronting limitations, changing an interpretation when necessary, and being clear with readers about what the evidence does and does not allow us to say.