Risk assessment algorithms are being adopted by public sector agencies to
make high-stakes decisions about human lives. Algorithms model "risk" based on
individual client characteristics to identify clients most in need. However,
this understanding of risk is primarily based on easily quantifiable risk
factors that present an incomplete and biased perspective of clients. We
conducted a computational narrative analysis of child-welfare casenotes and
draw attention to deeper systemic risk factors that are hard to quantify but
directly impact families and street-level decision-making. We found that beyond
individual risk factors, the system itself poses a significant amount of risk
where parents are over-surveilled by caseworkers and lack agency in
decision-making processes. We also problematize the notion of risk as a static
construct by highlighting the temporality and mediating effects of different
risk, protective, systemic, and procedural factors. Finally, we draw caution
against using casenotes in NLP-based systems by unpacking their limitations and
biases embedded within them