Large language models have been useful in expanding mental health care
delivery. ChatGPT, in particular, has gained popularity for its ability to
generate human-like dialogue. However, data-sensitive domains -- including but
not limited to healthcare -- face challenges in using ChatGPT due to privacy
and data-ownership concerns. To enable its utilization, we propose a text
ambiguation framework that preserves user privacy. We ground this in the task
of addressing stress prompted by user-provided texts to demonstrate the
viability and helpfulness of privacy-preserved generations. Our results suggest
that chatGPT recommendations are still able to be moderately helpful and
relevant, even when the original user text is not provided