Project Rationale & Big Picture Questions
Since my mother was diagnosed with dementia in late 2023, my goal has been to learn more about the wicked problem of cognitive degeneration, and come up with a concept for and then build a technology platform using web, mobile, and communication technologies to bring people experiencing cognitive decline, families, and care providers together to improve lives, improve treatment outcomes, and preserve shared histories while doing so.
Is this something technology can help solve or at least mitigate?
Can something as purely mathematical as software help families emotionally or even practically? Could it even warn people of potential cognitive health issues before things progress too far to ameliorate?
This is not just a problem for my family. It is a growing global problem with 2% of the entire population of Earth suffering from some form of dementia - and 60-80% of that is Alzheimer’s (Alzheimer’s Association). By 2050, that number is expected to triple due to the aging of the world’s population (GBD 2019 Dementia Forecasting Collaborators).
The Sheaf Solution
My design solution combines both mobile and browser-based applications to schedule consistent calls with loved ones who may have cognitive issues. Social engagement is one of the most effective known ways to slow cognitive decline, and a caller can ask for stories, recipes or other types of structured conversation, and otherwise engage on a regular basis. Calls are transcribed, summarized, organized thematically, parsed for names, dates, and locations using AI, and saved in a family archive. For those who give permission, those calls are also securely analyzed for linguistic and speech indicators, called &$8220;linguistic markers”, that correlate with cognitive function, This data can be shared in a user-friendly way with privileged family members.
A separate, HIPAA-compliant web portal for care providers will integrate our cognitive analyses with electronic health records using industry-standard schemas and APIs (HL7 International).
Capturing audio and video of people telling treasured stories provides a legacy for families, while at the same time potentially improving care and understanding by quantifying cognitive fitness. Sheaf can be used remotely, or in person, or even process pre-recorded audio. Family members will be able to create print or audiobooks and even videos with the help of AI-enhanced tools, impacting current and future family and friends of every Sheaf participant.
Now for the documentation video. Everything other than the logo you see in this demo I built for this capstone.
Process
Concept: User Interviews
To better understand the difficult issues surrounding cognitive health, and how to support families with elderly relatives suffering, or potentially suffering, cognitive decline, I interviewed domain experts in health, neurology, linguistic marker technology, and natural language processing by phone, videoconference, and in person.
Tech Stack and Implementation
For a tech stack, I went with Python, React, Postgres, and am using APIs for making phone calls (Twilio), transcriptions (Whisper, PyAnnote) and story analysis (OpenAI), and story-based image generation (Leonardo AI). How did I make it? Many MANY long days and nights, and pair-programming with AI as much as I could stand. Even I’m a bit surprised at what I have built in just 30 days.
Overall, my development process is very organic, which is a euphemism for “idiosyncratic and disorganized”. I get a rough idea of what I feel is most important to bang away at first, and dive in, expanding or changing direction as I go, getting feedback and inspiration (or hitting brick walls) along the way. It’s sort of a low-grade agile process. At such time as I'm able to bring on other developers, things will get more structured. In fact, the next step is to work with a partner who is expert in design and UX, so that will change everything.
My target has been to have core pieces working in time for this assignment deadline, and I feel I’m there or close to it. It’s been many long, late nights but things are beginning to take shape.
At this point, one can place a call, record audio in browser, or upload an audio file, have those be processed for story content and for cognitive markers, and the system returns that data to show in charts and story-derived metadata. I also have the basics for a Provider Portal that would enable privileged care providers to view the cognitive data Sheaf captures juxtaposed with their care plans over time.
Concept: Evolution of the Product Idea
The project has already evolved in many directions, and many aspects.
- It began as a VR concept for brain games, virtual travel and conversation, but the technical complexity of both the headsets, the setup, the software seemed far too involved for elderly people suffering from cognitive issues.
- Next it evolved to the idea of a storytelling app, to increase social engagement and preserve stories that might be lost.
- Subsequently, we came across the idea of adding cognitive analysis with storytelling, we finally felt we had something unique and powerful.
- A serendipitous Uber ride with medical software salespeople led to the next piece - a portal with longitudinal cognitive data that care providers could use to evaluate the effectiveness of their treatment plans.
- A user- research interview with a medical professional gave us the idea to make our anonymized data available in aggregate for research purposes (and I will say, all the studies I've read regarding linguistic markers have had pools of 30 to 250 users - nothing like the thousands we are looking at).
Additional Resources: Design & Architecture Process
Early Sketches
Implementation
Artifacts from my personal process...
Early UX Modeling & Wireframes (Mobile):
Codebase:
Database Schema (ERD):
API Documentation:
Works Cited
- Alzheimer's Association. “2024 Alzheimer's Disease Facts and Figures.” Alzheimer's & Dementia, vol. 20, no. 5, 30 Apr. 2024, pp. 3708–3821, alz-journals.onlinelibrary.wiley.com/doi/10.1002/alz.13809. Accessed 13 Apr. 2025.
- CDC. “About Dementia.” Alzheimer's Disease and Dementia, U.S. Centers for Disease Control and Prevention, 17 Aug. 2024, www.cdc.gov/alzheimers-dementia/about/index.html. Accessed 13 Apr. 2025. Dementia statistics.
- GBD 2019 Dementia Forecasting Collaborators. “Estimation of the Global Prevalence of Dementia in 2019 and Forecasted Prevalence in 2050: An Analysis for the Global Burden of Disease Study 2019.” The Lancet Public Health, vol. 7, no. 2, 6 Jan. 2022, www.thelancet.com/journals/lanpub/article/PIIS2468-2667(21)00249-8/fulltext, https://doi.org/10.1016/s2468-2667(21)00249-8. Accessed 13 Apr. 2025.
- HL7 International. “CarePlan - FHIR V5.0.0.” Hl7.org, 2023, hl7.org/fhir/careplan.html. Accessed 14 Apr. 2025.