publications
2025
- Tandem: At-Home Behavior Assessment Using Multimodal Signals from the Parent-Child DyadProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Dec 2025
The quality of parent-child interactions at an early age has been linked to children’s social-emotional development, executive function, and risk for behavior problems. As such, parent-child interactions in naturalistic settings could present a unique opportunity to screen for at-risk behavior in young children, enabling timely and targeted interventions. In this work, we validate the feasibility of using structured at-home play sessions, completed via the Tandem smartphone app, to enable highly accurate and scalable behavioral assessments. We demonstrate that audio and physiological signals recorded during the play session can be used to capture key markers of parent-child interaction dynamics, which are more indicative of at-risk behavior compared to features from each individual alone. We propose novel audio-based dyadic interaction features that significantly outperform conventional speech features at predicting risk for behavior problems, achieving an F1 score of 0.87. Furthermore, we show that dyadic physiological synchrony features, extracted from privacy-preserving wearable sensor data, can classify at-risk behavior with an F1 score of 0.91. Tandem thus sets the stage for automated at-home behavior assessment tools for young children that balance screening accuracy with practical deployment considerations.
2024
- Playlogue: Dataset and Benchmarks for Analyzing Adult-Child Conversations During PlayKalanadhabhatta, Manasa, Rastikerdar, Mohammad Mehdi, Rahman, Tauhidur, Grabell, Adam, and Ganesan, DeepakProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. Nov 2024
There has been growing interest in developing ubiquitous technologies to analyze adult-child speech in naturalistic settings such as free play in order to support children’s social and academic development, language acquisition, and parent-child interactions. However, these technologies often rely on off-the-shelf speech processing tools that have not been evaluated on child speech or child-directed adult speech, whose unique characteristics might result in significant performance gaps when using models trained on adult speech. This work introduces the Playlogue dataset containing over 33 hours of long-form, naturalistic, play-based adult-child conversations from three different corpora of preschool-aged children. Playlogue enables researchers to train and evaluate speaker diarization and automatic speech recognition models on child-centered speech. We demonstrate the lack of generalizability of existing state-of-the-art models when evaluated on Playlogue, and show how fine-tuning models on adult-child speech mitigates the performance gap to some extent but still leaves considerable room for improvement. We further annotate over 5 hours of the Playlogue dataset with 8668 validated adult and child speech act labels, which can be used to train and evaluate models to provide clinically relevant feedback on parent-child interactions. We investigate the performance of state-of-the-art language models at automatically predicting these speech act labels, achieving significant accuracy with simple chain-of-thought prompting or minimal fine-tuning. We use inhome pilot data to validate the generalizability of models trained on Playlogue, demonstrating its utility in improving speech and language technologies for child-centered conversations. The Playlogue dataset is available for download at https://huggingface.co/datasets/playlogue/playlogue-v1.
- Multi-stakeholder Perspectives on Mental Health Screening Tools for ChildrenKalanadhabhatta, Manasa, Mateo Santana, Adrelys, Mayorga, Lynnea, Rahman, Tauhidur, Ganesan, Deepak, and Grabell, AdamIn Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. May 2024
Pediatric mental health is a growing concern around the world, affecting children’s social-emotional development and increasing the risk of poor behavioral outcomes later in life. However, obtaining a behavioral diagnosis in early childhood is challenging due to lack of access to resources, low parental mental health literacy, and children’s dependence on several stakeholders to coordinate care for them. While app-based, at-home screening tools could offer a scalable and convenient diagnostic solution for families, stakeholder perspectives on their utility and usability remain to be examined. This work reports on a survey of child mental health practitioners and interviews with parents to illustrate existing barriers to care that stakeholders encounter, the perceived benefits of app-based screening tools in meeting their needs, and the challenges in scaling up these tools. We identify where stakeholders agree or disagree, delineate key design tensions, and provide recommendations for the development of future screening technologies.
2023
- Towards Accurate and Scalable Mental Health Screening Technologies for Young ChildrenKalanadhabhatta, Manasa, Ganesan, Deepak, and Rahman, TauhidurIn Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing. Oct 2023
Mental, emotional, and behavioral disorders are highly prevalent in preschool-aged children and can significantly affect their social-emotional development and adaptive functioning. However, identifying signs of problematic behavior at this age is extremely challenging due to several structural and phenomenological barriers. This work leverages mobile and wearable devices to build accurate, usable, and scalable assessment tools that can be deployed in home settings to screen for common disorders in young children. It describes the development of novel screening algorithms that utilize behavioral and neurophysiological signals recorded during brief, naturalistic tasks, and presents stakeholder perspectives toward the usability and clinical utility of such screening tools.
- Detecting PTSD Using Neural and Physiological Signals: Recommendations from a Pilot StudyKalanadhabhatta, Manasa, Roy, Shaily, Grant, Trevor, Salekin, Asif, Rahman, Tauhidur, and Bergen-Cico, DessaIn 11th International Conference on Affective Computing and Intelligent Interaction (ACII). Sep 2023
Post-traumatic stress disorder (PTSD) is a serious condition that is characterized by negative mood and affect, hyperarousal, irritability, and reactivity, as well as deterioration of cognitive processes such as attention and memory. Timely identification and treatment of PTSD symptoms can significantly improve symptom management and recovery. However, accurate prediction of PTSD outside clinical settings is often challenging. In this work, we investigate whether deficits in cognitive performance can be used to classify individuals with and without PTSD. We further examine whether neural and physiological signals such as prefrontal cortex activity, heart rate, respiration, and electrodermal activity recorded in conjunction with cognitive task performance can be leveraged to improve PTSD classification. Our results indicate that working memory tasks can achieve an F1 score of 0.80 at classifying individuals with PTSD, which can be further improved to 0.91 by combining multimodal information from neurophysiological signals. Based on our findings, we provide recommendations for in-the-wild PTSD classification.
2022
- Extracting Multimodal Embeddings via Supervised Contrastive Learning for Psychological ScreeningKalanadhabhatta, Manasa, Mateo Santana, Adrelys, Ganesan, Deepak, Rahman, Tauhidur, and Grabell, AdamIn 10th International Conference on Affective Computing and Intelligent Interaction (ACII). Oct 2022
The diagnosis of psychological disorders in early childhood is of utmost importance given their severe impact on children’s academic and social skills as well as general adaptive functioning. Wearable and video-based systems have the potential to collect important diagnostic information in the form of neurophysiological and behavioral signals. However, accurate prediction of psychological disorder status from multimodal data streams necessitates their combination into meaningful features for classification models. In this work, we present a multitask supervised contrastive learning approach to learn useful multimodal embeddings from functional Near-Infrared Spectroscopy, galvanic skin response, and facial video data collected during a frustration-inducing task. The generated embeddings are able to accurately infer emotion regulation-related psychological disorders with an F1 score of 0.91, having significant implications for early-childhood mental health diagnoses.
- EarlyScreen: Multi-scale Instance Fusion for Predicting Neural Activation and Psychopathology in Preschool ChildrenKalanadhabhatta, Manasa, Mateo Santana, Adrelys, Zhang, Zhongyang, Ganesan, Deepak, Grabell, Adam, and Rahman, TauhidurProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. Jul 2022
Emotion dysregulation in early childhood is known to be associated with a higher risk of several psychopathological conditions, such as ADHD and mood and anxiety disorders. In developmental neuroscience research, emotion dysregulation is characterized by low neural activation in the prefrontal cortex during frustration. In this work, we report on an exploratory study with 94 participants aged 3.5 to 5 years, investigating whether behavioral measures automatically extracted from facial videos can predict frustration-related neural activation and differentiate between low- and high-risk individuals. We propose a novel multi-scale instance fusion framework to develop EarlyScreen - a set of classifiers trained on behavioral markers during emotion regulation. Our model successfully predicts activation levels in the prefrontal cortex with an area under the receiver operating characteristic (ROC) curve of 0.85, which is on par with widely-used clinical assessment tools. Further, we classify clinical and non-clinical subjects based on their psychopathological risk with an area under the ROC curve of 0.80. Our model’s predictions are consistent with standardized psychometric assessment scales, supporting its applicability as a screening procedure for emotion regulation-related psychopathological disorders. To the best of our knowledge, EarlyScreen is the first work to use automatically extracted behavioral features to characterize both neural activity and the diagnostic status of emotion regulation-related disorders in young children. We present insights from mental health professionals supporting the utility of EarlyScreen and discuss considerations for its subsequent deployment.
2021
- FatigueSet: A Multi-modal Dataset for Modeling Mental Fatigue and FatigabilityIn 15th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth). Dec 2021
A comprehensive understanding of fatigue and its impact on performance is a prerequisite for fatigue management systems in the real world. However, fatigue is a multidimensional construct that is often poorly defined, and most prior work does not take into consideration how different types of fatigue collectively influence performance. The physiological markers associated with different types of fatigue are also underexplored, hindering the development of fatigue management technologies that can leverage mobile and wearable sensors to predict fatigue. In this work, we present FatigueSet, a multi-modal dataset including sensor data from four wearable devices that are collected while participants are engaged in physically and mentally demanding tasks. We describe the study design that enables us to investigate the effect of physical activity on mental fatigue under various situations. FatigueSet facilitates further research towards a deeper understanding of fatigue and the development of diverse fatigue-aware applications.
- Effect of Sleep and Biobehavioral Patterns on Multidimensional Cognitive Performance: Longitudinal, In-the-Wild StudyKalanadhabhatta, Manasa, Rahman, Tauhidur, and Ganesan, DeepakJ Med Internet Res. Feb 2021
Background: With nearly 20% of the US adult population using fitness trackers, there is an increasing focus on how physiological data from these devices can provide actionable insights about workplace performance. However, in-the-wild studies that understand how these metrics correlate with cognitive performance measures across a diverse population are lacking, and claims made by device manufacturers are vague. While there has been extensive research leading to a variety of theories on how physiological measures affect cognitive performance, virtually all such studies have been conducted in highly controlled settings and their validity in the real world is poorly understood. Objective: We seek to bridge this gap by evaluating prevailing theories on the effects of a variety of sleep, activity, and heart rate parameters on cognitive performance against data collected in real-world settings. Methods: We used a Fitbit Charge 3 and a smartphone app to collect different physiological and neurobehavioral task data, respectively, as part of our 6-week-long in-the-wild study. We collected data from 24 participants across multiple population groups (shift workers, regular workers, and graduate students) on different performance measures (vigilant attention and cognitive throughput). Simultaneously, we used a fitness tracker to unobtrusively obtain physiological measures that could influence these performance measures, including over 900 nights of sleep and over 1 million minutes of heart rate and physical activity metrics. We performed a repeated measures correlation (rrm) analysis to investigate which sleep and physiological markers show association with each performance measure. We also report how our findings relate to existing theories and previous observations from controlled studies. Results: Daytime alertness was found to be significantly correlated with total sleep duration on the previous night (rrm=0.17, P<.001) as well as the duration of rapid eye movement (rrm=0.12, P<.001) and light sleep (rrm=0.15, P<.001). Cognitive throughput, by contrast, was not found to be significantly correlated with sleep duration but with sleep timing—a circadian phase shift toward a later sleep time corresponded with lower cognitive throughput on the following day (rrm=–0.13, P<.001). Both measures show circadian variations, but only alertness showed a decline (rrm=–0.1, P<.001) as a result of homeostatic pressure. Both heart rate and physical activity correlate positively with alertness as well as cognitive throughput. Conclusions: Our findings reveal that there are significant differences in terms of which sleep-related physiological metrics influence each of the 2 performance measures. This makes the case for more targeted in-the-wild studies investigating how physiological measures from self-tracking data influence, or can be used to predict, specific aspects of cognitive performance.
2018
- Analysis of Peer Group Behavior Among University StudentsKunchay, Sahiti, and Kalanadhabhatta, Lakshmi ManasaIn Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. Apr 2018
Satisfactory peer group interactions within a university, through the formation of close associations, define a student’s personality and help in deterring the rise of depression caused by academic, financial or emotional troubles. In this work, we conduct a pre-study survey of 177 students in a University setting to assess the requirement for a smartphone-based study to detect and monitor group formation, evolution and engagement. The preliminary results from this investigation reveal that students social interactions are not limited to one but several groups, and the satisfaction levels associated with each type of group are indicative of the average time spent engaging with said group(s). Intra-group bond strength took precedence as a satisfaction determinant over the location or activity engaged in. Further, we present design recommendations for a minimally invasive smartphone-based study.
2017
- Smartphone-based Qualitative Analyses of Social Activities During Family TimeSahiti, Kunchay, Kalanadhabhatta, Lakshmi Manasa, Bhunia, Suman Sankar, Singhal, Akshit, and Majethia, RahulIn Proceedings of the First International Workshop on Human-centered Sensing, Networking, and Systems. Nov 2017
With the evolution of nuclear families and diverse career options, families as social groups are spending lesser time together than in the past decades. In this work, we study both quantitative as well as qualitative aspects of time spent with family members through a smartphone-based pervasive study on a sample of 12 families over 14 days. Further, we also examine the perception of 78 millennials on what they feel about, and expect from, the time they spend with their families, however long it may be. We aim to identify the key parameters that shape family life in this day and age, along with examining the participation of individuals of various roles within the family in activities such as conversations, workout sessions, eating together and other social interactions. Among all activities detected to be performed by families reporting high satisfaction with familial life, Eating Together and Using Smartphones Together emerged as the most prominent ones. We discover a greater disparity among the habits of family members, especially millennials, staying away from each other as compared to those staying together.
- Application Overchoice: Preliminary Lessons from a Longitudinal StudyIn Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers. Sep 2017
In this paper, we investigate how smartphone users navigate the dilemma of application overchoice, i.e., the scenario of having multiple competing apps available to serve a similar purpose. We analyze app installs, app usage behavior and notification attendance behavior to paint an initial picture of app overchoice and to explore how overchoice is impacted by smartphone notifications. We hope that this paper will provoke discussions and more research in the UbiTtention community on developing systems that help users navigate the dilemma of overchoice.
- Moving Beyond Market Research: Demystifying Smartphone User Behavior in IndiaProceedings of the ACM on Interactive, Mobile, Wearable, and Ubiquitous Technologies. Sep 2017
Large-scale mobile data studies can reveal valuable insights into user behavior, which in turn can assist system designers to create better user experiences. After a careful review of existing mobile data literature, we found that there have been no large-scale studies to understand smartphone usage behavior in India – the second-largest and fastest growing smartphone market in the world. With the goal of understanding various facets of smartphone usage in India, we conducted a mixed-method longitudinal data collection study through an Android app released on Google Play. Our app was installed by 215 users, and logged 11.9 million data points from them over a period of 8 months. We analyzed this rich dataset along the lines of four broad facets of smartphone behavior – how users use different apps, interact with notihcations, react to different contexts, and charge their smartphones – to paint a holistic picture of smartphone usage behavior of Indian users. This quantitative analysis was complemented by a survey with 55 users and semi-structured interviews with 26 users to deeply understand their smartphone usage behavior. While our first-of-its-kind study uncovered many interesting facts about Indian smartphone users, we also found striking differences in usage behavior compared to past studies in other geographical contexts. We observed that Indian users spend significant time with their smartphones after midnight, continuously check notifications without attending to them and are extremely conscious about their smartphones’ battery. Perhaps the most dramatic finding is the nature of mobile consumerism of Indian users as shown by our results. Taken together, these and the rest of our findings demonstrate the unique characteristics that are shaping the smartphone usage behavior of Indian users.
2016
- AnnoTainted: Automating Physical Activity Ground Truth Collection Using SmartphonesMajethia, Rahul, Singhal, Akshit, Kalanadhabhatta, Lakshmi Manasa, Sahiti, Kunchay, Kishore, Shubhangi, and Nandwani, VijayIn Proceedings of the 3rd International on Workshop on Physical Analytics. Jun 2016
In this work, we provide motivation for a zero-effort crowdsensing task: auto-annotated ground truth collection for physical activity recognition. Data obtained through Smartphones for classification of human activities is prone to discrepancies, which reiterates the need for better and larger activity datasets. Artificial data generation algorithms fail to efficiently generate quality instances for minority data. In the proposed model, crowd-sourced sensor data is classified by a robust classifier built by researchers ground up. We nominate a Generic Classifier with ≥ 95% accuracy for this purpose. Data collection and distribution models which ensure that the crowd client receives non-skewed, quality data from locations with higher degree of activity occurrence are elucidated upon. Also integrated within our proposed model are Location-Specific Classifiers, which can be utilized by developers to optimize on location-specific tasks. Effective validation of classified activities using diverse sensor data streams improves the proposed classifier systems and boosts ground-truth accuracy.
- PeopleSave: Recommending Effective Drugs Through Web CrowdsourcingMajethia, Rahul, Mishra, Varun, Singhal, Akshit, Kalanadhabhatta, Lakshmi Manasa, Sahiti, Kunchay, and Nandwani, VijayIn 2016 8th International Conference on Communication Systems and Networks (COMSNETS). Jan 2016
In this paper, we describe PeopleSave - a drug recommendation and feedback system for doctors on the basis of contextual patient reviews crowd-sourced from the Internet. Unlike other systems proposed in the past, we filter information sources to check for crowdsourcing feasibility and then assess the drug’s effectiveness based on its reported detrimental effect on a patient. This helps in eliminating certain drugs that would almost certainly have an adverse effect on the patient’s health and thereby obtain a set of recommendable drugs. These recommendations are further refined by analyzing the sentiment behind the opinions of patients who have been administered these drugs in the past. The resultant set of prescribable drugs agrees with those suggested by the consulted physicians for the considered sample set of diabetes patients. The critical assessment of the prototype system for Diabetes Type II drugs by both doctors and patients also reiterates the need for a feedback system that can possibly go a long way in improving patient experience of a drug. This leads us to conclude that PeopleSave, as a combination of the recommendation system prototype and the proposed feedback system, can be successful in improving the process of prescription of medicines for a varied range of medical conditions.