Skip to main content
Top
Gepubliceerd in: Journal of Behavioral Medicine 2/2019

25-08-2018

Can the artificial intelligence technique of reinforcement learning use continuously-monitored digital data to optimize treatment for weight loss?

Auteurs: Evan M. Forman, Stephanie G. Kerrigan, Meghan L. Butryn, Adrienne S. Juarascio, Stephanie M. Manasse, Santiago Ontañón, Diane H. Dallal, Rebecca J. Crochiere, Danielle Moskow

Gepubliceerd in: Journal of Behavioral Medicine | Uitgave 2/2019

Log in om toegang te krijgen
share
DELEN

Deel dit onderdeel of sectie (kopieer de link)

  • Optie A:
    Klik op de rechtermuisknop op de link en selecteer de optie “linkadres kopiëren”
  • Optie B:
    Deel de link per e-mail

Abstract

Behavioral weight loss (WL) trials show that, on average, participants regain lost weight unless provided long-term, intensive—and thus costly—intervention. Optimization solutions have shown mixed success. The artificial intelligence principle of “reinforcement learning” (RL) offers a new and more sophisticated form of optimization in which the intensity of each individual’s intervention is continuously adjusted depending on patterns of response. In this pilot, we evaluated the feasibility and acceptability of a RL-based WL intervention, and whether optimization would achieve equivalent benefit at a reduced cost compared to a non-optimized intensive intervention. Participants (n = 52) completed a 1-month, group-based in-person behavioral WL intervention and then (in Phase II) were randomly assigned to receive 3 months of twice-weekly remote interventions that were non-optimized (NO; 10-min phone calls) or optimized (a combination of phone calls, text exchanges, and automated messages selected by an algorithm). The Individually-Optimized (IO) and Group-Optimized (GO) algorithms selected interventions based on past performance of each intervention for each participant, and for each group member that fit into a fixed amount of time (e.g., 1 h), respectively. Results indicated that the system was feasible to deploy and acceptable to participants and coaches. As hypothesized, we were able to achieve equivalent Phase II weight losses (NO = 4.42%, IO = 4.56%, GO = 4.39%) at roughly one-third the cost (1.73 and 1.77 coaching hours/participant for IO and GO, versus 4.38 for NO), indicating strong promise for a RL system approach to weight loss and maintenance.
Literatuur
go back to reference Abbeel, P., & Ng, A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. Paper presented at the Proceedings of the twenty-first international conference on Machine learning. Abbeel, P., & Ng, A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. Paper presented at the Proceedings of the twenty-first international conference on Machine learning.
go back to reference Anderson, J. W., Vichitbandra, S., Qian, W., & Kryscio, R. J. (1999). Long-term weight maintenance after an intensive weight-loss program. Journal of the American College of Nutrition, 18, 620–627.CrossRefPubMed Anderson, J. W., Vichitbandra, S., Qian, W., & Kryscio, R. J. (1999). Long-term weight maintenance after an intensive weight-loss program. Journal of the American College of Nutrition, 18, 620–627.CrossRefPubMed
go back to reference Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47, 235–256.CrossRef Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47, 235–256.CrossRef
go back to reference Aydin, M. E., & Öztemel, E. (2000). Dynamic job-shop scheduling using reinforcement learning agents. Robotics and Autonomous Systems, 33, 169–178.CrossRef Aydin, M. E., & Öztemel, E. (2000). Dynamic job-shop scheduling using reinforcement learning agents. Robotics and Autonomous Systems, 33, 169–178.CrossRef
go back to reference Brindal, E., Freyne, J., Saunders, I., Berkovsky, S., Smith, G., & Noakes, M. (2012). Features predicting weight loss in overweight or obese participants in a web-based intervention: Randomized trial. Journal of medical Internet research, 14, e173.CrossRefPubMedPubMedCentral Brindal, E., Freyne, J., Saunders, I., Berkovsky, S., Smith, G., & Noakes, M. (2012). Features predicting weight loss in overweight or obese participants in a web-based intervention: Randomized trial. Journal of medical Internet research, 14, e173.CrossRefPubMedPubMedCentral
go back to reference Butryn, M. L., Forman, E. M., Lowe, M. R., Gorin, A. A., Zhang, F., & Schaumberg, K. (2017). Efficacy of environmental and acceptance-based enhancements to behavioral weight loss treatment: The ENACT trial. Obesity, 25, 866–872.CrossRefPubMed Butryn, M. L., Forman, E. M., Lowe, M. R., Gorin, A. A., Zhang, F., & Schaumberg, K. (2017). Efficacy of environmental and acceptance-based enhancements to behavioral weight loss treatment: The ENACT trial. Obesity, 25, 866–872.CrossRefPubMed
go back to reference Butryn, M. L., Zhang, F., Remmert, J. E., Roberts, S. R., & Forman, E. M. (2018). Baseline Executive Functioning Predicts Weight Loss and Physical Activity Outcomes in A Lifestyle Modification Program. Paper presented at the 39th Annual Meeting and Scientific Sessions of the Society of Behavioral Medicine, New Orleans, LA. Butryn, M. L., Zhang, F., Remmert, J. E., Roberts, S. R., & Forman, E. M. (2018). Baseline Executive Functioning Predicts Weight Loss and Physical Activity Outcomes in A Lifestyle Modification Program. Paper presented at the 39th Annual Meeting and Scientific Sessions of the Society of Behavioral Medicine, New Orleans, LA.
go back to reference Carels, R. A., Darby, L., Cacciapaglia, H. M., Douglass, O. M., Harper, J., Kaplar, M. E., et al. (2005). Applying a stepped-care approach to the treatment of obesity. Journal of Psychosomatic Research, 59, 375–383.CrossRefPubMed Carels, R. A., Darby, L., Cacciapaglia, H. M., Douglass, O. M., Harper, J., Kaplar, M. E., et al. (2005). Applying a stepped-care approach to the treatment of obesity. Journal of Psychosomatic Research, 59, 375–383.CrossRefPubMed
go back to reference Carels, R. A., Darby, L., Cacciapaglia, H. M., Konrad, K., Coit, C., Harper, J., et al. (2007). Using motivational interviewing as a supplement to obesity treatment: A stepped-care approach. Health Psychology, 26, 369.CrossRefPubMed Carels, R. A., Darby, L., Cacciapaglia, H. M., Konrad, K., Coit, C., Harper, J., et al. (2007). Using motivational interviewing as a supplement to obesity treatment: A stepped-care approach. Health Psychology, 26, 369.CrossRefPubMed
go back to reference Carels, R. A., Hoffmann, D. A., Hinman, N., Burmeister, J. M., Koball, A., Ashrafioun, L., et al. (2013). Step-down approach to behavioural weight loss treatment: A pilot of a randomised clinical trial. Psychology & Health, 28, 1121–1134.CrossRef Carels, R. A., Hoffmann, D. A., Hinman, N., Burmeister, J. M., Koball, A., Ashrafioun, L., et al. (2013). Step-down approach to behavioural weight loss treatment: A pilot of a randomised clinical trial. Psychology & Health, 28, 1121–1134.CrossRef
go back to reference Carels, R. A., Selensky, J. C., Rossi, J., Solar, C., & Hlavka, R. (2017). A novel stepped-care approach to weight loss: The role of self-monitoring and health literacy in treatment outcomes. Eating Behaviors, 26, 76–82.CrossRefPubMed Carels, R. A., Selensky, J. C., Rossi, J., Solar, C., & Hlavka, R. (2017). A novel stepped-care approach to weight loss: The role of self-monitoring and health literacy in treatment outcomes. Eating Behaviors, 26, 76–82.CrossRefPubMed
go back to reference Diabetes Prevention Program Research Group. (2003). Costs associated with the primary prevention of type 2 diabetes mellitus in the diabetes prevention program. Diabetes Care, 26, 36–47.CrossRef Diabetes Prevention Program Research Group. (2003). Costs associated with the primary prevention of type 2 diabetes mellitus in the diabetes prevention program. Diabetes Care, 26, 36–47.CrossRef
go back to reference Forman, E. M., Butryn, M. L., Manasse, S. M., Crosby, R. D., Goldstein, S. P., Wyckoff, E. P., et al. (2016). Acceptance-based versus standard behavioral treatment for obesity: Results from the mind your health randomized controlled trial. Obesity, 24, 2050–2056.CrossRefPubMed Forman, E. M., Butryn, M. L., Manasse, S. M., Crosby, R. D., Goldstein, S. P., Wyckoff, E. P., et al. (2016). Acceptance-based versus standard behavioral treatment for obesity: Results from the mind your health randomized controlled trial. Obesity, 24, 2050–2056.CrossRefPubMed
go back to reference Jakicic, J. M., Tate, D. F., Lang, W., Davis, K. K., Polzien, K., Rickman, A. D., et al. (2012). Effect of a stepped-care intervention approach on weight loss in adults: A randomized clinical trial. JAMA, 307, 2617–2626.CrossRefPubMedPubMedCentral Jakicic, J. M., Tate, D. F., Lang, W., Davis, K. K., Polzien, K., Rickman, A. D., et al. (2012). Effect of a stepped-care intervention approach on weight loss in adults: A randomized clinical trial. JAMA, 307, 2617–2626.CrossRefPubMedPubMedCentral
go back to reference Jeffery, R. W., Epstein, L. H., Wilson, G. T., Drewnowski, A., Stunkard, A. J., & Wing, R. R. (2000). Long-term maintenance of weight loss: Current status. Health Psychology, 19, 5.CrossRefPubMed Jeffery, R. W., Epstein, L. H., Wilson, G. T., Drewnowski, A., Stunkard, A. J., & Wing, R. R. (2000). Long-term maintenance of weight loss: Current status. Health Psychology, 19, 5.CrossRefPubMed
go back to reference Joo, N.-S., & Kim, B.-T. (2007). Mobile phone short message service messaging for behaviour modification in a community-based weight control programme in Korea. Journal of Telemedicine and Telecare, 13, 416–420.CrossRefPubMed Joo, N.-S., & Kim, B.-T. (2007). Mobile phone short message service messaging for behaviour modification in a community-based weight control programme in Korea. Journal of Telemedicine and Telecare, 13, 416–420.CrossRefPubMed
go back to reference Korinek, E. V., Phatak, S. S., Martin, C. A., Freigoun, M. T., Rivera, D. E., Adams, M. A., et al. (2018). Adaptive step goals and rewards: A longitudinal growth model of daily steps for a smartphone-based walking intervention. Journal of Behavioral Medicine, 41, 74–86.CrossRefPubMed Korinek, E. V., Phatak, S. S., Martin, C. A., Freigoun, M. T., Rivera, D. E., Adams, M. A., et al. (2018). Adaptive step goals and rewards: A longitudinal growth model of daily steps for a smartphone-based walking intervention. Journal of Behavioral Medicine, 41, 74–86.CrossRefPubMed
go back to reference Kramer, F. M., Jeffery, R. W., Forster, J. L., & Snell, M. K. (1989). Long-term follow-up of behavioral treatment for obesity: Patterns of weight regain among men and women. International Journal of Obesity, 13, 123–136.PubMed Kramer, F. M., Jeffery, R. W., Forster, J. L., & Snell, M. K. (1989). Long-term follow-up of behavioral treatment for obesity: Patterns of weight regain among men and women. International Journal of Obesity, 13, 123–136.PubMed
go back to reference Krukowski, R. A., Tilford, J. M., Harvey-Berino, J., & West, D. S. (2011). Comparing behavioral weight loss modalities: Incremental cost-effectiveness of an internet-based versus an in-person condition. Obesity, 19, 1629–1635.CrossRefPubMed Krukowski, R. A., Tilford, J. M., Harvey-Berino, J., & West, D. S. (2011). Comparing behavioral weight loss modalities: Incremental cost-effectiveness of an internet-based versus an in-person condition. Obesity, 19, 1629–1635.CrossRefPubMed
go back to reference Kuderer, M., Gulati, S., & Burgard, W. (2015). Learning driving styles for autonomous vehicles from demonstration. Paper presented at the 2015 IEEE International Conference on Robotics and Automation (ICRA). Kuderer, M., Gulati, S., & Burgard, W. (2015). Learning driving styles for autonomous vehicles from demonstration. Paper presented at the 2015 IEEE International Conference on Robotics and Automation (ICRA).
go back to reference McGraa, K. L. K. (2010). The effects of persuasive motivational text messaging on adherence to diet and exercise programs across different personality traits. Santa Barbara: Fielding Graduate University. McGraa, K. L. K. (2010). The effects of persuasive motivational text messaging on adherence to diet and exercise programs across different personality traits. Santa Barbara: Fielding Graduate University.
go back to reference Ng, A. Y., Coates, A., Diel, M., Ganapathi, V., Schulte, J., Tse, B., et al. (2006). Autonomous inverted helicopter flight via reinforcement learning. In M. H. Ang & O. Khatib (Eds.), Experimental Robotics IX (pp. 363–372). Berlin: Springer.CrossRef Ng, A. Y., Coates, A., Diel, M., Ganapathi, V., Schulte, J., Tse, B., et al. (2006). Autonomous inverted helicopter flight via reinforcement learning. In M. H. Ang & O. Khatib (Eds.), Experimental Robotics IX (pp. 363–372). Berlin: Springer.CrossRef
go back to reference Ontanón, S. (2013). The combinatorial multi-armed bandit problem and its application to real-time strategy games. Paper presented at the Ninth Artificial Intelligence and Interactive Digital Entertainment Conference. Ontanón, S. (2013). The combinatorial multi-armed bandit problem and its application to real-time strategy games. Paper presented at the Ninth Artificial Intelligence and Interactive Digital Entertainment Conference.
go back to reference Ontanón, S. (2017). Combinatorial multi-armed bandits for real-time strategy games. Journal of Artificial Intelligence Research, 58, 665–702.CrossRef Ontanón, S. (2017). Combinatorial multi-armed bandits for real-time strategy games. Journal of Artificial Intelligence Research, 58, 665–702.CrossRef
go back to reference Paredes, P., Gilad-Bachrach, R., Czerwinski, M., Roseway, A., Rowan, K., & Hernandez, J. (2014). PopTherapy: Coping with stress through pop-culture. Paper presented at the Proceedings of the 8th International Conference on Pervasive Computing Technologies for Healthcare. Paredes, P., Gilad-Bachrach, R., Czerwinski, M., Roseway, A., Rowan, K., & Hernandez, J. (2014). PopTherapy: Coping with stress through pop-culture. Paper presented at the Proceedings of the 8th International Conference on Pervasive Computing Technologies for Healthcare.
go back to reference Patrick, K., Raab, F., Adams, M. A., Dillon, L., Zabinski, M., Rock, C. L., et al. (2009). A text message–based intervention for weight loss: Randomized controlled trial. Journal of medical Internet research, 11, e1.CrossRefPubMedPubMedCentral Patrick, K., Raab, F., Adams, M. A., Dillon, L., Zabinski, M., Rock, C. L., et al. (2009). A text message–based intervention for weight loss: Randomized controlled trial. Journal of medical Internet research, 11, e1.CrossRefPubMedPubMedCentral
go back to reference Phatak, S. S., Freigoun, M. T., Martín, C. A., Rivera, D. E., Korinek, E. V., Adams, M. A., et al. (2018). Modeling individual differences: A case study of the application of system identification for personalizing a physical activity intervention. Journal of Biomedical Informatics, 79, 82–97.CrossRefPubMed Phatak, S. S., Freigoun, M. T., Martín, C. A., Rivera, D. E., Korinek, E. V., Adams, M. A., et al. (2018). Modeling individual differences: A case study of the application of system identification for personalizing a physical activity intervention. Journal of Biomedical Informatics, 79, 82–97.CrossRefPubMed
go back to reference Piette, J. D., Farris, K. B., Newman, S., An, L., Sussman, J., & Singh, S. (2014). The potential impact of intelligent systems for mobile health self-management support: Monte Carlo simulations of text message support for medication adherence. Annals of Behavioral Medicine, 49, 84–94.CrossRefPubMedCentral Piette, J. D., Farris, K. B., Newman, S., An, L., Sussman, J., & Singh, S. (2014). The potential impact of intelligent systems for mobile health self-management support: Monte Carlo simulations of text message support for medication adherence. Annals of Behavioral Medicine, 49, 84–94.CrossRefPubMedCentral
go back to reference Piette, J. D., Krein, S. L., Striplin, D., Marinec, N., Kerns, R. D., Farris, K. B., et al. (2016). Patient-centered pain care using artificial intelligence and mobile health tools: Protocol for a randomized study funded by the US Department of Veterans Affairs Health Services Research and Development Program. JMIR Research Protocols, 5, e53.CrossRefPubMedPubMedCentral Piette, J. D., Krein, S. L., Striplin, D., Marinec, N., Kerns, R. D., Farris, K. B., et al. (2016). Patient-centered pain care using artificial intelligence and mobile health tools: Protocol for a randomized study funded by the US Department of Veterans Affairs Health Services Research and Development Program. JMIR Research Protocols, 5, e53.CrossRefPubMedPubMedCentral
go back to reference Poirier, J., Bennett, W. L., Jerome, G. J., Shah, N. G., Lazo, M., Yeh, H.-C., et al. (2016). Effectiveness of an activity tracker-and internet-based adaptive walking program for adults: A randomized controlled trial. Journal of Medical Internet Research, 18, e34.CrossRefPubMedPubMedCentral Poirier, J., Bennett, W. L., Jerome, G. J., Shah, N. G., Lazo, M., Yeh, H.-C., et al. (2016). Effectiveness of an activity tracker-and internet-based adaptive walking program for adults: A randomized controlled trial. Journal of Medical Internet Research, 18, e34.CrossRefPubMedPubMedCentral
go back to reference Rabbi, M., Aung, M. H., Zhang, M., & Choudhury, T. (2015). MyBehavior: Automatic personalized health feedback from user behaviors and preferences using smartphones. Paper presented at the Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. Rabbi, M., Aung, M. H., Zhang, M., & Choudhury, T. (2015). MyBehavior: Automatic personalized health feedback from user behaviors and preferences using smartphones. Paper presented at the Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing.
go back to reference Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., et al. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529, 484–489.CrossRefPubMed Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., et al. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529, 484–489.CrossRefPubMed
go back to reference Snook, K. R., Hansen, A. R., Duke, C. H., Finch, K. C., Hackney, A. A., & Zhang, J. (2017). Change in percentages of adults with overweight or obesity trying to lose weight, 1988–2014. JAMA, 317, 971–973.CrossRefPubMed Snook, K. R., Hansen, A. R., Duke, C. H., Finch, K. C., Hackney, A. A., & Zhang, J. (2017). Change in percentages of adults with overweight or obesity trying to lose weight, 1988–2014. JAMA, 317, 971–973.CrossRefPubMed
go back to reference Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction (Vol. 1). Cambridge: MIT press Cambridge. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction (Vol. 1). Cambridge: MIT press Cambridge.
go back to reference Wadden, T. A., Butryn, M. L., & Wilson, C. (2007). Lifestyle modification for the management of obesity. Gastroenterology, 132, 2226–2238.CrossRefPubMed Wadden, T. A., Butryn, M. L., & Wilson, C. (2007). Lifestyle modification for the management of obesity. Gastroenterology, 132, 2226–2238.CrossRefPubMed
go back to reference Wadden, T. A., Sternberg, J., Letizia, K., Stunkard, A., & Foster, G. (1989). Treatment of obesity by very low calorie diet, behavior therapy, and their combination: A five-year perspective. International Journal of Obesity, 13, 39–46.PubMed Wadden, T. A., Sternberg, J., Letizia, K., Stunkard, A., & Foster, G. (1989). Treatment of obesity by very low calorie diet, behavior therapy, and their combination: A five-year perspective. International Journal of Obesity, 13, 39–46.PubMed
go back to reference Webber, K. H., Tate, D. F., & Bowling, J. M. (2008). A randomized comparison of two motivationally enhanced Internet behavioral weight loss programs. Behaviour Research and Therapy, 46, 1090–1095.CrossRefPubMed Webber, K. H., Tate, D. F., & Bowling, J. M. (2008). A randomized comparison of two motivationally enhanced Internet behavioral weight loss programs. Behaviour Research and Therapy, 46, 1090–1095.CrossRefPubMed
go back to reference Wilson, G. T. (1994). Behavioral treatment of obesity: Thirty years and counting. Advances in Behaviour Research and Therapy, 16, 31–75.CrossRef Wilson, G. T. (1994). Behavioral treatment of obesity: Thirty years and counting. Advances in Behaviour Research and Therapy, 16, 31–75.CrossRef
go back to reference Wilson, G. T., & Brownell, K. D. (2002). Behavioral treatment for obesity. In C. G. Fairburn & K. D. Brownell (Eds.), Eating disorders and obesity. New York: Guilford Press. Wilson, G. T., & Brownell, K. D. (2002). Behavioral treatment for obesity. In C. G. Fairburn & K. D. Brownell (Eds.), Eating disorders and obesity. New York: Guilford Press.
go back to reference Yom-Tov, E., Feraru, G., Kozdoba, M., Mannor, S., Tennenholtz, M., & Hochberg, I. (2017). Encouraging physical activity in patients with diabetes: Intervention using a reinforcement learning system. Journal of Medical Internet Research, 19, e338.CrossRefPubMedPubMedCentral Yom-Tov, E., Feraru, G., Kozdoba, M., Mannor, S., Tennenholtz, M., & Hochberg, I. (2017). Encouraging physical activity in patients with diabetes: Intervention using a reinforcement learning system. Journal of Medical Internet Research, 19, e338.CrossRefPubMedPubMedCentral
go back to reference Zhou, M., Fukuoka, Y., Mintz, Y., Goldberg, K., Kaminsky, P., Flowers, E., et al. (2018). Evaluating machine learning-based automated personalized daily step goals delivered through a Mobile Phone App: Randomized controlled trial. JMIR mHealth and uHealth, 6, e28.CrossRefPubMedPubMedCentral Zhou, M., Fukuoka, Y., Mintz, Y., Goldberg, K., Kaminsky, P., Flowers, E., et al. (2018). Evaluating machine learning-based automated personalized daily step goals delivered through a Mobile Phone App: Randomized controlled trial. JMIR mHealth and uHealth, 6, e28.CrossRefPubMedPubMedCentral
Metagegevens
Titel
Can the artificial intelligence technique of reinforcement learning use continuously-monitored digital data to optimize treatment for weight loss?
Auteurs
Evan M. Forman
Stephanie G. Kerrigan
Meghan L. Butryn
Adrienne S. Juarascio
Stephanie M. Manasse
Santiago Ontañón
Diane H. Dallal
Rebecca J. Crochiere
Danielle Moskow
Publicatiedatum
25-08-2018
Uitgeverij
Springer US
Gepubliceerd in
Journal of Behavioral Medicine / Uitgave 2/2019
Print ISSN: 0160-7715
Elektronisch ISSN: 1573-3521
DOI
https://doi.org/10.1007/s10865-018-9964-1

Andere artikelen Uitgave 2/2019

Journal of Behavioral Medicine 2/2019 Naar de uitgave