Changes

{{Project Infobox|Self researchers=User:DG|Related tools=Anki,Spaced Repetition|Related topics=Tools for Cognitive Testing|Related projects=Of Trivial Value? Lessons From Using SuperMemo, Spaced Repetition: A Cognitive QS Method for Knowledge Acquisition, Knowledge Tracking, Memory and Learning}}  

Main place for news of this project is on a [https://forums.ankiweb.net/t/really-thorough-statistical-analysis-of-your-data/36355 thread in Anki forum] because most of the audience would be there.{{Project Infobox|Self researchers=User:DG|Related tools=Anki,Spaced Repetition|Related topics=Tools for Cognitive Testing|Related projects=Of Trivial Value? Lessons From Using SuperMemo, Spaced Repetition: A Cognitive QS Method for Knowledge Acquisition, Knowledge Tracking, Memory and Learning}}  

Flash cards are cards with question on one side and answers on opposite. They are used for memorization<ref>en.wikipedia.org/wiki/Testing_effect</ref>, making explicit<ref>en.wikipedia.org/wiki/Explicit_memory</ref> (requires effort to remember) declarative<ref>en.wikipedia.org/wiki/Declarative_learning</ref> semantic<ref>en.wikipedia.org/wiki/Semantic_memory</ref> memory though the goal of language learning is to make each memory automatic and therefore implicit.<ref>en.wikipedia.org/wiki/Implicit_memory</ref> Several computer apps automated the process and have recorded a lot of data. I started this project to analyze the data that [[Anki]] records because I expected electronic flashcard data to be useful as a [[Tools for Cognitive Testing|continuous cognitive test]]. If this works, the resulting information would be useful for Citizen Science and Health Tracking. Academic papers may be successfully replicated here even though neither complicated multi-step analysis (though common some places<ref>www.kaggle.com/competitions?hostSegmentIdFilter=2&searchQuery=university</ref><ref>paperswithcode.com/dataset/shhs</ref>) nor single subject longitudinal observational studies like this project are common on google scholar.   

Flash cards are cards with question on one side and answers on opposite. They are used for memorization<ref>en.wikipedia.org/wiki/Testing_effect</ref>, making explicit<ref>en.wikipedia.org/wiki/Explicit_memory</ref> (requires effort to remember) declarative<ref>en.wikipedia.org/wiki/Declarative_learning</ref> semantic<ref>en.wikipedia.org/wiki/Semantic_memory</ref> memory though the goal of language learning is to make each memory automatic and therefore implicit,<ref>en.wikipedia.org/wiki/Implicit_memory</ref> system one.<ref>bokcenter.harvard.edu/how-memory-works</ref> They do this through the principal of spaced repetition.<ref><nowiki>https://www.gwern.net/Spaced-repetition</nowiki></ref> Several computer apps automated the process and have recorded a lot of data. Inspired by [[Piotr Wozniak|Wozniak's]] and Gwern's<ref>gwern.net/treadmill#treadmill-effect-on-spaced-repetition-performance-randomized-experiment</ref> experiments, I started this project to analyze the data that [[Anki]] records because I expected electronic flashcard data to be useful as a [[Tools for Cognitive Testing|continuous cognitive test]]. If this works, the resulting information would be useful for Citizen Science and Health Tracking. Academic papers may be successfully replicated here even though neither complicated multi-step analysis (though common some places<ref>www.kaggle.com/competitions?hostSegmentIdFilter=2&searchQuery=university</ref><ref>paperswithcode.com/dataset/shhs</ref>) nor single subject longitudinal observational studies like this project are common on google scholar.   

Turns out the project will teach users about learning and allow them to experiment with and optimize their own learning process. All flashcard apps already optimize their student's learning but do not open the process to the user, except [[Piotr Wozniak|Super Memo 18]]. (this project already has different features) Resulting visualizations encourage studying by illustrating success in different way than existing apps, similar to gamification.   

Turns out the project will teach users about learning and allow them to experiment with and optimize their own learning process. All flashcard apps already optimize their student's learning but do not open the process to the user, except [[Piotr Wozniak|Super Memo 18]]. (this project already has different features) Resulting visualizations encourage studying by illustrating success in different way than existing apps, similar to gamification.   

The goal of teaching is almost as easy to guarantee because analyzing the data requires delving into the learning process. This goal may actually be the most impactful.  

The goal of teaching is almost as easy to guarantee because analyzing the data requires delving into the learning process. This goal may actually be the most impactful.  

Project will advise user on how to optimize their studying by comparing what actually happened with what would have happened if they had done something different, according to a machine learning model. Effects will be illustrated using Partial Dependence Plots and ICE<ref>Visualizing ML Models with LIME · UC Business Analytics R Programming Guide (uc-r.github.io)</ref>.       

Project will advise user on how to optimize their studying by comparing what actually happened with what would have happened if they had done something different,<ref>scite.ai/reports/very-long-term-memory-for-knowledge-nLe1R6</ref> according to a machine learning model. Effects will be illustrated using Partial Dependence Plots and ICE<ref>Visualizing ML Models with LIME · UC Business Analytics R Programming Guide (uc-r.github.io)</ref>.       

To help user conduct experiments, project will compare one part of time series with another or progress on one set of cards with another. User may have to use another tool like [[Open Humans]].     

To help user conduct experiments, project will compare one part of time series with another or progress on one set of cards with another. User may have to use another tool like [[Open Humans]].     

==== Potential Tests of Emotion ====

==== Potential Tests of Emotion ====

=== Cognitive Health ===

=== Cognitive Health ===

Everyone should track their cognitive ability as much as health-conscious people track their heartrate and exercise. IQ tests are supposed to have high 'reliability' and not change much between days for any individual. [[Quantified Mind|Formal cognitive testing]] takes too much time and effect on daily life of the specific thing the cognitive test tests is often questioned. Skill trainers and testers like typing tutors have none of the mentioned problems. However, even if a skill test is useful for optimizing the skill, things like dependence on psychological factors may not make it a good cognitive test. The skill test will have to correlate with validated cognitive tests or obviously important health things or at least transfer to other tests to become validated<ref>[[wikipedia:Validity_(statistics)|en.wikipedia.org/wiki/Validity_(statistics)]]</ref> as a cognitive or health test. The tests are unlikely to be pure in the sense of [[Quantified Mind]]' science page. This makes them better checks for general unhealth but harder to diagnose a specific problem with. The skill of memorizing from flashcards may be trainable. It may even transfer to memory in general.       

Everyone should track their cognitive ability as much as health-conscious people track their heartrate and exercise. Faster learning performance decline is a warning sign in elderly.<ref>n.neurology.org/content/42/2/396.short</ref><ref>www.tandfonline.com/doi/abs/10.1080/13825580600954256</ref> IQ tests are supposed to have high 'reliability' and not change much between days for any individual. [[Quantified Mind|Formal cognitive testing]] takes too much time and effect on daily life of the specific thing the cognitive test tests is often questioned. Skill trainers and testers like typing tutors have none of the mentioned problems. However, even if a skill test is useful for optimizing the skill, things like dependence on psychological factors may not make it a good cognitive test. The skill test will have to correlate with validated cognitive tests or obviously important health things or at least transfer to other tests to become validated<ref>[[wikipedia:Validity_(statistics)|en.wikipedia.org/wiki/Validity_(statistics)]]</ref> as a cognitive or health test. The tests are unlikely to be pure in the sense of [[Quantified Mind]]' science page. This makes them better checks for general unhealth but harder to diagnose a specific problem with. The skill of memorizing from flashcards may be trainable. It may even transfer to memory in general.       

Wozniak of supermemo has found correlation between sleep (recency, debt) and flash card skill.<ref>supermemo.guru/wiki/Sleep_and_learning#Studying_sleep_and_learning_with_SuperMemo</ref><ref>supermemo.guru/wiki/Biphasic_life#Biphasic_learning</ref><ref>supermemopedia.com/wiki/SleepChart</ref> Here alertness corresponds to recall.

Many elements of important theory of memory should be reproducible in this project even if the specific tests do not match exactly.<ref>Bjork, R. A., & Bjork, E. L. (1992). A new theory of disuse and an old theory of stimulus fluctuation. In A. F. Healy, S. M. Kosslyn, & R. M. Shiffrin (Eds.), ''Essays in honor of William K. Estes, Vol. 1. From learning theory to connectionist theory; Vol. 2. From learning processes to cognitive processes'' (pp. 35–67). Lawrence Erlbaum Associates, Inc.</ref> Most of the tests in the cited paper tested if item seen before (recognition) not if response was remembered with the help of a cue (cued recall).<ref>techpsych.wordpress.ncsu.edu/2020/03/18/hello-world/</ref> In the scientific papers the delay of the recall is usually small relative to spaced retrieval, and it is almost never performed continuously or longitudinally. Hopefully, the tests transfer to cognitive concept of memory and be important in everyday functioning.      

Many elements of important theory of memory should be reproducible in this project even if the specific tests do not match exactly.<ref>Bjork, R. A., & Bjork, E. L. (1992). A new theory of disuse and an old theory of stimulus fluctuation. In A. F. Healy, S. M. Kosslyn, & R. M. Shiffrin (Eds.), ''Essays in honor of William K. Estes, Vol. 1. From learning theory to connectionist theory; Vol. 2. From learning processes to cognitive processes'' (pp. 35–67). Lawrence Erlbaum Associates, Inc.</ref> Most of the tests in the cited paper tested if item seen before (recogntion) not if response was remembered with the help of a cue (cued recall).<ref>techpsych.wordpress.ncsu.edu/2020/03/18/hello-world/</ref> In the scientific papers the delay of the recall is usualy small relative to spaced retrieval, and it is never performed continuously or longitudinally. Hopefully, the tests transfer to cognitive concept of memory and be important in everyday functioning.        

Most aspects of 1-day delayed free recall become impaired different amounts as age increases. <ref>Acquisition, Recall, and Forgetting of Verbal Information in Long-Term Memory by Young, Middle-Aged, and Elderly Individuals,

 

Cortex, Volume 39, Issues 4–5, 2003,  

 

academiccommons.columbia.edu/doi/10.7916/9p9w-tc61/download</ref>         

=== Specific cognitive health tests ===

=== Specific cognitive health tests ===

Goal of Anik skill is to minimize user time required to remember information on a card to a target future date. Decomposition of the general skill<ref>[[wikipedia:Job_analysis|en.wikipedia.org/wiki/Job_analysis]]</ref><ref>Foundations of Fluency: An Exploration: Reading Psychology: Vol 26, No 2 (tandfonline.com) www.tandfonline.com/doi/abs/10.1080/02702710590930519</ref> by looking for sources of time loss results in learning speed, relearning speed, forgetting, ability to remember effectively today, slowness to answer, and distraction. Each corresponds to a potential test and it's validations. New theory of Disuse separates memory into retrieval strength and storage strength or middle and long term memory.<ref>www.learningscientists.org/blog/2016/5/10-1</ref><ref>www.researchgate.net/profile/Robert-Bjork-2/publication/281322665_A_new_theory_of_disuse_and_an_old_theory_of_stimulus_fluctuation/links/58b6f20945851591c5d55e96/A-new-theory-of-disuse-and-an-old-theory-of-stimulus-fluctuation.pdf</ref>

Goal of Anik skill is to minimize user time required to remember information on a card to a target future date. Decomposition of the general skill<ref>[[wikipedia:Job_analysis|en.wikipedia.org/wiki/Job_analysis]]</ref><ref>Foundations of Fluency: An Exploration: Reading Psychology: Vol 26, No 2 (tandfonline.com) www.tandfonline.com/doi/abs/10.1080/02702710590930519</ref> by looking for sources of time loss results in learning speed, relearning speed, forgetting, memory consolidation, ability to remember effectively today, slowness to answer, and distraction.<ref>compmem.princeton.edu/wp/wp-content/uploads/2020/04/replay-based-consolidation-governs-enduring-memory-storage.pdf</ref> Each corresponds to a potential test and it's validations. Consider "test-retest" reliability<ref>www.tandfonline.com/doi/abs/10.1080/13854046.2017.1310300</ref> and applicability to longitudinal studies. New theory of Disuse separates memory into retrieval strength and storage strength or middle and long term memory.<ref>www.learningscientists.org/blog/2016/5/10-1</ref><ref>www.researchgate.net/profile/Robert-Bjork-2/publication/281322665_A_new_theory_of_disuse_and_an_old_theory_of_stimulus_fluctuation/links/58b6f20945851591c5d55e96/A-new-theory-of-disuse-and-an-old-theory-of-stimulus-fluctuation.pdf</ref>

==== Consolidation of Memory<ref>supermemo.guru/wiki/Memory_consolidation</ref>====

 

Most tests are intended to diagnose clinical (sick) persons, not to analyze cognition in healthy people.<ref>www.ncbi.nlm.nih.gov/pmc/articles/PMC5829025/</ref>

==== Consolidation of Memory<ref>supermemo.guru/wiki/Memory_consolidation</ref><ref>www.ncbi.nlm.nih.gov/mesh/2009921</ref>====

If on a set day the cards were memorized better or worse as will be seen on their next review. This is like the following two potential identical tests but more generally about change and not about speed?     

If on a set day the cards were memorized better or worse as will be seen on their next review. This is like the following two potential identical tests but more generally about change and not about speed?     

==== Recall<ref>supermemo.guru/wiki/Recall</ref>====

==== Recall<ref>supermemo.guru/wiki/Recall</ref>====

Remembering on a given day. If long-term then "retention". Wozniak explains it better but I would say that this should be separated in to real forgetting that changes state for several reviews or just momentary lapse. Both are measured separately making two tests. There are also more mild state of card consolidation changes that may happen that are not just forgetting. This may be a third test.  

Remembering on a given day. If long-term then "retention". Wozniak explains it better but I would say that this should be separated in to real forgetting that changes state for several reviews or just momentary lapse. Both are measured separately making two tests. There are also more mild state of card consolidation changes that may happen that are not just forgetting. This may be a third test.  

Long delay cued recall (or CVLT?) declines with age but improves with testing (testing effect on specific memory or the general skill?).<ref>Longitudinal changes in verbal memory in older adults

Long delay cued recall (or CVLT?) declines with age but improves with testing (testing effect on specific memory or the general skill?).<ref>Longitudinal changes in verbal memory in older adults

Neurology Jan 2003, 60 (1) 82-86; DOI: 10.1212/WNL.60.1.82 </ref>

Neurology Jan 2003, 60 (1) 82-86; DOI: 10.1212/WNL.60.1.82 </ref>

===== Time taken to learn a new word =====

===== Time taken to learn a new word =====

Short-term memory, from seconds to hours.<ref>Memory, Short-Term - MeSH - NCBI (nih.gov) www.ncbi.nlm.nih.gov/mesh/68008570</ref> If considering only the first session, learning new words in Anki is like cued verbal learning like in the CVLT, a cognitive test. Though in the CVLT<ref>en.wikipedia.org/wiki/California_Verbal_Learning_Test</ref> the cue is much weaker. And verbal learning transfers to other abilities: "There is considerable evidence that verbal learning correlates reasonably strongly with performance in a number of important practical tasks. For instance, verbal learning tests have been demonstrated to be highly correlated with prospective remembering in real life, which means remembering to perform a planned action at the appropriate time [29]."<ref>www.quantified-mind.com/science</ref> Sever sleep deprivation harms verbal learning.<ref>pubmed.ncbi.nlm.nih.gov/10688201/</ref> Most of time spent on this task may be on short term memory (10 min) and not mid-long term memory (1 day) so the real test could be percent of successful learnings of new words and not speed.  

Short-term memory, from seconds to hours.<ref>Memory, Short-Term - MeSH - NCBI (nih.gov) www.ncbi.nlm.nih.gov/mesh/68008570</ref> If considering only the first session, learning new words in Anki is like cued verbal learning like in the CVLT, a cognitive test. Though in the CVLT<ref>en.wikipedia.org/wiki/California_Verbal_Learning_Test</ref> the cue is much weaker. And verbal learning transfers to other abilities: "There is considerable evidence that verbal learning correlates reasonably strongly with performance in a number of important practical tasks. For instance, verbal learning tests have been demonstrated to be highly correlated with prospective remembering in real life, which means remembering to perform a planned action at the appropriate time [29]."<ref>www.quantified-mind.com/science</ref> Sever sleep deprivation harms verbal learning.<ref>pubmed.ncbi.nlm.nih.gov/10688201/</ref> Most of time spent on this task may be on short term memory (10 min) and not mid-long term memory (1 day) so the real test could be percent of successful learnings of new words and not speed. Age and memory complaints correlate with poorer cued recall at 55min delay and the test may identify subtle impairments.<ref>pubmed.ncbi.nlm.nih.gov/35786222/</ref>

===== Time taken to relearn a forgotten word =====     

===== Time taken to relearn a forgotten word =====     

Judgement of learning could be after cue (question) or answer. Delayed, cue only, JOL produced highest correlation with actual recall. If answer is added though, the effect goes way down. Worse still, study decisions are not better than only rereading. This seems to be because users do not realize they learned because of last test.<ref>Kornell, Nate, and Matthew G. Rhodes. "Feedback Reduces the Metacognitive Benefit of Tests." ''Journal of Experimental Psychology: Applied'', vol. 19, no. 1, Mar. 2013, p. 1. </ref> Metacognitive. Making JOLs also increases memory of items, maybe.   

Judgement of learning could be after cue (question) or answer. Delayed, cue only, JOL produced highest correlation with actual recall. If answer is added though, the effect goes way down. Worse still, study decisions are not better than only rereading. This seems to be because users do not realize they learned because of last test.<ref>Kornell, Nate, and Matthew G. Rhodes. "Feedback Reduces the Metacognitive Benefit of Tests." ''Journal of Experimental Psychology: Applied'', vol. 19, no. 1, Mar. 2013, p. 1. </ref> Metacognitive. Making JOLs also increases memory of items, maybe.   

== References ==  

== References ==