flaw in your spaced repetition algorithm

zeppo's Avatar

zeppo

18 Jun, 2017 12:07 AM

I have used your app to study Spanish for years. So many notes (cards) are not scheduled to put me to the test again for a year or more. The algorithm has decided that my mastery of the note shows I don't need a follow up until then. Yet occasionally when reading Spanish I've struggled with word that I know I have included among my notes (cards) in Studies, and not remembering having studied recently, I've checked on its next due date to see if the algorithm might be failing me. Well, I came across a card that is non-reversable (Question and Answer), so there is only one prompt to be scheduled. It shows that I have studied it 32 times, with a 75% success rate, and with the last time being May 29, 2017. It shows that I failed it that date, because my "Recent Run" is 1 incorrect. Yet is has not scheduled to test me again with the card until Dec 4, 2021.

Obviously that is a flaw. If you miss a card, it should not wait to test you again until 3 1/2 years later!

  1. Support Staff 1 Posted by drewmccormack on 19 Jun, 2017 06:36 AM

    drewmccormack's Avatar

    Yes, that definitely sounds like a bug, though it is difficult to know exactly how it would come about. The basic scheduling algorithm is unlikely to have such a straightforward bug. More likely it is due to some complex situation with sync.

    Perhaps, e.g., you studied it twice, once on the Mac, and also on iOS. On the Mac, it was graded wrong, and on iOS it was correct an hour later. The algorithm will then have to decide what to do with that info, and may ignore one of the study events.

    That is a possible explanation, but I can't be entirely sure.

    The Note menu includes some options to get notes back into the study schedule if you do see something like this. Select the note, and choose to 'Restart the Study Schedule' (or Reset, which removes the whole history).

    Kind regards,
    Drew

  2. drewmccormack closed this discussion on 19 Jun, 2017 06:36 AM.

  3. zeppo re-opened this discussion on 20 Jun, 2017 01:16 AM

  4. 2 Posted by zeppo on 20 Jun, 2017 01:16 AM

    zeppo's Avatar

    It is the column to the left of where flags appear in list view. Left hand side, oppposite of where the tag column is.

  5. Support Staff 3 Posted by drewmccormack on 20 Jun, 2017 06:22 AM

    drewmccormack's Avatar

    I see. There is the flag column, and the due column. The last column there is just indentation for the section.

    I like the idea of having some sort of progress, probably the retention score as a pie chart. Is a good feature request.

    Thanks!
    Drew

  6. drewmccormack closed this discussion on 20 Jun, 2017 06:22 AM.

  7. zeppo re-opened this discussion on 20 Jun, 2017 05:05 PM

  8. 4 Posted by zeppo on 20 Jun, 2017 05:05 PM

    zeppo's Avatar

    However you do it, it would be best if you don't compromise function for aesthetics. I see it as something that could be very useful. So I'm not sure a pie chart would offer as much in the way of usefulness, and in fact might just make it more frustrating in trying to discern the difference in one pie the next. The way I see it as being useful is to add it column for quick visual identification, and different colors would be easier that a pie, especially if we are talking about small icons, or if you do a pie, change not only the fill of the pie, but the color of the fill. Having to dig for it would also not be useful, ie, having to tap on somewhere on each note individually as is already the case. If syncing issues can cause a failure in scheduling, this icon might be a way of noticing easily if something has gone wrong, perhaps even to a whole group of cards. Especially if it were to unbeknownst to the user schedule cards for a longer delay in studying than is appropriate to the mastery of the card, as is this case (or vice versa). I like the idea of indicating the span of time between the previous study and the next schedule date, rather than, say, the percentage of success (I don't think that would be useful), because it is what lets me know if there has been a problem in scheduling and is a fail safe to sync issues that might arise.

    But also, if one is able to sort by the degree of mastery, like you can with flagged cards, then that would allow me to easily review some well mastered cards and see if I might retire them.

    Another thought is that if adding an icon is a big deal, maybe just automatically adding a tag would be easier. It would just be best if it weren't a common word that is likely to be a tag the user might use or sort for within his tags or notes. So automatically adding tags like A^^1 B^^2 etc, for different levels would off the top of my head be unlikely to be used elsewhere by the user. Then one could sort by the tag to have a quick glance at what has been mastered. I think basic colors would be easier still, but not if your color blind I suppose.

  9. Support Staff 5 Posted by drewmccormack on 21 Jun, 2017 07:07 AM

    drewmccormack's Avatar

    I think some of what you would like may already be in there. There is no visual indication of how well you know the notes, which would be good to have, but there is a way to sort them based on that. Just click the button in the search field. You can sort based on retention probability, which is ‘mastery’ effectively.

    Drew

  10. 6 Posted by zeppo on 21 Jun, 2017 02:28 PM

    zeppo's Avatar

    Thanks for letting me know about this. Its certainly better than nothing, although it still separates and sorts by the stack when in "all notes" view. It would be better if in "all notes" it didn't group them by stack.

    Also, I much prefer the idea of using the time span between previous study and next scheduled study date because might be used as an alert that a sync error has occurred and botched up a bunch of cards, given the case of the card that I had gotten wrong on my last study date yet isn't scheduled to be studied again again until years later. The mastery percentage wouldn't indicate a problem where as the gap in study time did, because I knew it was a card that I didn't know as well as that scheduled date would imply. But I had to dig to find that error (assuming it is an error rather than an algorithm flaw), and as it stands I won't know how many other cards might be affected by that same sync problem unless I dig into each one. Sorting by mastery percentage would reveal nothing. Not to mention that the mastery percentage is just one factor in the gap until the next scheduled study date, as the number of times studied is also a factor, and the most recent successes or failures should also be a factor if the algorithm is right. The time gap between the previous scheduled study date and the next scheduled study date is the indicator of all these factors combined, or the true measure of how the algorithm has weighed these factors.

  11. Support Staff 7 Posted by drewmccormack on 21 Jun, 2017 03:17 PM

    drewmccormack's Avatar

    > Also, I much prefer the idea of using the time span between previous study and next scheduled study date
    >
    That only works for spaced repetition. For other schedules, the time interval doesn’t make much sense. The only thing that is the same across all schedules is the retention (mastery).

    >
    > because might be used as an alert that a sync error has occurred and botched up a bunch of cards
    >
    This doesn’t seem like a very good use case. The point of showing the progress would not be as a sync check, it would be to see what your progress is. In any case, you can already get a pretty good idea if sync is working from the due count.

    >
    > , given the case of the card that I had gotten wrong on my last study date yet isn't scheduled to be studied again again until years later. The mastery percentage wouldn't indicate a problem where as the gap in study time did, because I knew it was a card that I didn't know as well as that scheduled date would imply. But I had to dig to find that error (assuming it is an error rather than an algorithm flaw)
    >
    Not sure it is actually either. It is possible you studied the note twice in a short period, and graded it inconsistently, in which case there is not really a ‘correct’ way to schedule, the algorithm does the best it can.

    >
    > , and as it stands I won't know how many other cards might be affected by that same sync problem
    >
    While I agree this is annoying, it is not the reason why I would introduce the feature you are requesting. A feature just to check whether things are working is not a feature, it is a bug finder.

    >
    > unless I dig into each one. Sorting by mastery percentage would reveal nothing. Not to mention that the mastery percentage is just one factor in the gap until the next scheduled study date, as the number of times studied is also a factor, and the most recent successes or failures should also be a factor if the algorithm is right. The time gap between the previous scheduled study date and the next scheduled study date is the indicator of all these factors combined, or the true measure of how the algorithm has weighed these factors.
    >
    Again, this is only true of spaced repetition. The retention probability should be very well correlated to time interval in that case, and sorting on that should find the problems you are looking for. Note that the retention probability is calculated completely independently to scheduling.

    Drew

  12. 8 Posted by zeppo on 21 Jun, 2017 06:28 PM

    zeppo's Avatar

    You refer to a "retention score" and also a "retention probability". I'm assuming you are referring to the same thing in both cases. Are you referring to the "Ratio Correct"?

  13. Support Staff 9 Posted by drewmccormack on 21 Jun, 2017 06:48 PM

    drewmccormack's Avatar

    Yes retention score and retention probability are the same. It is estimate of the probability that you will know a particular note at a particular point in time, based on scientific models. The better you know a note, the higher that is. Over time, if you don’t study a note, it drops off exponentially for any given note.

    Ratio correct is something different. That is just the fraction of times you have graded the note correct.

    Drew

  14. 10 Posted by zeppo on 21 Jun, 2017 07:00 PM

    zeppo's Avatar

    Ok, I thought you were talking about the ratio correct.

    I'm not sure how meaningful the retention score is unless you are using long term study. But if for long term study, the retention score correlates directly to a time value determining when the card is next scheduled to be studied, then it would serve the purpose I am talking about. Given that it is apparently the result of the learning algorithm, I would think that would be the case, but perhaps I misunderstand. If it is instead just one factor that is combined with one or two others to determine scheduling, then while not ideal, at least it should be better than success ratio.

  15. Support Staff 11 Posted by drewmccormack on 21 Jun, 2017 07:16 PM

    drewmccormack's Avatar

    Yes, you are right. It is very closely related to the spaced repetition intervals. The intervals are actually based on a theory of forgetting which the retention probability is part of.

    Simply put,

    1. A human’s chance or remembering something decreases exponentially when they are not reminded of it, going in the long term to zero
    2. Each time the person is reminded (studies), the exponential curve (retention probability) starts from 1.0 and decays, but at a slower rate
    3. The more you study, the slower the ‘rate of forgetting’.

    Drew

  16. 12 Posted by zeppo on 21 Jun, 2017 07:43 PM

    zeppo's Avatar

    you write:
      "While I agree this is annoying, it is not the reason why I would introduce the feature you are requesting. A feature just to check whether things are working is not a feature, it is a bug finder.'

    But that is not the only reason I stated. The other reason is to be able to sort cards by what the app is saying the learning retention is and then be able to decide if you want to retire some cards. But it would be nice to be able to also spot cards that seem to be off as far as scheduling and reset them to whatever extent. If I can sort cards for instance that aren't scheduled to be studied again for another year, but I can scroll through and spot some that I know I'm already having trouble with at the present time, much less a year later, then it would be useful to me, while at the same time bringing to light any errors by the app that would have otherwise gone unnoticed and caused the space repetition model of learning these cards to fail.

    I've always gotten the impression that you don't actually use the app for any long term study yourself, but instead, if you use it, you use it for short term study. I think if you did you would understand why I would find some things useful and others problematic (like losing the wifi sync option for instance, and its more efficient and speedy syncing between iOS and OSX. ) But language study is a long term study that probably relies more on spaced repetition flashcards than any other area. It means *lots* of cards with a long term goal in mind. So I have cards going back to when I first bought the Mental Case in 2012 or 13. If the app, device and cloud technology keep up with the robustness needed to for me to continue to add more cards to learn, then I wouldn't even bother with the thought of the retiring cards because the algorithm would effectively take card of that for me. But the iOS app starts to struggle when my due notes count gets close to 200 for a day and then I have to baby it. Also if I were to add 40 new cards it may struggle with syncing them into the mix (even if I don't turn on scheduling for them.) The mac app on the other hand seldom struggles. I may do in 30 seconds what takes the iOS app 20 minutes to sync. So with this feature of being able to sort cards by mastery, I may retire cards since it would make it easier to do without retiring some in error that I haven't shown to know as well as I might guess.

    This is with a 2015 macbook pro and iPhone 6.

  17. drewmccormack closed this discussion on 23 Jun, 2017 05:10 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac