TLDR

• “Feature, not a bug” (FNB) is a saying for something that feels ‘bad’ but is actually ‘good’.

  • An example of this is the fatigue that comes from exercise.

• “Bug, not a feature” (BNF) is the opposite: something that feels ‘good’ but is actually ‘bad’

  • An example of this is compulsive social-media use.

• High-quality, digital flashcard systems have several important examples of FNBs that students can often fail to recognise.

• Low-effectiveness apps have several examples of BNFs that students can also often fail to recognise.

• The result is that users may invest substantial time on methods or apps that objectively produce considerably less learning than their apparent progress or own subjective feelings suggest.

• Understanding the difference between FNBs and BNFs can help distinguish features that feel productive and those that are productive.

• Students who understand how to study effectively consistently make greater progress, and in one recent study were 2x more likely to receive an A grade, and 3x more likely to pass.

Introduction

Our brains are the product of more than ~500m years of vertebrate brain evolution. They support cognitive abilities that far exceed what their relatively small size might suggest. They can remain functioning despite being partly missing (see here, here and here). They can even continue doing cutting-edge physics after having a high-energy proton beam, travelling at nearly the speed of light, penetrate directly through them.

Unfortunately, our brains are often also poor at judging some of it’s own limitations. The same brain that will instinctively pull a finger away from a hot surface before the individual realises is also easily coerced into activities that are generally deemed fairly unhealthy (over-consuming fast-food, compulsive social media use).

Worse, the same brain that recognises and accepts the importance of studying will very often convince a student not only that the most effective means of studying are the least effective, but can also often convince them that the least effective methods are the most effective!

Why is this? Why can methods that produce less learning very often feel more productive, while methods that are productive can often feel less productive? And what can an individual do to best increase their chances of success with regards to their studying?

Feature, Not A Bug (FNB)

A feature, not a bug (FNB) describes a characteristic that initially appears undesirable, but actually exists to serve an important function. Simply, it is something beneficial that is there for a reason. Traffic lights, password requirements, exercise fatigue and so on are examples of FNBs. All of these are things that at the time feel ‘bad’, or at the very least not desirable, but nonetheless are there for a reason: traffic lights are there to make journeys smoother and safer; password requirements are there to prevent accounts from getting compromised; respiratory rate increases during exercise in order to meet the increased oxygen demands. These things are all FNBs.

As it so happens, digital flashcard systems have their own FNBs.

1. Backlogs

A common concern regarding the use of digital flashcards is that if they are left for a few days there will be a backlog of cards to work through upon return. This is seen as a bad thing - it is seen as a bug.

The problem is that this is an FNB. Not only is it not a bug, it is actually an important feature. When a user misses several study sessions, overdue material accumulates because the spaced-repetition system (specifically, the scheduling algorithm) continues to account for reviews that were scheduled in order to prevent or reduce forgetting.

In fact, one reason that a large percentage of learning-based websites, apps or study methods in general often produce poor long-term learning and retention is that this cumulative review feature is absent, or heavily reduced. This can often give rise to a not-widely-known concept in research known as the Illusion of Competence: the observation that students very often overestimate their own progress and knowledge in the absence of it having it tested.

A learner cannot reliably determine whether information remains retrievable without attempting to retrieve it themselves, unaided. How exactly is an individual to know whether they have forgotten material if they are not required or reminded to think about said material? I, fact, it can almost be thought of as a loose form of Schrödinger’s cat. Until the student does attempt to think about it, only then will they be able to tell if they have retained it.

A low cognitive-effort method or app can often allow learners to progress through a large amount of material rapidly, with minimal attempt to establish how much of said material will remain retrievable later. Users therefore can often continue encountering new material, further increasing their feeling of progress, without being required to actually demonstrate retention of prior material.

A flashcard backlog therefore is just a reflection of a fundamental property of memory: information generally becomes more difficult to retrieve when it is not successfully revisited over time. This is one reason the principles underlying effective flashcard use (recall and spacing) are so well supported by decades of research. Put simply: use it, or lose it.

(Note: Shaeda has an in-built Recovery Mode feature designed exactly for this purpose. In the event that some days are missed, users will be prompted to consider using Recovery Mode in order to assist getting back on track in the most optimal way.)

[Important: A very common scenario for users who are new to flashcards is to cover too much new material, or create too many new cards, right away. On Shaeda, we have tried to prevent this by setting up learning plans from the beginning and calculating a overly-conservative Daily New Card limit based on the context of what you are learning. It is strongly recommended to adhere to this, starting slow and gradually building up. Introducing excessive amounts of new material at once can create review demands that quickly become unsustainable. For example, consider drinking from a bottle of water: tipping the bottle upside down and squeezing will certainly rapidly increase the rate at which water will leave the bottle, but the majority will simply be wasted as it overflows from one’s mouth. Introducing new material behaves similarly: beyond a certain point, increasing the rate of exposure beyond what’s manageable creates more future review than the learner is able to realistically maintain.]

2. Effort and Fatigue

Another common concern regarding flashcards is that they can be mentally fatiguing. Indeed this is true: being required to recall an answer unaided is considerably harder than simply rereading notes or recognising the correct answer from a multiple-choice list. Both of these generally require substantially less effort, which is why they often provide substantially less retention and learning.

Luckily, this greater cognitive load of flashcards is a feature, not a bug.

The greater effort required by recall practice is one of the reasons it is so effective. In other words, if the greater effort was not there, it would not be nearly as effective. Retrieval does not simply test whether something has been learned. Of course it does do that, which is an additional benefit, but the actual act of attempting and successfully retrieving material itself is a strong form of learning. In research this has been a very robust finding for decades known as the Testing Effect. Naturally, students see ‘tests’ as something that is done at the end of a learning period in order to test for knowledge. This can often make it confusing and difficult to understand that the very act of being ‘tested’ not only is an effective means of learning, it is actually one of the most effective forms of learning. Flashcards do both, and at the optimal time.

It ties in to something that researchers call desirable difficulties: conditions that can make learning feel slower or harder during practice, while improving learning later. In other words, the fact that a study method feels more difficult does not necessarily mean it is working less effectively. Very often, the opposite is true.

So yes, flashcards can be tiring. Effective retrieval is cognitively demanding. But provided the difficulty is appropriate*, that additional effort is not necessarily a bug in the method, it is instead a feature.

[*Important: Fatigue can also be unproductive, depending on the quality of the cards. If the cards are too difficult, too long, poorly worded, or not relevant, they will not be effective. This is the risk to creating one’s own cards: if the student does not know general principles behind effective flashcards, their cards can easily become low-quality, fatigue will become excessive and their studying will become ineffective. Flashcards on Shaeda are designed around these principles automatically: atomicity, concise wording, direct relevance, broad coverage and level-specific.]

Bug, Not A Feature (BNF)

A bug, not a feature describes something that provides a subjective benefit while producing an undesirable underlying outcome. Simply, it is something that feels ‘good’, but is actually ‘bad’. Eating fast-food, staying up late to watch films and compulsive use of social media are familiar examples of these. All of these things are things that, at the time, feel ‘good’ despite being ‘bad’ over the long-term.

These things are bugs, not features.

As it happens, many study apps or websites make product design choices that function as BNFs. These BNFs generally fall into two categories: A) design choices that reduce the cognitive demands of the learning task, and B) those that redirect attention or motivation away from learning itself. Some common examples are below:

• Recognition-heavy multiple-choice questions

• Cloze-deletion-style questions with answer options provided

• Highly salient and interactive visual design and animations

• Leaderboards prioritising activty

• Pseudo-questions

Naturally, the resulting high success rate often creates a strong, subjective impression of rapid learning. The student, however, is likely to only realise they were not learning later on. This is again related to the Illusion of Competence.

What To Do?

The subjective feeling of progress is a poor measure of learning. When evaluating a study method, website or application, learners should instead try considering what the system is actually requiring them to do.

For instance, students can consider asking themselves questions such as:

• Am I being required to repeatedly retrieve information unaided?

• Am I repeatedly revisiting earlier material, ideally at an algorithmically-determined date and in a systematic fashion?

• Could I still produce and explain the answer if the interface, hints and answer options were all removed, as would be the case in real life?

• What is ‘progress’ actually measuring?

• Is the app or website providing detailed statistics for me to track my learning?

• Am I progressing too quickly without any degree of effort required?

A simple rule may simply be: remain sceptical of any learning system in which substantial progress can seemingly be made while requiring very little effort or thought about the material itself.

Summary

How productive studying feels and how much learning it actually produces are not always aligned. In fact, they are often often at opposite ends of the spectrum.

Some characteristics of effective learning (effortful retrieval, repeated review, forgetting, interleaving, and the accumulation of overdue material) can, and often will, feel undesirable despite serving an important function. These are features, not bugs. Conversely, features that make learning feel fast, easy and rewarding can, and again often will, inflate the appearance of progress without producing equivalent gains in knowledge. These are bugs, not features.

When evaluating a study method, the single most important question is “Am I actually learning, or does it just feel like I am?”

Subscribe now

Share

shaeda - flashcards, just better

• As the release of Shaeda nears, I wanted to write a quick, laid-back (opinion-based) post addressing some arguments against flashcards that I’ve seen over the years.

• I think arguments against flashcards are either not well thought out, or simply misguided due to a potential lack of awareness

• Shaeda update coming this week. It is 90% ready for pre-beta. If you’d like to be a part of stage 2 testing, please get in touch below:

“They promote memorisation, not true learning”

I am starting with this one. Not only because it is the most common but also due to the fact that there multiple problems with the statement - as I see it.

I’ll try to bullet-point some of my thoughts below:

1a: How exactly can new learning take place if prior learning has been forgotten? This risks a House of Cards-type scenario with one’s knowledge. It would be illogical to teach a child about multiplication and division prior to them having learnt addition and subtraction. The same applies just the same as one progresses.

1b: If a student demonstrating that they can recall xyz is not valid learning, then what exactly is and, importantly, how could they display it?

• “I know the answer. Trust me.”

1c: What exactly is ‘true’ learning?

• Typically this ‘just’ means “a deep understanding”.

• That is, proponents of this belief seem to suggest that learning x via flashcards is largely futile if you do not learn everything about x.

• 1. Why exactly flashcards cannot be used for deep understanding is not clear to me. If anything, the ability to repeatedly answer questions (many of which new) on a subject is the very definition of possessing a deep understanding. Simply rereading the textbook and saying “Yeah, looks about right” is not the same for reasons covered in previous posts (passive =/= active). Perhaps being able to implement something in the real world? Maybe, it depends on the subject. But even then, how will any given thing be implemented if the prerequisites have been forgotten? Regardless, the act of trying to implement something is still an indirect form of recall.

• But let’s just go with the statement as is purely for arguments sake:

• Deep understanding, as suggested by the name itself, is inherently incredibly difficult: it takes tens of thousands of hours of active work and practice.

• 2. To suggest that flashcards will not help with studying simply because someone is of the belief that they might not directly lead immediately to a deep understanding seems akin to suggesting that a young child learning their 7 times tables should stop as this will not make them the next Von Neumann or Ramanujan.

• What exactly does lead to a deep understanding that does not involve thousands of hours of further active study and practice? Nothing, of course.

• Therefore this is not a valid argument against flashcard study, it is a flawed argument against all forms of non-PhD-level study.

1d: What if the student does not require or even seek a deep understanding? (Aimed mostly at casual language learners and/or for students studying ‘side’ subjects)

• The belief that all learning requires extensive depth is misguided.

• Would it be nice to have deep expertise in everything one begins to learn? That’s debatable but sure, let’s go with it. (Un)fortunately our brains did not evolve for this - and luckily they didn’t have to.

• Take for example our language: English. We all speak English. We all read English. We all use English. But let’s say a (very bored, likely retired) linguistics professor started quizzing us on it one night. We would probably struggle to explain English1. Does this mean we don’t know the language? Such a suggestion of course would never be made. For some students, they are actually simply aiming for a certain grade in a particular subject or class in a years time to then be done with it to never touch again (probably Cellular Biology). Ironically, it is actually these very scenarios where flashcards possess even more utility: a decrease in interest and allocated study time necessitates an increase in the efficiency and effectiveness of the study.

• If someone is already naturally engaged with a subject 24/7, the utility of flashcards actually does decrease as the student is constantly cycling through the material in their brain performing continuous mini-recalls. This is why, for example, a native speaker would get virtually zero value from flashcards to learn their own language, but immense value if they were to study a new language or subject - and especially if early on in their journey. As the learner reaches higher levels (say, B1 and above), they’ll probably start to feel more comfortable engaging with both natives and native material more.

(To reiterate, it is not clear to me why exactly flashcards cannot be used for ‘deep understanding’ - I strongly believe that not only can they be, they’re arguably one of the most efficient means of doing so, if not the most efficient)

“They’re boring”

• 1. This is likely just a function the subject being studied. That’s fine, but it’s important to understand the difference.

  • If it is the subject, then as previously outlined, the student should still be using flashcards for that particular subject even more. We can simplify their choices down into the following 3 options:

    • Study effectively and efficiently, but for less time (flashcards)

    • Study ineffectively and inefficiently, but for more time to make up (reread, highlight notes)

    • Refuse to study (scroll TikTok)

• 2. If it isn’t the subject, and the student really does feel it is just the flashcard that are boring them, it’s likely that they’ve been drawn to certain (unnamed) heavily-gamified apps that have lulled them into a false sense of learning via constant easy dopamine hits.

• Indeed, such apps are incredibly entertaining and addictive. Of course we’ve touched on the risks here in previous posts about how this is misleading as there is minimal learning actually occurring: the student is just 99% gaming.

• Further, once this 99%-gaming ‘study’ session is over, they’ll typically then go and engage in 100% gaming (or TikTok’ing). Thus the total time spent learning or studying is actually substantially less despite spending considerably longer on the gamified apps.

• Given children are the main audience of these apps, they will simply not yet have developed the meta-cognitive awareness to recognise that they aren’t actually learning anything.

• Worse still, the parents see that their child is ‘studying’ on said apps, likely collecting substantial points. They’re therefore happy to continue paying for the membership for their child’s ‘education’. And they’re then happy for their child to go and relax/game once their ‘study session’ is over. The problem becomes apparent.

“They’re too difficult”

Well, being asked questions repeatedly certainly is difficult. But:

• This is a feature, not a bug. The fatigue experienced during flashcards is a useful proxy to indicate that you’re brain is beginning the slow process of learning. To avoid this cognitive challenge is no different than going to the gym and avoiding the physical challenge. The fatigue is a large part of what drives the adaptation.

• Too difficult is a function of the specific card itself, and not of flashcards as a whole. Flashcards are very commonly formatted poorly (too long, not atomic, irrelevant, too advanced etc) which will needlessly lead to poor(er) retention.

• If you’re brand new to learning a new language, and, say, the 4th card of the deck you’ve downloaded online prompts you to translate “He walked to the shop yesterday”, of course this will be too difficult at this early stage. First you need to learn and remember how to say all of “he”, then “walked”, so on and so forth. With Shaeda this won’t happen. You customise exactly what you want to learn, and if a card appears that you don’t want (for whatever reason), you just skip.

Learning something new without undergoing discomfort is just not possible. Are some people naturally good at certain things? Sure, perhaps2. But that does not mean all things come naturally to them - even within their domain.

A young math student spends hours on long division

A senior student spends hours factoring polynomials.

A PhD student spends hours solving equations that are as long as their arm.

But the degree of effort and difficulty involved is relative and consistent throughout.

“Physical cards are better as you need to write them”

• Outside of very specific cases, it is illogical to opt for handwritten cards in this day and age.

• Advances in algorithmic scheduling far (far) outweigh the potentially slightly increased retention of card-specific hand-written cards.

• Another thing to bear in mind with “handwritten cards leads to better retention” is that this comes with much, much slower creation. That is, I don’t know if the slightly increased retention is a function of the act of hand-writing, or just a function of taking longer to write it.

• Further, if a handwritten physical deck comes with, say, 10% better retention but with only 10% of the total cards you’d like to learn, the tradeoff is clearly not worth it: the student has missed the forest for the trees. Remember that New Cards should be gradually added over time whilst maintaining Review of Due Cards. Trying to balance this, keep track of each day’s Reviews and make time for New Cards becomes unfeasible with even the most basic of scheduling intervals. This means that even if the student does make time (and keeps topping up their paper and pen reserves), they’ll quickly actually lose that time and more with the inefficient intervals.

• With Shaeda, you have FSRS built-in to determine the best time to review the card, as well as additional unique, one-click features such as Transliterate/Breakdown/Look Up/Deep Dive/Ask/Chirp3 TTS/Audio speed/Toggle Text visibility/Autoloop etc. None of these currently exist within flashcard apps that I’m aware of. They do loosely exist as individual add-ons, but this is again even more friction.

“They don’t work for certain subjects”

• Every time I’ve encountered this, the actual subjects differ. This to me suggests that the issue isn’t the tool (flashcards), but rather the implementation (how the cards are made and used) and/or a misunderstanding of the subject’s learning requirements.

• However, subjects that do sometimes pop up are Maths and CS.

• But I think this is just a misunderstanding of what flashcards are: questions.

• To suggest, then, that Maths and CS are not subjects that can be studied by answering questions would of course be quite strange.

• On the Maths side, you just choose an appropriate level and prompt it accordingly. Kyle Kabasares actually has some interesting videos on this where he tests the ability of new models’ to answer incredibly complex (probably bleeding-edge) PhD-level Physics questions. Would I recommend LLMs for this level of learning? Probably not. But most resources online are probably not going to provide sufficiently accurate material at this level.

• And on the CS side, for some examples of what it can look like when CS students either do not think foundational knowledge is important, or simply no longer possess the knowledge due to forgetting, consider watching this video (which is just one of hundreds on this channel - here’s another). Importantly, these students are not outliers. Does this mean that students who do use flashcards know everything? No, of course not3. In fact, quite the opposite: they’ll actually be hyper-aware that they most definitely do not know everything and will then be able to take the necessary steps to correct that. Students who study only via passive means will only make this realisation when it’s slightly too late. Again, flashcards are just questions.

• To any 3rd/4th year college/university Math/CS students here, consider if you can answer the following:

“Reviews pile up too fast, causing stress when thinking about each day’s reviews”

• This is a function of how, not what.

• If the flashcards are poorly formatted, retention will be low, frustration will be high and Daily Due Cards will climb excessively high.

• If the flashcards are formatted correctly, but the level is too high or they’re not relevant, retention will be low and Daily Due will climb.

• If the flashcards are formatted correctly, relevant, and the level is correct, but the amount of New Cards added each day is too high, the Daily Due will climb.

Remember that the (recommended) daily nature of flashcards is merely a reflection of our brains - both on the habit-forming side but also on the minimise forgetting side4 . Cards appear daily because we forget things daily.

Flashcards are daily, highly-relevant, well-formatted questions that leverage the proven two most effective learning techniques with modern algorithms determining when the optimal time for review is to extract maximum retention.

It’s (cognitive) science.

That’s everything - thank you for reading.

Please consider leaving a like or sharing. 50% of users who have signed up for Shaeda updates have all posts go to their spam - this isn’t ideal. Sharing or liking, Substack say, will help emails be directed to inboxes instead.

Share

Subscribe now

PS: Shaeda update is coming in the next few days. I have added some more screenshots to the landing page under ‘Latest’ for a look at these. As always, if you would like to be a pre-beta tester, simply get in touch.

1

A quiz on the English language. Do we need to know these in order to speak English?

Answers