Thoughts

A collection of opinions and frameworks I've come to about learning, writing, and building — mostly aimed at making technical material easier to absorb and easier to apply. None of it is rigorous, but I thought that maybe by sharing some of my thoughts it could help you too.

Small pages

The pages may have differing amounts of detail, but I try to make sure that they each have some amount of information whether it's a program or math or some concept. If I think something could be potentially useful I might make it a page even though it could be a pretty small page or it may seem "basic". I remember the days when I could not import numpy as np correctly when I started programming so this site is built with that in mind.

An easy way to solve difficult problems: solve a different problem

So you want to solve difficult problems. News flash. It's not easy. This is just my personal approach and you are free to disregard it. Sometimes you have something you want to learn, like some derivation from a paper and you have spent hours and hours trying to understand. After all these hours you are starting to lose hope that you will solve this problem. Let me introduce you to seven words. Give up for now, solve something else. I designed this website to follow this philosophy. If you can't figure it out, just hit the random page button and find something you can. When you are in college not understanding is not an option because you are going to be tested probably next week on the thing you don't understand. The difference is that there is an entire apparatus around you to help. You can ask a classmate, a professor, a teaching assistant, or the internet. The thing is that these problems in your class are part of a curriculum, a well-trodden path and there are people with a vested interest in learning the same thing. If you have a problem of your own making you can try to find an answer somewhere, but the odds are that the resources may be quite a bit more scarce. Also, you may not have people you can ask. You have to be responsible for your own education, and the first step of that is realizing there may be problems you cannot solve right now. It's like going to the gym, you don't leave the gym just because you fail at one exercise, you switch exercises. That problem you left behind will still be there waiting next time you feel like trying. Some things take time to understand. The funny thing about all this is that by taking a break and letting that problem "simmer" one day you usually will understand. Poincaré wrote something about this, when he journaled about the process of solving a mathematical problem. You don't get to choose what you understand today, but you can choose today to study something you can understand. TLDR: Take a break and solve an easier problem.

Use of LLMs

Everyone is using LLMs these days and for good reason, they're useful! I developed this website with the help of LLMs so I cannot say I "wrote every line of code on this site from scratch". I will say that I have "read, understood, and verified to the best of my ability every line of code on this site". The algorithms, programming, and coding interview sections were especially influenced by LLM suggestions. The amount of searches that would be required to aggregate a similar amount of information would be ridiculous. After all, these large-language models do have a huge database to draw from.

A five page paper becomes 1000 lines of code

I always find it fascinating how condensed information can become when it is written in a paper. When you read a paper that has any amount of programming involved they often resort to just writing equations. I understand the need for brevity because you can't exactly submit 1000 lines of source code to a journal. The problem I have is that the specification is so broad that it takes a lot of work to re-create the equations in software. Basically the problem of translating the equations to code is a difficult one and not a "technicality".

If you can't build it you probably don't understand it

Any time I am presented with an equation or an idea or something quantitative I am convinced that I do not understand unless I can implement it. It's one thing to be able to repeat a few words which are the "correct answer", but it is another thing entirely to be able to implement the concept.

If you can't explain it simply, you don't understand it well enough

When writing explanations of things I think it is best to keep the lexical density of the text lower and keep the readability high. This is just my opinion and it's shaped by my experiences as a student. If you want dense writing you may need to look elsewhere. It is my personal preference to have a less dense document that is twice as long as opposed to a very dense document that is half the length. I was reading this book about compilers and it got me thinking that reading is a lot like compilation. You have to read the words and translate them into some kind of meaning. Dense writing saves paper, but at times it can cause "unnecessary strain" for the reader. A completely informal non-scientific argument and analogy that I can draw is from moving heavy objects. I once had to move some 2 x 6 boards and took two one time and then four the next time from one place to another. In that moment I realized that I had unnecessarily strained myself carrying the four boards and should have just carried two. I think that the same exact phenomenon happens when you are reading technical documents. I am not arguing that reading dense papers is not a valuable exercise, for sure it is. However, not every time you open up a book should you have to strain yourself because it is a waste of time. If you write the content at a lower level, with fewer syllables per word, shorter sentences, a few more prepositions your readers can read more and sit there contemplating the last sentence less. If you are able to write a succinct 5 page paper that takes 2 days to understand you may have been better off to write a 50 page pedantic paper. I'm not a writer, I just like explaining things and it means that I need to write things down somewhere. Maybe think of these as "lecture transcripts". One interesting point about lexical density is that it is notably lower when the spoken word is used, which leads me to yet another hypothesis: the reason that lectures are often effective is that they present topics at a lower lexical density than the textbooks which are provided. Couple this with some active participation at a desirable difficulty level (homework, note-taking) and you have the format for your average university class.

Lexical Density and Flesch-Kincaid Readability

These are two measures I like to use in order to keep my explanations simple. Along with my own opinion.

Did I understand that last sentence?

This is maybe a very obvious thing to ask, but I really recommend this strategy when you are reading something technical. Rate for yourself how much you think you understand a sentence. For example: The cat sat on the mat. 10/10. In the variational minimization of the Slater-determinantal ansatz under orthonormality constraints, the nonlinear integro-differential structure of the Hartree–Fock equations emerges from the stationarity of the energy functional with respect to orbital rotations, yielding a Fock operator whose implicit dependence on its own eigenfunctions enforces self-consistency through the idempotency of the one-particle density matrix and the exchange-induced nonlocality of the effective mean field. 0/10. Actually I do understand this sentence because I had an LLM write "a complicated sentence about Hartree-Fock" and checked that it mostly made sense. It could have some error in it, but as far as I can tell it is not wrong. The point of the sentence is that it is extremely dense and mostly unreadable. Some books have sentences like this and you need to make sure you understand sentences like these because they might take a few passes.

Incremental Programs and Concept Application Lead Time

I think that people spend too much time reading large walls of text and not enough time applying what they have learned. To avoid this pattern this requires the text to teach you just enough to apply what you learned and so on. I have learned this pattern from many tutorials which walk you through gradually building computer programs. At every stage you should be making something that works instead of reading a massive specification and then building it. You need to make small, testable programs which test your understanding and give you some product that you can ship. Otherwise you spend a week reading about stuff and don't program a line of code.

A lesson on over-optimizing from TCP/IP and general programming

Build something that works no matter how small. You can perfect your design and standard later. Sometimes you are tempted to program something "perfectly" you have a design schematic in your mind. The beautiful, elegant, solution that you just need to etch onto the screen. While you should enjoy writing an elegant solution there are certain implementation details you just don't know until you start programming. Legend goes that the original OSI model was a standards-first code-second approach for networking. TCP/IP was a code-first standards-second approach. TCP/IP won out in the end. TCP/IP had concrete implementations of the components, which meant that many small problems were solved. The organizational problem was solved second. From the start of a problem you think that you understand the problem completely, but new problems will arise once you start implementing. The problem is that if you spend too much time making a perfect plan you don't spend enough time coding to see the problems that have not arisen yet and then when you do start coding you have written a program which has a fatal flaw. I think the closest metaphor for this type of thinking is from chess. I am terrible at chess. The problem is that you need to be able to calculate next moves of your opponent. If you can't calculate you might make some mistake that will force you to lose pieces or the game. In the case of programming your opponent is the problem. You have this perfect plan, but then you make some mistake about some detail and it all falls apart. The thing is that in contrast to chess, programming will heavily penalize your time for making the "wrong" moves because you will have to refactor. If you don't spend enough time implementing small pieces of your code you haven't explored enough. Organizing programs and creating standards is difficult. The moral of the story is make one good move, build a minimum viable product. Build something that works no matter how small. Your perfect design that you imagined two weeks ago probably didn't account for some invisible problem.

A list of facts versus understanding

I was thinking about how we understand things and I believe that compiling a list of facts which are true might be a good starting point. The reason is that you need to find some way to create an inventory of the necessarily true statements. Now, just knowing these statements isn't enough. You must be able to simply explain the statements, use them and construct/derive them. To foundationally understand something you must want to understand from first principles. Like a large word can be described by smaller words, a concept can be described by its relation to other concepts.

Some sentences are more important than others

I have noticed that in my studies I have this habit of not reading full books. I own numerous books which I have read probably 20 pages out of here and there. The reason for this is that I believe some sentences are more important than others. You ever read something and it just clicks? These are the important sentences in my opinion. My "important sentences" may be different from yours, but when I read a book I now try to look for these sentences which I think are profound. These are sentences which you can base your entire understanding around. If you look at any book you've read I'm sure you could extract out the most important sentences and be able to build a strong understanding. Practically one could make a flash card deck of the most important sentences from a book.

Some information is commutative

The order of some information is not important. The sky is blue. Grass is green. Dihydrogen monoxide is water. I could switch the order of any of those statements and it doesn't change much. This is basically the concept of a flashcard, you shuffle the deck and you learn the same thing. The order doesn't change your ability to understand what is on each card. It is my opinion that many scientific/mathematical/programming concepts are similar.

Optimizing sentences

I believe that you can view education as an optimization problem. Given an individual if you can send them a list of sentences they may understand a concept. This set of words is not unique and may very from person to person.

Rewriting things, or "taking notes"

Writing to learn. Taking notes. Two ways to say the same thing. This is precisely the point. If you can rephrase a sentence it suggests you understand some kind of meaning. When you translate English to Spanish you need to understand, but it is also true when you translate English to English. I challenge you to take a book and go sentence by sentence re-wording the whole book. I can bet that you would understand more than if you just read it.

Derive Everything

If your goal is to foundationally understand something you should understand how it is derived. This is a time-consuming exercise, but you will be rewarded with a deep understanding.

Don't Derive Anything

If your goal is to make progress quickly you need to press the "I believe" button. Don't derive anything, accept the results as they are and use them as such. When you need to get somewhere in a car wondering about the design of the car won't get you to your final location any faster. Sometimes your goal is to make something work and pedantically understanding every detail is not useful.

Not Everything is Derived. Most things are invented.

Who derived the wheel? Who derived the transistor? Who derived the lightbulb? These are all essential technologies today and it took a little bit of creativity and some understanding to invent all of them. What does it mean to invent something? My definition is that it is sort of a "conceptual stochastic process". You have an idea that something could work, but you aren't sure exactly how to make it happen, so you try a few different configurations until it works.

The differing philosophies of science, technology, engineering, and mathematics

The world runs on technology. Not just the technology of your phone or computer. Technology used to mean something like "the collection of inventions that are useful" now it seems to mean "stuff that uses transistors". How do we make technology? We use science, engineering, mathematics to varying degrees. Science and mathematics can be seen as providing the basic tools, while engineering can be seen as their application.

A definition of engineering

My personal favorite definition of engineering is the application of science, technology, mathematics, and the design process to solve a real-world problems, which admit more than one solution, but have definite performance metrics. Think about a car. There are many ways to build a car, there is no "car equation" a car can optimize for speed, cost, durability, but how to do this is not fully quantitative.

A Lesson from the Theoretical Minimum

Landau had a famous test, the theoretical minimum, which was designed to test the aptitude of students. Of those that passed the test, they went on to be successful scientists. While I am sure that I could not make any sort of test as difficult as Landau's I think that the principle is quite interesting. The idea of a comprehensive exam, which could ask anything from a broad list of topics is probably an excellent way to gauge understanding. With this in mind, I began to think of knowledge as a list of questions that can be answered.

A List of Questions for Every Topic

What are the things you must know to be an expert in a field? I imagine that you could collect a list of 1000 questions for a field and if a person could answer these questions they could be considered an expert. It is not enough to memorize the answers. You must deeply understand the answers. Imagine an exam that has 1000 possible questions, but on the final exam only five are asked. If the questions are given ahead of time, even the solutions, no person could memorize it all. An easier approach is to understand the concepts. My guess is that a person who knows how to apply and understands 1000 things in a field is probably an expert.

Overfitting and Understanding

Neural networks have a phenomenon known as overfitting, where they memorize data. I believe that students also exhibit this phenomenon when learning. They know that there are certain problems on a test so they memorize those problems and then pass the exam. The secret to understanding is forgetting. If you remember the logic and the derivations to things and the concepts, you don't have to memorize anything. Learning happens after you remember the "general idea" and no longer the "specific problem". It's like learning to improvise with an instrument as opposed to playing sheet music. Only by playing notes that are not written and understanding the concepts can you truly understand. If you were provided a book of 1000 problems with solutions, you could try memorizing them, but my guess is that you would gradually forget the specific problems and remember the general concepts and logic. You sort of develop this skill with mathematical tools. In my estimation, I believe there are plenty of patterns and tricks that are not named, but if you solve enough problems you learn the patterns and can then apply them. An example from my studies was positronium, there was some kind of center of mass transformation that needed to be done. After that, I remembered that I could look for a center-of-mass transformation. When I see similar problems I see them as "center-of-mass" problems, even though they are not labeled as such.

Fluency in a language and fluency in a field

It is estimated that 90% of intelligible conversations take place using only 2000 words. The difference between a native speaker and a new learner can be the difference of 16,000 words. In order to learn something, you must learn concepts and how to apply those concepts. In short, you should have a long list of things you know, and things you can do.

The list of "things you know" versus "things you can do"

What are the things you know, and what are the things you can do? If you know a lot of things, but can't do anything, well, you're in trouble. The application of knowledge into real-world applications is the key. Write a list of things you know. Write a list of things you can do. The odds are that a future employer or you yourself will be more interested in what you can do. On top of that, what are the things that you can do which save people time and improve their quality of life. Think about the services which you use on a regular basis. I'm sure that they provide both.

The list of "things you can do" which are useful

There are certain "things that you can do" which are highly useful. PID controllers run a lot of stuff, so if you can tune a PID controller you may be entitled to compensation. Other things, like "diagonalizing a matrix" are also useful, but not in the same way that the business logic of a POS system is. You can find a place where diagonalizing a matrix is useful, but odds are those places will be more difficult to find. I recommend diversifying your portfolio to things that you can do which are immediately useful to many people.

Disrupted Fields: Knowledge which soon becomes useless or forgotten

Remember that one time you had to set up a website and needed to learn how that one javascript framework works that you don't use anymore? Me either. I don't think quantum mechanics has changed though recently. When you learn a technology which is soon-to-be-replaced your knowledge of that technology is suddenly not that valuable. If you learn things which are more "stable" that knowledge can compound.

The concept acquisition protocol: an idea from TCP and UDP

As I was studying networking I learned something new, which is that UDP just sends out a message and doesn't check to see if you got it. There is no confirmation. TCP on the other hand has failsafes to make sure that the information was transmitted. When you are reading a book, there is no failsafe to make sure you understand. When you go to an instructor and say "I don't understand" this is the failsafe. However, if a book is all you have, you might be out of luck. The reason you would go to an instructor is probably because they can "explain it better" or differently. What this means in english is: the sentence you read in the book does not make sense. I need a different sentence. This is perfectly legitimate and I enjoy having 5 books for a single topic because each explains it differently. The solution to this lack of understanding is to have 5 different explanations, 5 conceptual questions which test understanding, and 5 demos which apply the knowledge which is learned. Another key to this is that the solutions should be eventually provided. This gives feedback to the student so that they may assess their understanding. I call this the concept acquisition protocol.

The need for feedback: The use of a "solutions manual"

Once you leave school, there is no feedback. In order to get better, you need practice and get feedback. In my opinion, the only way to do this is to responsibly use a solutions manual, or someone else's prior work. The "solutions manual" could be someone else's program, or it could literally be Irodov's problems in general physics. There is no avoiding the need for feedback and if you want to be good I believe you must solve many problems, write many programs, understand them all and consolidate the work you have done into a conceptual understanding which will allow you to solve problems which have not been solved. Some of the problems that you read in your book would be inconceivable 200 years ago. I wouldn't worry if you can't solve the problems in your physics book because my guess is that if you're going to solve a problem that is useful, it probably won't be problem 45 in your physics textbook. You can use the skills you have learned from that problem to solve different problems.

Everything is a flashcard: Atomization of Information

Many times I prefer to learn stuff as a flashcard with no particular order. This is sort of like the idea of atoms. Break everything down into their smallest functioning unit. I think that the interesting part of this philosophy is that it actually allows us to think of different ways of thinking of things and see patterns. In the music world, there is this technique called the cut-up technique where the songwriter oftentimes would take a bunch of lyrics and cut them up into small pieces and re-arrange these pieces. This is a common method and I believe William S. Burroughs is one of the forefathers of the technique. Anyway, this idea of breaking things into pieces and rearranging them is an interesting way to work when most of the information you are given has a particular order. I mean no one would buy a randomly ordered book of sentences, but people do make flashcards all the time. By breaking everything into trivia or small atomized pieces you can think about each piece individually instead of in context. Context is useful, but sometimes only by taking an idea out of context can you find a new useful context.

Teaching and Writing to Learn

Everyone has a different preference for how to learn. One that I have which is kind of unique is that I would rather pretend like I am an expert in the field and synthesize as best as I can my understanding. The act of actually creating some kind of document or presentation or educational material is a mode of learning that I actually enjoy quite a bit. I can't say that there is any particular reason for this except it is my predisposition. Some people like reading books just to understand I would rather read a book to write down my own thoughts on the topic. If you want any evidence of this preference, just look around on this site. I put this content on this site mostly so I could learn and others could learn.

The Privilege of Teaching

Expertise takes feedback, we all know this. When you are teaching, you have the privilege of feedback every time you teach a concept. You are forced to phrase things different ways and really examine if you understand a concept. Few people get the privilege of constant feedback for multiple hours every day. When a student asks a question you are forced to reconsider and reexamine and recall information you ordinarily would not have to access if you weren't teaching.

How to use AI tools to learn: The Monte Carlo Idea Engine and The Branches of a Tree

AI tools are very good at getting the "average" of something. It knows "on average" how to do a lot of stuff. Ask it any algorithm problem and it probably knows it if it was out there somewhere on the internet. Let's take a step back and think about how the search for information has evolved over the last few years. In 1990, you probably were going to a library somewhere to find a book that wasn't there or look on another shelf until you found it. This is only anecdotal evidence. I did not live during those ancient times. Information was kind of difficult to find. Now, fast-forward to the invention of the internet. People never knew which websites to go to so people literally kept lists of websites. Again, information was limited. Now, continue to the invention of search engines. Information was starting to become more available. You could "look stuff up" reliably. However, you still had to look around a little bit to find the info. Now, we enter the AI/LLM age. We have machinery that can find us any idea at any time. The information is instantly accessible via a series of questions. The problem with this configuration is that sometimes it throws out nonsense. It hallucinates. All in all it is a machine that can synthesize on average a bigger idea from some small sentence. The problem is that the information is not so certain sometimes and the models are pretty confident. This is where you have to take initiative. Use the AI model to get an answer then cross-examine it. Interrogate the information. It is telling you something wrong and you have to find it. You don't have to build the house, you are now the building inspector. This is how you can teach yourself with AI, ask it a question, but don't take it at face value, test, test, test and study the output. If you ask a model, "how do I make a finite-element mesh" it will give you an answer. An average answer. This answer forms the trunk of your knowledge and you must check that the knowledge is factual and then build branches on that trunk by applying your own judgement. i.e. "I think that's wrong" "Tell me more about X" "Why is this true?". You need to be the most annoying student on earth. Every time you prompt the engine it provides you with a "Monte Carlo" idea and you are tasked with discerning whether it is inside the circle of knowledge and outside the circle of knowledge.

The high-level summary and extreme detail

If you can summarize how to do something at a high level and also how to do it in extreme detail with basically no references you probably understand.