Highlights

Techonology

• The spelled-out intro to neural networks and backpropagation: building micrograd. Big take aways:
• the chain of grad calculation along the reversed order of the graph.
• for the grad update, every node updates its grad.
• The granuity of functions and backward functions can be adjusted based on the requirements. For example, using tanh() directly or take part into the combination of exp(), add, power and multiply.

• Let’s Build the GPT tokenizer. Big take aways:
• BPE algorithm to create tokens automatically
• Examples of why LLM doesn’t work related with tokenizer, in particular the examples of inconsistency between tokenizer corpus and training corpus which introduces “dead” tokens.

Life

• Danced chkchkboom with my dancing buddies in the Odeonplatz.

• Watched “the dark” TV show. Completed the first season and the first half of the second season. Then, I went for video commentary, which makes thing much clear! Several impressive and interesting takeaways
• “Grandfather paradox”: if a guy travels back to when his grandfather was still young and hadn’t had his father yet and kills his grandfather. Will it be possible?
• If it is, then his father will not exist, then the guy will not exist, then the grandfather will not be killed.
• If it is not, why?
• “Casual loop”: information or an object looping through time. If the future you give the current you a box. Where does this box come from?
• Since the future you give you this box, then you can have this box, then the future you can have this box. Then the future you can give this box to you. That means this box comes from the future you. But the box of the future you comes from the current you.