Search for a command to run...
I am curious kid on the web, trying to learn more about life and other things. I enjoy building things and my medium of expression is mostly code. I have been dabbling with OpenEdx recently. I love to talk about distributed system, I have taken up learning Rust recently and been writing Python, Golang and JavaScript for a while.
No comments yet. Be the first to comment.
Reverse-engineering a ā¹2,000 Wi-Fi toy drone to understand its firmware, protocols, and limits.
This year was full of surprises, mostly pleasant ones; the year was filled with friendships and travels. Towards the end of last year, Anirudha and I started working on Rust and tried picking it up. He started a project called Feluda, and I started c...
What to Remember from PyCon India 2024
Get More Done: Discover the Magic of Tiling Window Managers
Understanding SSL Certificates: Enhancing Security and Trust Online
Recently I have been dwelling with a lot of NLP problems and jargons. The more I read about it the more I find it intriguing and beautiful of how we humans try to transfer this knowledge of a language to machines.
How much ever we try because of our laid back nature we try to use already existing knowledge or existing materials to be used to make machines understand a given language.
But machines as we know it can only understand digits or lets be more precise binary(0s and 1s). When I first laid my hands on NLP this was my first question, how does a machine understand that something is a word or sentence or a character.
I am still a learner in this field(and life 
) but what I could understand information that we are going to use has to be converted into binary or some kind of a numerical representation for a machine to understand.
There are various ways to āencodeā this information into numerical form and that is what is called word embeddings.
Word embedding is the collective name for a set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from the vocabulary are mapped to vectors of real numbers. Conceptually it involves a mathematical embedding from a space with many dimensions per word to a continuous vector space with a much lower dimension.
In short word embedding is a way to convert a textual information into numerical form so that it can help us analyse it.
Analysis like similarity between words or sentences, understand the context in which a phrase or word is being spoken etc.
Lets try to convert a given sentence into a numerical form:
A quick brown fox jumps over the lazy dog
How do we convert the above sentence into a numerical form such that our machine or even we can perform operations on it. And its hard to figure out the mathematics of language but we can always try.
So lets try, what we can do is, get all unique words and sort the words in the sentences and then makes a list of them. But then how do we get a numerical representation for it. Itās time for us to visit our long lost friend ā Matrix.
Letās get the words in proper order i.e unique and sorted
Now we will try to convert these words into numerical form using some matrix concepts(mostly representation) so that we can make a word look different from another word.
If you see there are totally 10 words and so we took 10 blocks to represent it. In a more mathematical term each representation is called a vector and the dimension of this vector is 1 x 10. So each word in this universe can be represented by a vector of that dimension and we can now carry operations on it to get our desired result.
Few prominent operations are how similar are two vectors or how different are two vectors. We can dive into that later.
Now the method that we just followed is a very brute force way of doing this and is officially called as **One**-**Hot Encoding** or Count Vectorizing.
Now the way we encoded above words can be really useless because itās just a representation and it doesnāt have any other idea so we donāt know how two words are related or are they morphologically similar etc.
The prime reason we want to have encoding is to find similar words, gauge the context of the topics etc.
There are various other techniques which actually produce intelligent embeddings that has an idea about what is going on.
As Hunter puts it
When constructing a word embedding space, typically the goal is to capture some sort of relationship in that space, be it meaning, morphology, context, or some other kind of relationship
and a lot of other embeddings like Elmo, USE etc. does a good job at that.
As we go ahead and explore more embeddings you will see it goes on becoming more complex. There are layers of training models introduced etc.
We even have sentence embeddings which are way different from just word embeddings.
This was just a tip of the iceberg or may be not even that but I thought it will be helpful for someone who is starting their exploration because it took time for me to get around this concept. Thanks a lot for reading.
Happy Hacking!
References:
http://hunterheidenreich.com/blog/intro-to-word-embeddings/
https://towardsdatascience.com/document-embedding-techniques-fed3e7a6a25d?gi=7c5fcb5695df
