Distributed representation, Hyperbolic Space, Gaussian/Graph Embedding 详细介绍

往期文章链接目录

Overview of various word representation and Embedding methods

Local Representation v.s. Distributed Representation

• One-hot encoding is local representation and is good for local generalization; distributed representation is good for global generalization.

Comparison between local generalization and global generalization:

Here is an example for better understanding this pair of concepts. Suppose now you have a bunch of ingredients and you’re able to cook 100 different meals with these ingredients.

• Local Representation way of cooking:

  • You cook every meal directly from these ingredients.
  • Suppose you cook several meals a lot, then you would be good at cooking these meals. If you want to cook something new but similar to these meals (concept of “local”), you would probably also cook them well. Here we say you have a good local generalization.

• Global Representation way of cooking:

  • You first cook some semi-finished products. from these ingredients and then cook every meal from these semi-finished products.
  • For every meal, you use all of semi-finished products. That means if you want your meal to be delicious, you have to improve the quality of these semi-finished products. Therefore, if one meal become more delicious, then all semi-finished products become better in general, which in turn make other meal more delicious (concept of “parameter sharing”). Here we say you have a good global generalization.

What are advantages of Distributed Representation over Local Representation?

• Representational power: distributed representation forms a more efficient code than localist representations. A localist representation using $n$ neurons can represent just $n$ different entities. A distributed representation using $n$ binary neurons can represent up to $2^{}$