Attention Mechanism - Part 2 : Hebrew Version
?? ??? ??? ??? ??????, ????? ??? ??Michael(Mike) Erlihson, PhD????? ???? ?????? ????? ??? ????? ??????? ??????. ????? ???? ?? ????? ?????? ????? ?? ?????? ?? ????? ???, ????? ?? ?????? ?????? ???.
??? ????, ???? ????? ?????:
???? ?????? ????? ????? ??? ?????? ?? ?????? ????? ??? ??????? ??????, ??? ????? ????? ???? ??? ?????.
???? ?????? ????? ??? ????? ??????? ?????? ?
????? ?????? ????? ????? ??????? ????? ??? ???? ?????? ?? ????? ??? ????? ????? ??????????? ?????-????? (encoder-decoder architecture), ??????? ????? 1, ????? ????? ??????? ???????. ????? ?????? ?????? ?? ?????? ????? ?????? ??, ???? ????? ????? "?????" (alignment), ?????, ????? ?? ?????, ?? ??????? ??? ????/????? ???? ????? ???? ????? ???? ????. ?????? ?????, ??? ????? ?? "????? ????" ??? ?????? ?? ????? ???? ???? ???? ????? ???? ?????.
????? ??????????? ?????-?????. ?????? ???? ???? (?? ??? ?? ????) ???? ?' ????? ??????, ?????? ?? ????? ????? ?????? ????? ???? (latent representation). ?? ??? ????, ?????? ???? ???? ?????? ?????? ?? ????? ???????? ????? ????? ?? ????? ???? ?'. ???? ?????? ????? ???, ?????, ????? ????? ?????? ????? ????. ?????, ?? ?????? ??? ?????? ????? ?? ??? ?????? GRU ???????? (???? ???? ?????? ?? ?-RNN ?? LSTM), ???? ?????? ????? ??????? GRU ??-???????? (bidirectional GRU). ?????????? GRU ?? ??????? ?????? ???? ??????? ?? ?????? GRU ???? ?????? ?????? ???? ?????? ???? ??????? ?????, ??????? ???? ?? ?????? ?-GRU ???? ?????? ???? ????? ?????? (??? ???? 2). ????? ?????? ??????????? ??-??????? ????? ??? ???? ????????? ???? ???? ???? ?? ????, ????? ?? ??????? ??? ?????, ??? ?? ????? (???? ?????) ???? ??? (????? ??????? ????? ????? ??? ????? ???? ????).
???????????? ???? ?????-????? ????? ?????, ???? ??????? ???? ??? ??????? ???? ???? ?????? Hi ????? Yi-1 ????????? ?????? (?? ??????). ?? ??? ???? ?? ????? ??????? ??? ?????, ?????? ???? ????? ?? ????? ????? ?? ?????? ????? ?? ???? C. ???????????? ?????-????? ??????, C ???? ????? ?????? ???????? ?????? ???????? ??????? ??? ??? ??????? ??? ???????? ?????? ([hi = [→hi;←hi) ??? ??? ???? ???? ?? ?? ????. ???? 3 ???? ?? ??????????? ??????? ????? ???.?
?? ???? ?? ???? ????? ?????? ????? ??? ?????? ??? ??????? ???? ???, ???? ?? ???? ?????? ????? ?? ????? ?????? (alignment) ??? ???? ????. ??? ????? ?? ???????????? ????? ??? ?? ?????, ????? ????? ???? ?????? ?????? ?? ??????, ??? ??? ????? ???? ?? ??????? ???????? ??? ?????? ???????? ?? ??????, ??? ?? ??????. ????? ???, ????? ?????? ?????, ????? ????? ??????? ???? ??? ????? ?? ????? ??? i, ????? ???? ?????? ???????? ?????? ??? i ????. ????? ???? ???? ????? ??? i ???? ????? ?????? ?? ?? ?????? ???????? ?? ?????? ,???? ??????? ?????? ?? ?????? ??? ?? ?????? ???? ?????? ??? i.
?????? ????? ???: ?????? ??????
????? ?????? ?????? ????? ???? ????? ?????? (alignment score), ?????? ?? ???? ??? ???? ?????? i-1 ?????? ???? ??? ????? j ????? ??????. ?? ??? ????? ?????, ????? ?? ?????? ???????? ?? ?????? ?-s (??? ???? ???? ?????) ??? ?????? ???????? ?????? ????? ?-h. ????????? i, j ??????? ?? ????? ??????? ?? ?????? ???? ????? ?-i, j. ??? ???? ????? ??? i ????? ????? e_ij (????? ????? ?? ???????) ????? ???:
????? ??????, ??? ???????? ?????? ??????? ???? ??????, ?????? ????? ?? ???????? ????? ???. ??? ?????? ????? ?? ????? ???? ???? ???? ?????? ?? ?????? ?????? ???????? ?? ??????. ?????? ???????? ???? ???????? ?????? ?????????? tanh ?????? ????? ????? nx1, ???? ????? ??? ??? ??? ?????? ???? ???? ????? ????, ?????? ?????? va ????? 1xn ????? ????? ?????. ???? ?? ?? ??????? W ??????? va ???? ??????? ?????? (???????) ?? ?????.
????? ???? ?????? ????? ???? ?????? ????? ??? (attention weight). ??? ??????? ?????? ????, ???? ??? ????? ??????? ????? ???, ??????? ??? ????? ?????? ?? ???? ???? ?? ??? ?? ???? ????, ??????? ???? ????? ??????? ????? ??????. ??? ???? ?????? ????? ?????? ????? ???: ??? ?????? ?? ???? ?????? ?????? ?? ?????? (???? ????? ?????? ??????? ?? ??????), ??????? ????? ??? ???????? ?- softmax ???? ??? ??? ????? ?????? ???????. ????? ?? ????? ???? ?? ????? ?????? ?????? ???? ?? ????, ?????? ????????? ?-softmax ????? ?? ????? ??????, ????? ???? ????, ????? ???????? ?????? ??????? ??????? ??? ?????? ???????? ?? ??????. ?????? ?????, ????? ????? i ???? ??? ??? ??????? ??? ??????? ????.
??? Tx ??? ???? ??????? ?????? ????? ???? ??????? ?? ???? ???? ????? ?????? ?? ????? ???.
???? ?????? ????? ???? ??? ????? ????? ???? ????? C. ?????? 3 ????? ???? ?????? ???? ????? ?????? ?? ?????? ????? ??? ??????? ???? ????? i ???????. ????? ?? ?????? ???? ?? ??????. ???? ?? ?????? ???? ???? ?????? ????? ??? Si-1 ?????? ????? ?????? ?????? Y ??? .?? ????? ????? 1, ???? ????? ?? ????? ??? ???????, ?? ???? ??? ????? ?????? (?????) ????? ????? ????? ????, ???? ???? ?????? ??????? ?????? ????? ???? ??????? ??????? ????? Ci (????? 3). ????? ???, ?????? ???? ???? ???? ????? (????? ???????? ????? ???) ???? ????? ?????, ????? ????? ???? ????? ????? ?????? ?? ????? ??????? ?? ??????.?
??? ??????, ????? ?????? Ci ?????? ?????? ????? ??????? ???? ??? ??? ??????? ???? ??????. ??????? ????, ????? ?? ??? ????? ?? ??????, ????? ???? ?? ????? ?????. ???? 4 ???? ?? ??????????? ??? ???? ????? ????? ???????.
?? ??? ?????? ????? ??? ????? ?????
???? ?????? ????? ?????? ?????? ???? ????? ??? ?????? ?? ????? ???????, ?????? ????? ??? ???? ????? ???? ??????? skip connection ?????? ??????. ??? ?????? ???? ????? ??? ?????? ???????? ?? ??????? ?????? ??????? ?????? ?????? ???? ??????, ???? ??????? ????? ????? ??? "?????" ???? ??? ?????. ???? ????? ???????????? ?????-????? ??? ?????? ?? ???? ?????? ???? ??? ????? ?????? ?????? ??????. ?????? ?? ???? ???? ???? ???????? ???????? ?? ?????? ???????????: ????? ?????? ?? ????? ????? ???? ?????? ?????? ????????. ????? ?????? ????? ??? ????? "??????" (Interpretability) ???? ???? ????. ?????? ??? ???? ????? ?? ?????? ???? ???? ??? ??? ????? ?? ???????? ???????? ???? ????? ??????. ????? ?? ???????????? ???????? ??????? ????? ??, ???? ????? ???? ????? ????? ?? ???? ??????? ????? ??? ????? ???? ?????, ????? ?? ????? ?? ?????? ???? ?????? ????? ???? ????.
?? ??? ???????? ?? ??????????? ?????? ?? ???
?? ?? ?? ???????????? ?????? ?????? ??????? ????? ????? ????? ??? ?????, ??? ?? ???? ??????, ????? ????? ????? ?????? ???? ???? ????? ??????. ????? ?? ??????? ??? ?????? ??????????? ?????? ??????????? ????????????.?
?????? ??????? ???? ????? ??????? ?????? ?????? ????? ????? C, ?????? ?? ???? ?????? ????? ?? ?? ?????? ??????????. ??? ??????? O(n*m)? ?????? ?? ???????? ?????? (alignment score) ???? m ??? ???? ??????? ???? ?-n ??? ???? ??????? ????. ??? ?? ??? ????? ????? ????? (inference) ??????.
?????? ?????? ?? ???? ???? ????? ????? ?????? ??? (limited context representation). ???? ????? ?? ???? ??? ??? ????, ????, ??? ???? ????? ?? ??????? ??????? ????? ?????? ??? ?? ?????? (?????? ????) ???? ?????, ???? ?? ????? "?????" ???????, ???? ???? ??????, ??????? ????? ?????? ??????? ???? (????? ??? ??? ??? ?????? ?????).??? ?? ????? ??? ??????? ???? ?????? ???? ??? ????? ?????.
???? ???? ????? ???? ???
?????? ????? ??? ???? ?????? ?? ?? ???? ????? ?? ?????? (cross attention). ???, ??? (????? ???) ?????? ?? ???? ??????? ?? ?? ?????? ???????. ???, ???? ?????? ????? ?? ????, ????? ?? ????? (self attention) ????? ??????? ????? ??. ????? ?? ????? ????? ?? ???????? ??? ?? ????, ?? ?? ???? ??? ????? ?????? ?? ????? ???? ????. ?????, ????????????? ?????-?????, ?? ?????? ??? ?????? ?????? ???? ????? ??????, ???? ???? ???? ?? ?????? ????? ??? ?????? ??? ??? ???. ?????????? ???????????? ????? ??????? ???? ?? ??? ???????? ???.
?? ???? ????? ?? ????? ????? ?????????
??? ??????? ?? ??????????? ?????? ?????? ????? ????? ?????? ???? ?????? ???????, ??????? ?????? ?????? ?????, ?? ???????? ????? ???? ?????? ?????? ????? ????? ?? ?? ???? ???? ????? ???? ?' ?????? ?????? ???? ?' ??????? ?????? ???. ????, ???? ??? ?????? ?????? ?????, ??? ?????? ????? ?? ?????? ???????? ????? ????? ?? ??? ????? ????? ?????. ???? ????? ???? ????? ????? ??? ???????, ???? ?????? ?? ?????? ????? ??? ?????? ???? ????, ?? ??? ???? ?? ?????????? ???.
????? ???? ?? ??????? ????? ?????? ????? ??????? ??????, ????? ???? ?????????? ?????? ?? ??? ??????? ????? ??? ????? ?? ????? ????? ?????????? ?? ?????? ????? ?? ?????? ????. ???? ?????? ???? ??? ????????? ????? ???:
"Despite the stormy weather causing some delays, the couple, who were accompanied by?their close friends, managed to reach the mountaintop and enjoy the breathtaking view."
?????????? ??????? ???? ????????? ????? ?? ????? ????? ???:
"????? ???????? ???? ??? ?????? ?????, ????, ????? ???? ????? ??????, ????? ????? ????? ??? ?????? ????? ???? ??????"
????? ??? ?????????? ??????? ?? ??????? ????? ??? ??????? ????? ????? ?? ????? ??:
"????, ????? ???? ????? ??????, ????? ????? ????? ???, ????? ???????? ???? ?????? ?????, ?????? ????? ???? ??????."
????? ???? ???????? ????????? ????? ?? ???? ????? ??????, ?????? ???? ???? ????? ???? ???? ?????? ???? ??????. ????? ???, ??????????? ??? ?????? ?????? ?? ????? ?????? ????? ??? ??, ?????? ???? ???? ????? ??????.
?????? ?????? ????:
Talent Acquisition Specialist @ QPOINT | ?? ??'?? ????????????! ?? Recruiter | Technical recruiter
1 年Awesome
Algorithm Developer
1 年????? ????? ???? ?????? : ????? ????? ??? ???? ???? ????? ?? ?????? ??? ??? ?????? ?? ??????),(??? ???? ?? ??? ????) ????????. ???? ???? ?????? https://machinelearningmastery.com/the-bahdanau-attention-mechanism/