{"@attributes":{"version":"2.0"},"channel":{"title":"Hoya012's Research Blog","description":"Study Blog with paper reviews.","link":"https:\/\/hoya012.github.io\/\/","pubDate":"Mon, 07 Feb 2022 01:33:53 +0000","lastBuildDate":"Mon, 07 Feb 2022 01:33:53 +0000","generator":"Jekyll v3.9.0","item":[{"title":"Image Data Augmentation Overview","description":["<p>\uc548\ub155\ud558\uc138\uc694, \uc624\ub298\uc740 Image Recognition \ubd84\uc57c\uc5d0\uc11c \uac70\uc758 \ud544\uc218\ub85c \uc0ac\uc6a9\ub418\ub294 Data Augmentation, \ub370\uc774\ud130 \uc99d\uac15 \uae30\ubc95\ub4e4\uc744 \uc815\ub9ac\ud574\ubcfc \uc608\uc815\uc785\ub2c8\ub2e4. <a href=\"https:\/\/journalofbigdata.springeropen.com\/articles\/10.1186\/s40537-019-0197-0\" target=\"_blank\"><b> \u201cA survey on Image Data Augmentation for Deep Learning\u201d <\/b><\/a> \ub17c\ubb38\uc744 \uae30\ubc18\uc73c\ub85c \uc81c\uac00 \uacf5\ubd80\ud588\ub358 \ub0b4\uc6a9\ub4e4\uc744 \uc815\ub9ac\ud588\uc73c\uba70, \uc5ec\ub7ec \ubc29\ubc95\ub860\ub4e4\uc758 \ud575\uc2ec\ub9cc \uc9e7\uac8c \uc18c\uac1c\ub4dc\ub9b4 \uc608\uc815\uc785\ub2c8\ub2e4.<\/p>\n\n<blockquote> Data Augmentation \uae30\ubc95\uc774\ub780? <\/blockquote>\n\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/1.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>Data Augmentation\uc740 \ub370\uc774\ud130\uc758 \uc591\uc744 \ub298\ub9ac\uae30 \uc704\ud574 \uc6d0\ubcf8\uc5d0 \uac01\uc885 \ubcc0\ud658\uc744 \uc801\uc6a9\ud558\uc5ec \uac1c\uc218\ub97c \uc99d\uac15\uc2dc\ud0a4\ub294 \uae30\ubc95\uc785\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/2.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc704\uc758 \uadf8\ub9bc\uacfc \uac19\uc774 original training data\uc758 \ube44\uc5b4 \uc788\ub294 data point \ub4e4\uc744 Augmentation\uc744 \ud1b5\ud574 \ucc44\uc6b4\ub2e4\uace0 \ud45c\ud604\ud558\uae30\ub3c4 \ud569\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/3.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ubcf4\ud1b5 Training \ub2e8\uacc4\uc5d0\uc11c \ub9ce\uc774 \uc0ac\uc6a9\ub418\uc9c0\ub9cc \uc704\uc758 \uadf8\ub9bc\ucc98\ub7fc Test \ub2e8\uacc4\uc5d0\uc11c\ub3c4 \uc0ac\uc6a9\uc774 \uac00\ub2a5\ud558\uba70, \uc774\ub97c Test-Time Augmentation (TTA) \ub77c\uace0 \ubd80\ub985\ub2c8\ub2e4. \ud55c \uc7a5\uc758 Test image\ub97c \uc5ec\ub7ec \uc7a5\uc73c\ub85c \uc99d\uac15\uc2dc\ucf1c inference\ub97c \uc2dc\ud0a8 \ub4a4 \ub098\uc628 output\uc744 ensemble\ud558\ub294 \ubc29\uc2dd\uc774\uba70 Kaggle\uacfc \uac19\uc740 \ucc4c\ub9b0\uc9c0\uc5d0\uc11c \ub9ce\uc774 \uc0ac\uc6a9\uc774 \ub418\ub294 \uae30\ubc95\uc785\ub2c8\ub2e4.<\/p>\n\n<blockquote> Image Manipulation \uae30\ubc18 \ubc29\ubc95\ub860 <\/blockquote>\n<p>\uc774\uc81c \ubcf8\uaca9\uc801\uc73c\ub85c Data Augmentation \uae30\ubc95\ub4e4\uc744 \ud558\ub098\uc529 \uc0b4\ud3b4\ubcf4\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"pixel-level-transforms\">Pixel-Level Transforms<\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/4.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc6b0\uc120 Pixel \ub2e8\uc704\ub85c \ubcc0\ud658\uc744 \uc2dc\ud0a4\ub294 Pixel-Level Transform\uc740 \ub300\ud45c\uc801\uc73c\ub85c Blur, Jitter, Noise \ub4f1\uc744 \uc774\ubbf8\uc9c0\uc5d0 \uc801\uc6a9\ud558\ub294 \uae30\ubc95\uc785\ub2c8\ub2e4. Gaussian Blur, Motion Blur, Brightness Jitter, Contrast Jitter, Saturation Jitter, ISO Noise, JPEG Compression \ub4f1 \ub2e4\uc591\ud55c \uae30\ubc95\uc774 \uc0ac\uc6a9\ub429\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"spatial-level-transforms\">Spatial-Level Transforms<\/h3>\n\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/5.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub2e4\uc74c\uc73c\ub860 Image \uc790\uccb4\ub97c \ubcc0\ud654\uc2dc\ud0a4\ub294 Spatial-Level Transform\uc785\ub2c8\ub2e4. \ub300\ud45c\uc801\uc73c\ub85c Flip\uacfc Rotation\uc774 \uc788\uc73c\uba70, Image\uc758 \uc77c\ubd80 \uc601\uc5ed\uc744 \uc798\ub77c\ub0b4\ub294 Crop\ub3c4 \ub9ce\uc774 \uc0ac\uc6a9\ub429\ub2c8\ub2e4.<\/p>\n\n<p>\uc774 Augmentation\uc744 \uc0ac\uc6a9\ud560 \ub54c \uc8fc\uc758\ud574\uc57c\ud560 \uc810\uc740 Detection (Bounding Box), Segmentation (Mask) Task\uc758 \uacbd\uc6b0 Image\uc5d0 \uc801\uc6a9\ud55c Transform\uc744 GT\uc5d0\ub3c4 \ub3d9\uc77c\ud558\uac8c \uc801\uc6a9\uc744 \ud574\uc918\uc57c \ud558\uace0, Classification\uc758 \uacbd\uc6b0 \uc801\uc6a9\ud558\uc600\uc744 \ub54c Class \uac00 \ubc14\ub014 \uc218 \uc788\uc74c\uc744 \uace0\ub824\ud558\uc5ec \uc801\uc6a9\ud574\uc57c \ud569\ub2c8\ub2e4. (Ex, 6\uc744 180\ub3c4 \ud68c\uc804\ud558\uba74 9)<\/p>\n\n<h3 id=\"-patchshuffle-regularization-2017-\"><a href=\"https:\/\/arxiv.org\/abs\/1707.07103\" target=\"_blank\"><b> \u201cPatchShuffle Regularization\u201d, 2017 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/6.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>N x N non-overlapping sliding window \ub0b4\uc758 feature \uac12\ub4e4\uc744 random \ud558\uac8c shuffle\ud574\uc8fc\ub294 \uae30\ubc95\uc744 \uc81c\uc548\ud55c \ub17c\ubb38\uc774\uba70, sliding window\uc758 \ud06c\uae30\uc778 N\uc774 hyper parameter\uc785\ub2c8\ub2e4. \uc801\uc6a9\ud558\uba74 \uc131\ub2a5\uc774 \uc62c\ub77c\uac00\uae34 \ud558\uc9c0\ub9cc N \uac12\uc5d0 \ub530\ub77c \uc131\ub2a5\uc774 \ud06c\uac8c \uc88c\uc9c0\uc6b0\uc9c0\ub418\ub294 \uc810\uc774 \uc544\uc26c\uc6b4 \uc810\uc785\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-data-augmentation-by-pairing-samples-for-images-classification-2018-\"><a href=\"https:\/\/arxiv.org\/abs\/1801.02929\" target=\"_blank\"><b> \u201cData Augmentation by Pairing Samples for Images Classification\u201d, 2018 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/7.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>2\uc7a5\uc758 image A, B\ub97c training set\uc5d0\uc11c random\ud558\uac8c \ucd94\ucd9c\ud55c \ub4a4 224 \ud06c\uae30\ub85c random crop\ud55c \ub4a4 random horizontal flip\uc744 \uc801\uc6a9\ud569\ub2c8\ub2e4. \uadf8\ub807\uac8c \ud574\uc11c \uc5bb\uc740 2\uc7a5\uc758 patch\ub97c \ud3c9\uade0\uc744 \ub0b4\uc11c mixed patch\ub97c \ub9cc\ub4e4\uc5b4 \uc90d\ub2c8\ub2e4. \uc774 \ub54c Label\uc740 A\uc758 label\uc744 \uadf8\ub300\ub85c \uc0ac\uc6a9\ud569\ub2c8\ub2e4. Image\ub294 A\uc640 B\uac00 \uc11e\uc5ec \uc788\uc9c0\ub9cc Label\uc740 A\ub9cc \uc0ac\uc6a9\uc774 \ub418\ub294 \uc810\uc774 \uc57d\uac04 \uc560\ub9e4\ud55c \ubd80\ubd84\uc785\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-improved-mixed-example-data-augmentation-2018-\"><a href=\"https:\/\/arxiv.org\/abs\/1805.11272\" target=\"_blank\"><b> \u201cImproved Mixed-Example Data Augmentation\u201d, 2018 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/8.PNG\" alt=\"\" \/> \n<\/figure>\n<p>\ub450 image\ub97c mixing \ud558\ub294 \uae30\uc874 \ubc29\ubc95\ub4e4\uc744 \uac1c\uc120\uc2dc\ud0a8 \ub17c\ubb38\uc774\uba70 \ub2e8\uc21c\ud788 \ub450 image\ub97c \ud3c9\uade0\uc744 \ub0b4\ub294 \ubc29\uc2dd\uc744 \ub118\uc5b4\uc11c \uc704\uc758 \uadf8\ub9bc\uacfc \uac19\uc740 8\uc885\ub958\uc758 Mixing \ubc29\ubc95\uc744 \uc81c\uc548\ud558\uc600\uc2b5\ub2c8\ub2e4. \ud6c4\uc220\ud560 CutMix, Mosaic \uae30\ubc95\uc758 \ud615\ud0dc\ub97c \ubcf4\uc774\ub294 \ubc29\ubc95\ub4e4\ub3c4 \uc81c\uc548\ud55c \uc810\uc774 \ud2b9\uc9d5\uc785\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-mixup-beyond-empirical-risk-minimization-2018-\"><a href=\"https:\/\/arxiv.org\/abs\/1710.09412\" target=\"_blank\"><b> \u201cMixUp: Beyond Empirical Risk Minimization\u201d, 2018 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/9.PNG\" alt=\"\" \/> \n<\/figure>\n<p>\ub2e4\uc74c\uc740 \uc6cc\ub099 \uc720\uba85\ud55c \ubc29\ubc95\uc774\uc8e0. \ub450 image\uc640 Label\uc744 0~1 \uc0ac\uc774\uc758 lambda \uac12\uc744 \ud1b5\ud574 Weighted Linear Interpolation \ud574\uc8fc\ub294 \uae30\ubc95\uc785\ub2c8\ub2e4. \ubcf4\ud1b5 lambda \uac12\uc740 beta distribution\uc744 \ud1b5\ud574 \ubf51\uc544\ub0c5\ub2c8\ub2e4. \uc774 \ubc29\ubc95\uc740 \uad49\uc7a5\ud788 \ub2e8\uc21c\ud558\uc9c0\ub9cc \ubaa8\ub378\uc758 \uc77c\ubc18\ud654 \uc131\ub2a5\ub3c4 \uc88b\uc544\uc9c0\uace0 corrupt label\uc758 memorization\uc744 \ubc29\uc9c0\ud574\uc8fc\uace0, adversarial example\uc5d0 sensitive\ud574\uc9c0\ub294 \ub4f1 \ub2e4\uc591\ud55c \ud6a8\uacfc\ub97c \uc5bb\uc744 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-data-augmentation-using-random-image-cropping-and-patches-for-deep-cnns-2018-\"><a href=\"https:\/\/arxiv.org\/abs\/1811.09030\" target=\"_blank\"><b> \u201cData augmentation using random image cropping and patches for deep CNNs\u201d, 2018 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/10.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub2e4\uc74c\uc740 4\uc7a5\uc758 image\uc5d0\uc11c random crop\ud55c patch\ub4e4\uc744 \ud569\uccd0\uc11c \ud55c \uc7a5\uc73c\ub85c \ub9cc\ub4dc\ub294 random image cropping and patching (RICAP) \uae30\ubc95\uc744 \uc81c\uc548\ud55c \ub17c\ubb38\uc785\ub2c8\ub2e4. \ub610\ud55c mixup\ucc98\ub7fc label\ub3c4 4\uac1c\ub97c patch\uc758 \uba74\uc801 \ube44\uc728\uc5d0 \ub530\ub77c \uc11e\uc5b4\uc11c soft label\uc744 \ub9cc\ub4e4\uc5b4\uc11c \ud559\uc2b5\uc744 \uc2dc\ud0a4\ub294 \ubc29\ubc95\uc785\ub2c8\ub2e4.<\/p>\n\n<p>\ub2e4\ub9cc \uc774\ub807\uac8c patch\ub97c random crop\ud558\ub294 \uacbd\uc6b0 \uc704\uc758 \uadf8\ub9bc\uc758 \ud3ad\uadc4 image\ub97c \uc608\ub85c \ub4e4\uba74, \ubc30\uacbd \ubd80\ubd84\uc774 crop\ub41c \uacbd\uc6b0 patch\uc5d0\ub294 \ud3ad\uadc4\uc774 \uc544\uc608 \uc874\uc7ac\ud558\uc9c0 \uc54a\ub294\ub370 \uc0dd\uc131\ub41c image\uc5d0\ub294 \ud3ad\uadc4\uc758 label\uc774 \ubd80\uc5ec\ub420 \uc218 \uc788\uaca0\uc8e0? \uc774\ub7f0 \uc810\uc774 \uc774 \ubc29\uc2dd\uc758 \uc57d\uc810\uc774\uba70, \uc774\ub7ec\ud55c \uc810\uc744 \uace0\ub824\ud55c \ube44\uc2b7\ud55c \ubc29\ubc95\uc73c\ub860 \uc81c \ube14\ub85c\uadf8\uc5d0\uc11c \ub2e4\ub918\uc5c8\ub358 <a href=\"https:\/\/hoya012.github.io\/blog\/yolov4\/\" target=\"_blank\"><b> YOLO v4<\/b><\/a> \uc758 Mosaic Augmentation \uae30\ubc95\uc774 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/11.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>Mosaic Augmentation\ub3c4 4\uc7a5\uc758 image\ub97c \ud569\uce58\uc9c0\ub9cc random crop \ud558\ub294 \ub300\uc2e0 resize\ud558\uc5ec \ubc84\ub824\uc9c0\ub294 \uc601\uc5ed \uc5c6\uc774 \ub2e4 \uc0ac\uc6a9\ud558\uac8c \ub41c\ub2e4\ub294 \uc7a5\uc810\uc774 \uc788\uc2b5\ub2c8\ub2e4. RICAP \ubc29\uc2dd\uc740 Object Detection \uc5d0\uc11c\ub294 \uc0ac\uc6a9\ud560 \uc218 \uc5c6\uc5c8\uc9c0\ub9cc Mosaic \ubc29\uc2dd\uc740 Object Detection\uc5d0\uc11c\ub3c4 \uc0ac\uc6a9\uc774 \uac00\ub2a5\ud55c \ubc29\uc2dd\uc785\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-manifold-mixup-better-representations-by-interpolating-hidden-states-2018-\"><a href=\"https:\/\/arxiv.org\/abs\/1806.05236\" target=\"_blank\"><b> \u201cManifold Mixup: Better Representations by Interpolating Hidden States\u201d, 2018 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/12.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>Mixup\uc744 input image\uac00 \uc544\ub2cc hidden representation \ud639\uc740 feature map (Manifold) Level\uc5d0\uc11c \ud574\uc8fc\ub294 \ubc29\uc2dd\uc744 \uc81c\uc548\ud55c \ub17c\ubb38\uc785\ub2c8\ub2e4. \uc774 \ubc29\uc2dd\uc744 \ud1b5\ud574 decision boundary\ub97c smooth\ud558\uac8c \ud574\uc904 \uc218 \uc788\uace0 Mixup\uacfc \ub9c8\ucc2c\uac00\uc9c0\ub85c \ub2e4\uc591\ud55c \uc774\uc810\uc744 \ub204\ub9b4 \uc218 \uc788\ub2e4\uace0 \ud569\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-random-erasing-data-augmentation-2017-\"><a href=\"https:\/\/arxiv.org\/abs\/1708.04896\" target=\"_blank\"><b> \u201cRandom Erasing Data Augmentation\u201d, 2017 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/13.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ubc29\uae08\uae4c\uc9c0\ub294 \uc11e\ub294 \ubc29\uc2dd\ub4e4\uc774\uc5c8\ub2e4\uba74 \uc774\ubc88\uc5d0\ub294 \uc9c0\uc6b0\ub294 \ubc29\uc2dd\uc774\uba70, \uc774 \ub17c\ubb38\uc740 input image\uc758 random\ud55c \ud06c\uae30\uc758 bounding box\ub97c \ub9cc\ub4e0 \ub4a4 \uadf8 \uc548\uc744 random noise, ImageNet mean value, 0, 255 \ub4f1\uc73c\ub85c \ucc44\uc6cc\uc11c \ud559\uc2b5\uc744 \uc2dc\ud0a4\ub294 \ubc29\ubc95\uc744 \uc81c\uc548\ud558\uc600\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-improved-regularization-of-convolutional-neural-networks-with-cutout-2017-\"><a href=\"https:\/\/arxiv.org\/abs\/1708.04552\" target=\"_blank\"><b> \u201cImproved Regularization of Convolutional Neural Networks with Cutout\u201d, 2017 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/14.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc774\ubc88\uc5d4 random \ud55c bounding box\ub97c 0\uc73c\ub85c \ucc44\uc6b0\ub294 \ubc29\uc2dd\uc778 Cutout\uc785\ub2c8\ub2e4. Box\uc758 \ud06c\uae30\uc5d0 \ub530\ub77c \uc131\ub2a5\uc774 \ud06c\uac8c \ubc14\ub00c\ub294 \uc810\uc774 \ud2b9\uc9d5\uc774\uace0, \uc774 \ub17c\ubb38\uc740 <a href=\"https:\/\/hoya012.github.io\/blog\/Improved-Regularization-of-Convolutional-Neural-Networks-with-Cutout-Review\/\" target=\"_blank\"><b> \uc81c \ube14\ub85c\uadf8<\/b><\/a> \uc5d0\uc11c \uc774\ubbf8 \ub2e4\ub8ec \uc801\uc774 \uc788\uc73c\ub2c8 \uc774 \uae00\uc744 \ucc38\uace0\ud558\uc2dc\uba74 \uc88b\uc744 \uac83 \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-hide-and-seek-a-data-augmentation-technique-for-weakly-supervised-localization-and-beyond-2018-\"><a href=\"https:\/\/arxiv.org\/abs\/1811.02545\" target=\"_blank\"><b> \u201cHide-and-Seek: A Data Augmentation Technique for Weakly-Supervised Localization and Beyond\u201d, 2018 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/15.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc774 \ub17c\ubb38\uc5d0\uc120 image\ub97c grid\ub85c \ub098\ub208 \ub4a4 patch\ub97c \ub9e4 iteration \ub9c8\ub2e4 random\ud558\uac8c \uc9c0\uc6b0\uba74\uc11c \ud559\uc2b5\uc2dc\ud0a4\ub294 \ubc29\ubc95\uc744 \uc81c\uc548\ud558\uc600\uc2b5\ub2c8\ub2e4. \uc774\ub97c \ud1b5\ud574 Network\uac00 image\uc5d0 \uc788\ub294 object\uc758 \ud55c \ubd80\ubd84\uc5d0\ub9cc \uc9d1\uc911\ud558\ub294 \uac83\uc774 \uc544\ub2c8\ub77c \ub2e4\uc591\ud55c \ubd80\ubd84\uc744 \ubcf4\uba74\uc11c \uc608\uce21\ud558\uac8c \ud574\uc8fc\ub294 \ud6a8\uacfc\ub97c \uc5bb\uc744 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/16.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>Grad-CAM\uacfc \uac19\uc740 attribution \ubc29\ubc95\uc744 \uc0ac\uc6a9\ud558\uc600\uc744 \ub54c object\uc758 \ub354 \ub113\uc740 \uc601\uc5ed\uc744 \ubcf4\uba74\uc11c \uc608\uce21\ud558\ub294 \uac83\uc744 \ud655\uc778\ud560 \uc218 \uc788\uc73c\uba70, \uc774\ub97c \ud1b5\ud574 Weakly-Supervised Localization\uc5d0 \uc801\uc6a9 \uac00\ub2a5\ud568\uc744 \ubcf4\uc774\uace0 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-cutmix-regularization-strategy-to-train-strong-classifiers-with-localizable-features-2019-\"><a href=\"https:\/\/arxiv.org\/abs\/1905.04899\" target=\"_blank\"><b> \u201cCutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features\u201d, 2019 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/17.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub2e4\uc74c\uc740 \uc6b0\ub9ac \ub098\ub77c \uba4b\uc9c4 \uc5f0\uad6c\uc6d0\ubd84\ub4e4\uc758 \uba4b\uc9c4 \uc5f0\uad6c \uc131\uacfc\uc778 CutMix\uc785\ub2c8\ub2e4. MixUp\uc740 \ub450 image\ub97c \uc11e\ub294 \ubc29\uc2dd\uc774\uace0, Cutout\uc740 image\uc758 box\ub97c \uccd0\uc11c \uc9c0\uc6b0\ub294 \ubc29\uc2dd\uc774\uc5c8\ub2e4\uba74, CutMix\ub294 \ub450 \ubc29\ubc95\uc744 \ud569\uce5c \ubc29\ubc95\uc785\ub2c8\ub2e4. A image\uc5d0\uc11c box\ub97c \uccd0\uc11c \uc9c0\uc6b4 \ub2e4\uc74c \uadf8 \ube48 \uc601\uc5ed\uc744 B image\ub85c\ubd80\ud130 patch\ub97c \ucd94\ucd9c\ud558\uc5ec \uc9d1\uc5b4\ub123\uc2b5\ub2c8\ub2e4. Patch\uc758 \uba74\uc801\uc5d0 \ube44\ub840\ud558\uc5ec Label\ub3c4 \uc11e\uc5b4\uc8fc\ub294 \ubc29\uc2dd\uc785\ub2c8\ub2e4. \uc774 \ubc29\ubc95\uc744 \uc801\uc6a9\ud558\uba74 \uc131\ub2a5\uc774 \ub9ce\uc774 \uc88b\uc544\uc838\uc11c \uc800\ub3c4 \uac01\uc885 challenge\uc5d0 \ucc38\uc5ec\ud560 \ub54c \ud544\uc218\ub85c \uc0ac\uc6a9\ud558\ub294 \uae30\ubc95 \uc911\uc5d0 \ud558\ub098\uc785\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-augmix-a-simple-data-processing-method-to-improve-robustness-and-uncertainty-2019-\"><a href=\"https:\/\/arxiv.org\/abs\/1912.02781\" target=\"_blank\"><b> \u201cAugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty\u201d, 2019 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/18.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub2e4\uc74c\uc740 \ud55c \uc7a5\uc758 image\uc5d0 \uc5ec\ub7ec augmentation \uae30\ubc95\ub4e4\uc744 \uc9c1\ub82c, \ubcd1\ub82c\ub85c \uc5f0\uacb0\ud55c \ub4a4 \uc6d0\ubcf8\uacfc \ub2e4\uc2dc \uc11e\uc5b4\uc8fc\ub294 AugMix \ub77c\ub294 \ubc29\ubc95\uc785\ub2c8\ub2e4. \uc774 \ubc29\ubc95\uc740 \uc77c\ubc18 Test Accuracy\ub97c \ub192\uc774\ub824\uace0 \ub098\uc628 \ubc29\ubc95\uc740 \uc544\ub2c8\uace0, ImageNet-C, ImageNet-P\uc640 \uac19\uc740 Robustness\ub97c \uce21\uc815\ud558\uae30 \uc704\ud574 \ub098\uc628 \ub370\uc774\ud130 \uc14b\uc5d0\uc11c\uc758 \uc131\ub2a5\uc744 \ub192\uc774\uae30 \uc704\ud574 \uc81c\uc548\ub41c \ubc29\ubc95\uc785\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-smoothmix-a-simple-yet-effective-data-augmentation-to-train-robust-classifiers-2020-\"><a href=\"https:\/\/openaccess.thecvf.com\/content_CVPRW_2020\/html\/w45\/Lee_SmoothMix_A_Simple_Yet_Effective_Data_Augmentation_to_Train_Robust_CVPRW_2020_paper.html\" target=\"_blank\"><b> \u201cSmoothMix: A Simple Yet Effective Data Augmentation to Train Robust Classifiers\u201d, 2020 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/19.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc774\ubc88\uc5d0\ub3c4 \uc6b0\ub9ac \ub098\ub77c \uba4b\uc9c4 \uc5f0\uad6c\uc6d0\ubd84\ub4e4\uc774 \ucc38\uc5ec\ud558\uc2e0 \ub17c\ubb38\uc774\uba70, CutMix\ub294 patch\ub97c \uc798\ub77c \ubd99\uc774\ub294 \uacfc\uc815\uc5d0\uc11c edge \uc601\uc5ed\uc5d0\uc11c \uae09\uaca9\ud55c \ubcc0\ud654\uac00 \uc0dd\uae30\ub294 strong edge \ubb38\uc81c\uac00 \ubc1c\uc0dd\ud558\ub294\ub370, \uc774\ub97c \uc644\ud654\uc2dc\ud0a4\uae30 \uc704\ud574 \uacbd\uacc4 \uc601\uc5ed\uc744 smooth\ud558\uac8c \uc11e\uc5b4\uc8fc\ub294 SmoothMix \ubc29\uc2dd\uc744 \uc81c\uc548\ud558\uc600\uc2b5\ub2c8\ub2e4. CutMix\ubcf4\ub2e4 test accuracy\ub294 \uc57d\uac04 \ub0ae\uc9c0\ub9cc robustness\ub294 \ub354 \uc88b\uc544\uc9c0\ub294 \uacb0\uacfc\ub97c \ubcf4\uc785\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-puzzlemix-exploiting-saliency-and-local-statistics-for-optimal-mixup-2020-\"><a href=\"https:\/\/arxiv.org\/abs\/2009.06962\" target=\"_blank\"><b> \u201cPuzzleMix: Exploiting Saliency and Local Statistics for Optimal Mixup\u201d, 2020 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/20.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc774\ubc88\uc5d0\ub3c4 \uc5ed\uc2dc \uc6b0\ub9ac \ub098\ub77c \uba4b\uc9c4 \uc5f0\uad6c\uc6d0\ubd84\ub4e4\uc774 \ubc1c\ud45c\ud558\uc2e0 \ub17c\ubb38\uc774\uba70 \uac01 image\uc758 saliency information\uc744 \ubcf4\uc874\ud558\uba74\uc11c \uc11e\uc5b4\uc8fc\ub294 \ubc29\uc2dd\uc778 PuzzleMix \ubc29\ubc95\uc744 \uc81c\uc548\ud558\uc600\uc2b5\ub2c8\ub2e4. \uc774\ub97c \ud1b5\ud574 \uac01 image\uc758 local statistics\ub97c \ubcf4\uc874\ud560 \uc218 \uc788\uace0 \uae30\uc874 Mix \uacc4\uc5f4\ubcf4\ub2e4 \ub354 \ub192\uc740 \uc77c\ubc18\ud654 \uc131\ub2a5\uc744 \ubcf4\uc774\uace0, Adversarial Attack\uc5d0\ub3c4 Robust\ud574\uc9c0\ub294 \ud6a8\uacfc\ub97c \uc5bb\uc744 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-the-many-faces-of-robustness-a-critical-analysis-of-out-of-distribution-generalization-2020-\"><a href=\"https:\/\/arxiv.org\/abs\/2006.16241\" target=\"_blank\"><b> \u201cThe Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization\u201d, 2020 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/21.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uae30\uc874\uc758 Augmentation \uae30\ubc95\ub4e4\uc740 \ub300\uccb4\ub85c Input image \uc790\uccb4\ub97c \uc11e\uac70\ub098 \uc9c0\uc6b0\uac70\ub098 \uc790\ub974\uac70\ub098 \ud558\ub294 \ubc29\uc2dd \ub4f1\uc744 \ud1b5\ud574 \ubcc0\ud654\ub97c \uc8fc\ub294 \ubc29\uc2dd\uc774\uc5c8\ub2e4\uba74, \uc774 \ub17c\ubb38\uc5d0\uc11c \uc81c\uc548\ud55c DeepAugment \uae30\ubc95\uc740 \uae30 \ud559\uc2b5\ub41c Image-to-Image Network (Ex, Autoencoder, Super Resolution Network)\uc758 weight\uc640 activation\uc5d0 \ubcc0\ud654\ub97c \uc8fc\ub294 \ubc29\uc2dd\uc73c\ub85c Augmentation\uc744 \ud558\ub294 \ubc29\ubc95\uc744 \uc81c\uc548\ud558\uc600\uc2b5\ub2c8\ub2e4.\nDeepAugment\ub97c \uc0ac\uc6a9\ud558\uba74 \uae30\uc874 \uc601\uc0c1 \ucc98\ub9ac \uae30\ubc95\ub4e4\ub85c\ub294 \uc0dd\uc131\ud558\uae30 \uc5b4\ub824\uc6b4 \ub2e4\uc591\ud55c \uc720\ud615\uc758 image\ub97c \uc0dd\uc131\ud560 \uc218 \uc788\uace0, semantically consistent\ud55c image\ub97c \uc0dd\uc131\ud560 \uc218 \uc788\ub2e4\ub294 \uc7a5\uc810\uc774 \uc788\uc73c\uba70, \uc774 \ubc29\ubc95\uc744 \uc0ac\uc6a9\ud558\uba74 Robustness\uac00 \ud06c\uac8c \uc99d\uac00\ud558\ub294 \ud6a8\uacfc\ub97c \uc5bb\uc744 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<blockquote> Generative Model \uae30\ubc18 \ubc29\ubc95\ub860 <\/blockquote>\n<p>\ub2e4\uc74c\uc740 Generative Model, \ub300\uccb4\ub85c GAN \uae30\ubc18\uc758 Augmentation \uae30\ubc95\uc778\ub370 \uc81c\uac00 \uc11c\ubca0\uc774 \ub17c\ubb38\uc744 \uc77d\uc5c8\uc744 \ub550 \uc544\uc9c1 \uc774 \ubd84\uc57c\ub294 \ub354 \ub9ce\uc740 \uc88b\uc740 \uc5f0\uad6c\ub4e4\uc774 \ub098\uc62c \uac00\ub2a5\uc131\uc774 \ub192\uc544 \ubcf4\uc778\ub2e4\uace0 \ub290\uaf08\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-gan-based-synthetic-medical-image-augmentation-for-increased-cnn-performance-in-liver-lesion-classification-2018-\"><a href=\"https:\/\/arxiv.org\/abs\/1803.01229\" target=\"_blank\"><b> \u201cGAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification\u201d, 2018 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/22.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>DCGAN\uc744 \ud1b5\ud574 \uc0dd\uc131\ud55c Liver Lesion image\ub4e4\uc744 \ucd94\uac00\ub85c \ud559\uc2b5\uc5d0 \uc0ac\uc6a9\ud574\uc11c \ubd84\ub958 \uc131\ub2a5\uc744 \ub192\uc778 \ub17c\ubb38\uc785\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-data-augmentation-in-emotion-classification-using-generative-adversarial-networks-2017-\"><a href=\"https:\/\/arxiv.org\/abs\/1711.00648\" target=\"_blank\"><b> \u201cData Augmentation in Emotion Classification Using Generative Adversarial Networks\u201d, 2017 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/23.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>CycleGAN\uc744 \ud1b5\ud574 \uc0dd\uc131\ud55c \uc5bc\uad74 \uac10\uc815 \ubd84\ub958 \ub370\uc774\ud130\ub97c \ud559\uc2b5\uc5d0 \uc0ac\uc6a9\ud558\uc5ec Class imbalance\ub97c \uc644\ud654\uc2dc\ucf1c \ubd84\ub958 \uc131\ub2a5\uc744 \ub192\uc778 \ub17c\ubb38\uc785\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-singan-learning-a-generative-model-from-a-single-natural-image-2019-\"><a href=\"https:\/\/arxiv.org\/abs\/1905.01164\" target=\"_blank\"><b> \u201cSinGAN: Learning a Generative Model from a Single Natural Image\u201d, 2019 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/24.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ud55c \uc7a5\uc758 image\ub85c GAN\uc744 \ud559\uc2b5\uc2dc\ucf1c\uc11c \ube44\uc2b7\ud55c \uc218\ub9ce\uc740 \uadf8\ub7f4\uc2f8\ud55c image\ub97c \uc0dd\uc131\ud574\ub0b4\ub294 \uae30\ubc95\uc778 SinGAN\uc785\ub2c8\ub2e4. \uc774 \ub17c\ubb38\uc740 <a href=\"https:\/\/hoya012.github.io\/blog\/ICCV-2019_review_2\/\" target=\"_blank\"><b> \uc81c \ube14\ub85c\uadf8<\/b><\/a> \uc5d0\uc11c\ub3c4 \ub2e4\ub8ec \uc801\uc774 \uc788\uc5b4\uc11c \uc774 \uae00\uc744 \ucc38\uace0\ud558\uc2dc\uba74 \uc88b\uc744 \uac83 \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\uc774 SinGAN\uc744 \uc774\uc6a9\ud574 Data Augmentation\ub3c4 \uac00\ub2a5\ud558\uae34 \ud558\uc9c0\ub9cc image \ud55c \uc7a5\uc5d0 GAN \ud558\ub098\uc529 \ud559\uc2b5\uc744 \uc2dc\ucf1c\uc57c \ud558\ub294\ub370, \ud559\uc2b5 \uc2dc\uac04\uc774 \uc0dd\uac01\ubcf4\ub2e4 \uae38\uc5b4\uc11c \uac00\uc9c0\uace0 \uc788\ub294 image\uac00 \ub9ce\uc73c\uba74 Data Augmentation\ub3c4 \uad49\uc7a5\ud788 \uc624\ub798 \uac78\ub9b0\ub2e4\ub294 \ud55c\uacc4\ub3c4 \uc874\uc7ac\ud569\ub2c8\ub2e4.<\/p>\n\n<blockquote>AutoML \uae30\ubc18 \ubc29\ubc95\ub860 <\/blockquote>\n<p>\ub9c8\uc9c0\ub9c9 AutoML \uae30\ubc18 \ubc29\ubc95\ub860\uc740 \ucd5c\uc801\uc758 Data Augmentation Policy\ub97c AutoML\uc744 \ud1b5\ud574 \ucc3e\ub294 \ubc29\ubc95\uc744 \uc81c\uc548\ud558\uace0 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-autoaugment-learning-augmentation-policies-from-data-2018-\"><a href=\"https:\/\/arxiv.org\/abs\/1805.09501\" target=\"_blank\"><b> \u201cAutoAugment: Learning Augmentation Policies from Data\u201d, 2018 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/25.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>NAS \ucd08\uae30 \ub17c\ubb38\uacfc \ube44\uc2b7\ud558\uac8c RNN controller\ub97c \ud1b5\ud574 Augmentation Policy\ub97c \ubf51\uace0, Network\ub97c \ud559\uc2b5\uc2dc\ucf1c\uc11c Validation accuracy\ub97c \ubf51\uc740 \ub4a4 \uc774\ub97c \uac15\ud654 \ud559\uc2b5(PPO)\uc758 reward\ub85c \uc0ac\uc6a9\ud558\uc5ec \ud559\uc2b5\uc2dc\ud0a4\ub294 \ubc29\ubc95\uc744 \uc81c\uc548\ud558\uc600\uc2b5\ub2c8\ub2e4.<\/p>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/26.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ucd1d 16\uac00\uc9c0 Augmentation \uae30\ubc95\ub4e4\uc744 Search Space\ub85c \uc0ac\uc6a9\ud558\uc600\uace0, \ub192\uc740 \uc131\ub2a5\uc744 \ub2ec\uc131\ud560 \uc218 \uc788\uc5c8\uc9c0\ub9cc \uad49\uc7a5\ud788 \ub9ce\uc740 Computational Cost\uc640 Time\uc744 \uc18c\ubaa8\ud558\uae30\ub3c4 \ud569\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-population-based-augmentation-efficient-learning-of-augmentation-policy-schedules-2019-\"><a href=\"https:\/\/arxiv.org\/abs\/1905.05393\" target=\"_blank\"><b> \u201cPopulation Based Augmentation: Efficient Learning of Augmentation Policy Schedules\u201d, 2019 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/27.PNG\" alt=\"\" \/> \n<\/figure>\n<p>Hyper Parameter Optimization \uae30\ubc95 \uc911 \ud558\ub098\uc778 Population Based Training (PBT) \uc54c\uace0\ub9ac\uc998 \uae30\ubc18 \ubc29\uc2dd\uc774\uba70, AutoAugment \ub300\ube44 \uac70\uc758 1000\ubc30 \ube60\ub978 Search \uc2dc\uac04\uc744 \ubcf4\uc5ec\uc8fc\uba74\uc11c \ub3d9\uc2dc\uc5d0 \ube44\uc2b7\ud55c \uc815\ud655\ub3c4\ub97c \ub2ec\uc131\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc131\ub2a5\uc774 \uc88b\uc740 \ubaa8\ub378\uc758 weight\ub294 \ubcf5\uc81c\ud558\uace0(exploit), \uadf8 parameter\uc5d0 \uc57d\uac04\uc758 \ubcc0\ud615 (explore)\ub97c \uc8fc\ub294 \ubc29\uc2dd\uc785\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-fast-autoaugment-2019-\"><a href=\"https:\/\/arxiv.org\/abs\/1905.00397\" target=\"_blank\"><b> \u201cFast AutoAugment\u201d, 2019 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/28.PNG\" alt=\"\" \/> \n<\/figure>\n<p>\ub2e4\uc74c\uc740 \uc81c\uac00 \uc6b4\uc601 \uc911\uc778 SNUAI \uc2a4\ud130\ub514\uc758 \uc6b4\uc601\uc9c4 \uc120\ubc30\ub2d8\uc774\uc2dc\uc790 \uc774\uc81c\ub294 \uad50\uc218\ub2d8\uc73c\ub85c \uba4b\uc9c4 \uc5f0\uad6c\ub4e4\uc744 \uc9c4\ud589\ud558\uace0 \uacc4\uc2e0 \uc784\uc131\ube48 \uad50\uc218\ub2d8\uc758 Fast AutoAugment\uc785\ub2c8\ub2e4.<\/p>\n\n<p>Bayesian Optimization \uae30\ubc95\uc778 Tree-structured Parzen Estimator(TPE) \ubc29\ubc95\uc744 \ud1b5\ud574 Augmentation Policy\ub97c \ucd94\ucd9c\ud558\uace0, \ud559\uc2b5\uc2dc\ud0a8 \ubaa8\ub378\uc744 validation\uc744 \ud558\ub294 \uacfc\uc815\uc5d0\uc11c augmentation\uc744 \uc801\uc6a9\ud558\uba74\uc11c Search \uc2dc\uac04\uc744 \ud68d\uae30\uc801\uc73c\ub85c \uc904\uc77c \uc218 \uc788\uace0 PBA \ubcf4\ub2e4\ub3c4 \ub354 \ube60\ub974\uba74\uc11c \ube44\uc2b7\ud55c \uc815\ud655\ub3c4\ub97c \ub2ec\uc131\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-faster-autoaugment-learning-augmentation-strategies-using-backpropagation-2019-\"><a href=\"https:\/\/arxiv.org\/abs\/1911.06987\" target=\"_blank\"><b> \u201cFaster AutoAugment: Learning Augmentation Strategies using Backpropagation\u201d, 2019 <\/b><\/a><\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/29.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>RCNN\ucc98\ub7fc Fast \uac00 \uc788\uc73c\uba74 Faster \ub3c4 \uc788\uc2b5\ub2c8\ub2e4. Faster AutoAugment\ub294 \ubbf8\ubd84 \ubd88\uac00\ub2a5\ud55c image operation\ub4e4\uc744 \ubbf8\ubd84 \uac00\ub2a5\ud558\uac8c \ud574\uc8fc\ub294 gradient approximation \uae30\ubc95\uc744 \ud1b5\ud574 discrete search space\ub97c continuous search space\ub85c relaxing \uc2dc\ucf1c\uc8fc\uba74\uc11c gradient-based optimization\uc744 \ud1b5\ud574 \ub354 \ube60\ub974\uac8c search\ub97c \ud560 \uc218 \uc788\ub294 \ubc29\ubc95\uc744 \uc81c\uc548\ud569\ub2c8\ub2e4.<\/p>\n\n<p>\uc774\ub7ec\ud55c \ubc29\ubc95\uc740 gradient-based NAS\uc758 \ub300\ud45c\uc801\uc778 \ubc29\ubc95\uc778 <a href=\"https:\/\/arxiv.org\/abs\/1806.09055\" target=\"_blank\"><b> \u201cDARTS: Differentiable Architecture Search\u201d <\/b><\/a> \uc5d0\uc11c \uc601\uac10\uc744 \uc5bb\uc5c8\ub2e4\uace0 \ud569\ub2c8\ub2e4. Fast AutoAugment\ubcf4\ub2e4 \ud6e8\uc52c \ube60\ub974\uc9c0\ub9cc \uc57d\uac04 \uc815\ud655\ub3c4\ub294 \ub5a8\uc5b4\uc9c0\ub294 \uacb0\uacfc\ub97c \ubcf4\uc774\uace0 \uc788\uc2b5\ub2c8\ub2e4. ImageNet \ub370\uc774\ud130 \uc14b\uc5d0\uc11c 2.3 GPU hour\ub85c Search\uac00 \ub41c\ub2e4\ub294 \uc810\uc740 \uad49\uc7a5\ud788 \uc778\uc0c1\uae4a\uc740 \uac83 \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-randaugment-practical-automated-data-augmentation-with-a-reduced-search-space-2019-\"><a href=\"https:\/\/arxiv.org\/abs\/1909.13719\" target=\"_blank\"><b> \u201cRandAugment: Practical automated data augmentation with a reduced search space\u201d, 2019 <\/b><\/a><\/h3>\n\n<p>\uc704\uc758 \ubc29\ubc95\ub4e4\uc740 AutoML\ub85c \uafb8\uc5ed\uafb8\uc5ed \ucd5c\uc801\uc758 Augmentation Policy\ub97c \ucc3e\ub294 \ubc29\ubc95\uc774\uc5c8\ub294\ub370, \uc544\uc608 \ucc3e\ub294 \uac83\uc744 \uc0dd\ub7b5\ud558\uace0 \ub9e4 batch\ub97c \ucd94\ucd9c\ud560 \ub54c\ub9c8\ub2e4 \uc5ec\ub7ec Augmentation \uc635\uc158\ub4e4 \uc911\uc5d0\uc11c random\ud558\uac8c \ucd94\ucd9c\ud574\uc11c \uc801\uc6a9\uc744 \ud558\ub294 \uae30\ubc95\uc744 \uc81c\uc548\ud569\ub2c8\ub2e4. \uc5c4\uccad \ub2e8\uc21c\ud55c\ub370 \uc131\ub2a5\ub3c4 \uc5c4\uccad \uc88b\uc2b5\ub2c8\ub2e4. \ubc14\ub85c RandAugment \uae30\ubc95\uc785\ub2c8\ub2e4.<\/p>\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/30.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc704\uc758 \uadf8\ub9bc\uacfc \uac19\uc774 \ucf54\ub4dc \ub2e8 \uba87 \uc904\ub85c \uc27d\uac8c \uad6c\ud604\uc774 \uac00\ub2a5\ud558\uace0 \uc131\ub2a5\ub3c4 \uc5c4\uccad \uc88b\uc2b5\ub2c8\ub2e4. \uc804\uccb4 transform \uc911\uc5d0 \uba87 \uac1c\uc529 \ubf51\uc744 \uc9c0(N)\uc640 Augmentation\uc758 \uac15\ub3c4\ub97c \uc5b4\ub290 \uc815\ub3c4\ub85c \uc904\uc9c0(M)\uc774 hyper parameter\uc774\uba70, \uc5c4\uccad \ub2e8\uc21c\ud55c\ub370 \uc131\ub2a5\uc774 \ub418\uac8c \uc798 \ub098\uc640\uc11c CutMix\uc640 \ub9c8\ucc2c\uac00\uc9c0\ub85c \uc81c\uac00 \uac01\uc885 Challenge\uc5d0\uc11c \ud544\uc218\ub85c \uc0ac\uc6a9\ud558\ub294 \uae30\ubc95\uc785\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"-uniformaugment-a-search-free-probabilistic-data-augmentation-approach-2020-\"><a href=\"https:\/\/arxiv.org\/abs\/2003.14348\" target=\"_blank\"><b> \u201cUniformAugment: A Search-free Probabilistic Data Augmentation Approach\u201d, 2020 <\/b><\/a><\/h3>\n<p>\ub9c8\uc9c0\ub9c9\uc740 RandAugment\uc5d0\uc11c hyper parameter search\ub97c \ud574\uc57c \ud558\ub294 \uc810\uc5d0\uc11c \ucd9c\ubc1c\ud558\uc5ec \uc544\uc608 search \uc5c6\uc774 random\ud558\uac8c augmentation\uc744 \ud655\ub960\uc801\uc73c\ub85c \uc801\uc6a9\ud558\ub294 UniformAugment\ub77c\ub294 \uae30\ubc95\uc744 \uc81c\uc548\ud55c \ub17c\ubb38\uc785\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Augmentation\/31.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub9e4 batch\ub97c \ubf51\uc744 \ub54c RandAugment\ucc98\ub7fc N\uac1c\ub97c \uace0\uc815\ud574\uc11c \ucd94\ucd9c\ud558\ub294 \uac83\uc774 \uc544\ub2c8\ub77c, \ubaa8\ub4e0 \uc5f0\uc0b0\uc744 0 \uacfc 1 \uc0ac\uc774\uc758 \ud655\ub960 \uac12\uc744 \ud1b5\ud574 \ub123\uc744 \uc9c0 \ub9d0\uc9c0\ub97c \uc815\ud558\uace0, Magnitude\ub3c4 0 \uacfc 1 \uc0ac\uc774\uc758 \ud655\ub960 \uac12\uc744 \ud1b5\ud574 \uc815\ud558\uac8c \ub429\ub2c8\ub2e4. \uc989, N\uacfc M\uc744 Probabilistic \ud558\uac8c \ubc14\uafd4\uc8fc\uba74\uc11c \uc544\uc608 hyper parameter\uac00 \uc0ac\ub77c\uc9c0\uac8c \ub429\ub2c8\ub2e4. \uc774\ub807\uac8c \ud558\uba74 tuning\uc774 \ud544\uc694 \uc5c6\uc73c\uba70 RandAugment\uc5d0 \uc900\ud558\ub294 \uc131\ub2a5\uc744 \uc5bb\uc744 \uc218 \uc788\ub2e4\uace0 \ud569\ub2c8\ub2e4. \uc774 \ubc29\ubc95\ub3c4 \ub2e4\uc74c\uc5d0 \uae30\ud68c\uac00 \ub418\uba74 \uc0ac\uc6a9\ud574\ubcfc \uc608\uc815\uc785\ub2c8\ub2e4.<\/p>\n\n<blockquote> \uacb0\ub860 <\/blockquote>\n<p>\uc624\ub298\uc740 Image Data Augmentation \uc11c\ubca0\uc774 \ud398\uc774\ud37c\ub97c \uae30\ubc18\uc73c\ub85c \uac01\uc885 \ub17c\ubb38\ub4e4\uc744 \uc9e4\ub9c9\ud558\uac8c \uc18c\uac1c \ub4dc\ub838\uc2b5\ub2c8\ub2e4. \uac01 \ubc29\ubc95\ub4e4\uc758 \ub514\ud14c\uc77c\ud55c \uc815\ubcf4\ub4e4\uc774 \uad81\uae08\ud558\uc2dc\uba74 \uc6d0 \ub17c\ubb38\uc744 \ucc38\uace0\ud558\uc2dc\ub294 \uac83\uc744 \ucd94\ucc9c \ub4dc\ub9bd\ub2c8\ub2e4. \uae34 \uae00 \uc77d\uc5b4 \uc8fc\uc154\uc11c \uac10\uc0ac\ud569\ub2c8\ub2e4.<\/p>\n","<p>\uc548\ub155\ud558\uc138\uc694, \uc624\ub298\uc740 Image Recognition \ubd84\uc57c\uc5d0\uc11c \uac70\uc758 \ud544\uc218\ub85c \uc0ac\uc6a9\ub418\ub294 Data Augmentation, \ub370\uc774\ud130 \uc99d\uac15 \uae30\ubc95\ub4e4\uc744 \uc815\ub9ac\ud574\ubcfc \uc608\uc815\uc785\ub2c8\ub2e4. <a href=\"https:\/\/journalofbigdata.springeropen.com\/articles\/10.1186\/s40537-019-0197-0\" target=\"_blank\"><b> \u201cA survey on Image Data Augmentation for Deep Learning\u201d <\/b><\/a> \ub17c\ubb38\uc744 \uae30\ubc18\uc73c\ub85c \uc81c\uac00 \uacf5\ubd80\ud588\ub358 \ub0b4\uc6a9\ub4e4\uc744 \uc815\ub9ac\ud588\uc73c\uba70, \uc5ec\ub7ec \ubc29\ubc95\ub860\ub4e4\uc758 \ud575\uc2ec\ub9cc \uc9e7\uac8c \uc18c\uac1c\ub4dc\ub9b4 \uc608\uc815\uc785\ub2c8\ub2e4.<\/p>\n\n"],"pubDate":"Tue, 11 May 2021 00:00:00 +0000","link":"https:\/\/hoya012.github.io\/\/blog\/Image-Data-Augmentation-Overview\/","guid":"https:\/\/hoya012.github.io\/\/blog\/Image-Data-Augmentation-Overview\/"},{"title":"Fast and Accurate Model Scaling \ub9ac\ubdf0","description":["<p>\uc548\ub155\ud558\uc138\uc694, \uc624\ub298\uc740 CVPR 2021\uc5d0\uc11c \ubc1c\ud45c \uc608\uc815\uc778 Facebook Research\uc758 <a href=\" https:\/\/arxiv.org\/abs\/2103.06877v1 \" target=\"_blank\"><b> \u201cFast and Accurate Model Scaling\u201d <\/b><\/a> \ub17c\ubb38\uc744 \uc18c\uac1c\ub4dc\ub9b4 \uc608\uc815\uc785\ub2c8\ub2e4. \uc81c\ubaa9\uc5d0\uc11c \uc720\ucd94\uac00 \uac00\ub2a5\ud558 \ub4ef \ubaa8\ub378\uc758 \ud06c\uae30\ub97c \uc870\uc808\ud574\uc8fc\ub294 (Scaling) \ubc29\ubc95\uc744 \ub2e4\ub8ec \ub17c\ubb38\uc774\uba70, \ud56d\uc0c1 \ubaa8\ub378\uc758 \ud06c\uae30\ub97c \ud0a4\uc6cc\uc8fc\uba74 \uc815\ud655\ub3c4\uac00 \uc88b\uc544\uc9c0\uc9c0\ub9cc \uadf8\uc5d0 \ub530\ub77c\uc11c \ucc98\ub9ac \uc18d\ub3c4\uac00 \ub290\ub824 \uc9c0\ub294 Trade-off \uad00\uacc4\ub97c \uac00\uc9c0\ub294\ub370 \uc774\ub97c \uc798 \ud0c0\uac1c\ud558\uae30 \uc704\ud55c \ubc29\ubc95\uc744 \uc81c\uc548\ud55c \ub17c\ubb38\uc785\ub2c8\ub2e4.<\/p>\n\n<p>\uc774 \ub17c\ubb38\uacfc \uad00\ub828 \uc788\ub294 \uc5f0\uad6c\ub4e4\uc774 EfficientNet\uacfc RegNet\uc778\ub370 \ub450 \uc5f0\uad6c \ubaa8\ub450 \uc81c\uac00 \uc815\ub9ac\ud55c \uc801\uc774 \uc788\ub294\ub370\uc694, \uc774 \ub450 \uc5f0\uad6c\ub97c \uc798 \ubaa8\ub974\uc2dc\ub294 \ubd84\ub4e4\uc740 \uba3c\uc800 \uc774 \uc790\ub8cc\ub4e4\uc744 \ubcf4\uace0 \uc624\uc2dc\ub294 \uac83\uc744 \uad8c\uc7a5 \ub4dc\ub9bd\ub2c8\ub2e4.<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/hoya012.github.io\/blog\/EfficientNet-review\/\" target=\"_blank\"><b> \u201cEfficientNet: Rethinking Model Scaling for Convolutional Neural Networks \ub9ac\ubdf0\u201d <\/b><\/a><\/li>\n  <li><a href=\"https:\/\/www.slideshare.net\/HoseongLee6\/cnn-architecture-a-to-z\" target=\"_blank\"><b> \u201cCNN Architecture A to Z\u201d <\/b><\/a><\/li>\n<\/ul>\n\n<blockquote> Related Works <\/blockquote>\n<figure>\n\t<img src=\"\/assets\/img\/Model_Scaling\/1.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>2012\ub144 ImageNet \ubd84\ub958 \ub300\ud68c\uc5d0\uc11c \uc6b0\uc218\ud55c \uc131\ub2a5\uc744 \uac70\ub480\ub358 AlexNet \uc774\ud6c4\ub85c \uad49\uc7a5\ud788 \ub2e4\uc591\ud55c CNN architecture\ub4e4\uc774 \ud0c4\uc0dd\ub418\uc5c8\uc2b5\ub2c8\ub2e4. \uc720\uba85\ud55c VGG, ResNet \ub4f1\uc740 \ud558\ub098\uc758 \ubaa8\ub378\uc5d0\uc11c Layer \uac1c\uc218\ub97c \ub298\ub824 \uc815\ud655\ub3c4\ub97c \ub192\uc774\uae30\ub3c4 \ud558\uc600\ub294\ub370\uc694, \uc774\ucc98\ub7fc \ud558\ub098\uc758 \uc791\uc740 block \ud639\uc740 architecture\uc5d0\uc11c \ucd9c\ubc1c\ud558\uc5ec network\uc758 \ud06c\uae30\ub97c \ud0a4\uc6cc\uc8fc\ub294 \ubc29\uc2dd\uc744 <strong>Model Scaling<\/strong> \uc774\ub77c \ubd80\ub985\ub2c8\ub2e4. \uc77c\ubc18\uc801\uc73c\ub85c \uac00\uc7a5 \ub9ce\uc774 \uc4f0\uc774\ub294 \uac83\uc740 Layer\uc758 \uac1c\uc218 (depth)\ub97c \ub298\ub824\uc8fc\ub294 \ubc29\uc2dd\uc774\uba70, \uc885\uc885 \uac01 Convolution Layer\uc758 Filter \uac1c\uc218 (Width)\ub098 Input Image\uc758 Resolution\uc744 \ud0a4\uc6cc \uc8fc\uae30\ub3c4 \ud569\ub2c8\ub2e4. \uc774\ub807\uac8c \ub418\uba74 \uc5f0\uc0b0\ub7c9, \ucc98\ub9ac \uc18d\ub3c4\ub294 \ub290\ub824\uc9c0\uc9c0\ub9cc, \uadf8\ub9cc\ud07c \uc815\ud655\ub3c4\uac00 \uc88b\uc544\uc9c0\ub294 \ud6a8\uacfc\ub97c \ubcf4\uc785\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Model_Scaling\/2.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uadf8\ub9ac\uace0 \uc544\uc9c1\uae4c\uc9c0\ub3c4 \ub9ce\uc774 \uc0ac\uc6a9\ub418\uace0 \ub9ce\uc740 \ub17c\ubb38 \ub4e4\uc5d0\uc11c \ud0c0\uac9f\uc73c\ub85c \uc0bc\uc544\uc9c0\uace0 \uc788\ub294 <a href=\"https:\/\/arxiv.org\/abs\/1905.11946\" target=\"_blank\"><b> EfficientNet <\/b><\/a> \uc5d0\uc11c\ub294 \uc704\uc758 \uadf8\ub9bc\uacfc \uac19\uc774 width, depth, resolution 3\uac00\uc9c0 factor\ub97c \ub3d9\uc2dc\uc5d0 \uace0\ub824\ud558\uc5ec \ud0a4\uc6cc\uc8fc\ub294 <strong>Compound Scaling<\/strong> \uc774\ub77c\ub294 \uae30\ubc95\uc744 \uc81c\uc548\ud558\uc600\uace0, \uc2e4\uc81c\ub85c \ud070 \uc131\ub2a5 \ud5a5\uc0c1\uc744 \uc774\ub918\uc2b5\ub2c8\ub2e4. \ub2e4\ub9cc EfficientNet \ub17c\ubb38\uc5d0\uc11c\ub294 width, depth, resolution 3\uac00\uc9c0\ub97c \ub3d9\uc2dc\uc5d0 \ud0a4\uc6cc \uc8fc\uae34 \ud558\uc9c0\ub9cc \uc5b4\ub5a4 \uac83\uc5d0 \ub354 \ub192\uc740 \uac00\uc911\uce58\ub97c \ub458 \uc9c0\ub294 \ubcc4\ub2e4\ub978 \uace0\ub824 \uc5c6\uc774 \ub2e8\uc21c\ud788 Grid Search\ub97c \ud1b5\ud574 \ucc3e\uc558\ub294\ub370\uc694, \ubcf8 \ub17c\ubb38\uc5d0\uc11c\ub294 \uc2e4\uc81c Hardware\uc5d0\uc11c \ube60\ub974\uac8c \ub3d9\uc791\ud558\ub294 \uac83\ub3c4 \uac19\uc774 \uace0\ub824\ud558\uc5ec Model\uc744 Scaling \ud574\uc8fc\ub294 \uae30\ubc95\uc744 \uc81c\uc548\ud569\ub2c8\ub2e4.<\/p>\n\n<blockquote> Complexity of Scaled Models <\/blockquote>\n<p>\uc77c\ubc18\uc801\uc73c\ub85c \ubaa8\ub378\uc758 \ud06c\uae30\ub97c \uace0\ub824\ud560 \ub54c FLOPS (Floating Point Operations), Parameters (Parameter \uac1c\uc218), Activations (Activation \uac1c\uc218)\ub97c \uace0\ub824\ud569\ub2c8\ub2e4. \uc608\uc804 \ub17c\ubb38\ub4e4\uc5d0\uc11c\ub294 FLOPS, Parameters \uc815\ub3c4\ub9cc \uc5b8\uae09\ud558\uae30\ub3c4 \ud588\uc5c8\uc2b5\ub2c8\ub2e4. Activations\ub294 \ub9d0 \uadf8\ub300\ub85c Activation\uc758 \uac1c\uc218\uc774\uba70 \uc815\ud655\ud788\ub294 Conv Layer\ub97c \ud1b5\uacfc\ud558\uc5ec \ub098\uc628 Tensor\uc758 element \uac1c\uc218\uc778 \uc148\uc785\ub2c8\ub2e4.<\/p>\n\n<p>\uacb0\ub860 \uba3c\uc800 \ub9d0\uc500\ub4dc\ub9ac\uba74 \uc2e4\uc81c \uc5f0\uc0b0 \ucc98\ub9ac \uc18d\ub3c4\ub97c \uace0\ub824\ud574\uc57c \ud560 \ub54c, FLOPS\uc640 Parameter \uac1c\uc218\ub294 \uadf8\ub2e4\uc9c0 \ud070 \ub3c4\uc6c0\uc774 \ub418\uc9c0 \ubabb\ud569\ub2c8\ub2e4. \uadf8\ub798\uc11c \ub098\uc628 \uac83\uc774 \ubc14\ub85c Activations\uc785\ub2c8\ub2e4. \ud2b9\ud788 Parameter \uac1c\uc218\ub294 Input Resolution\uacfc \ubb34\uad00\ud558\uac8c \uacb0\uc815\ub418\ub294 \uc694\uc18c\uc774\ub2e4 \ubcf4\ub2c8 Input Resolution\uc774 \ucee4\uc9c0\uba74 \uc5f0\uc0b0 \uc18d\ub3c4\ub3c4 \ub290\ub824 \uc9c0\uace0 GPU Memory\ub3c4 \ub354 \ub9ce\uc774 \uc18c\ubaa8\ud558\uc9c0\ub9cc Parameter \uac1c\uc218\ub294 \ubcc0\ud558\uc9c0 \uc54a\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Model_Scaling\/3.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>Table 1\uc5d0\uc11c\ub294 depth, width, resolution \uc911 \ud558\ub098\uc758 dimension\uc744 \ud0a4\uc6cc\uc92c\uc744 \ub54c \uac19\uc740 FLOPS\uc5d0\uc11c Params, Acts \uac00 \uac01\uac01 \uc5b4\ub5bb\uac8c \ubcc0\ud558\ub294 \uc9c0\ub97c \ud45c\ub85c \ub098\ud0c0\ub0b4\uace0 \uc788\uace0, Table 2\uc5d0\uc11c\ub294 2\uac1c \ud639\uc740 3\uac1c\uc758 Dimension\uc744 \ub3d9\uc2dc\uc5d0 \uace0\ub824\ud588\uc744 \ub54c \uac19\uc740 FLOPS\uc5d0\uc11c Params, Acts\uac00 \uc5b4\ub5bb\uac8c \ubcc0\ud558\ub294 \uc9c0\ub97c \ub098\ud0c0\ub0b4\uace0 \uc788\uc2b5\ub2c8\ub2e4. \uc5b4\ub5bb\uac8c \uc774\ub7f0 \uc2dd\uc774 \uc720\ub3c4\ub418\ub294\uc9c0\ub294 \uc2e4\uc81c\ub85c FLOPS, Params, Acts\ub97c \uacc4\uc0b0\ud574\ubcf4\uc2dc\uba74 \uc54c \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<blockquote> Runtime of Scaled Models <\/blockquote>\n<p>\uc774\ub860\uc801\uc778 \uac12\ubcf4\ub2e4\ub294 \uc2e4\uc81c\ub85c GPU\uc5d0\uc11c \uc5bc\ub9c8\ub098 \ube60\ub974\uac8c \ub3d9\uc791\ud558\ub294 \uc9c0\uac00 \uc911\uc694\ud558\uaca0\uc8e0? 3\uac00\uc9c0 \uc694\uc18c (depth, width, resolution)\uc758 scaling\uc5d0 \ub530\ub978 FLOPS, Params, Acts\uc758 \ubcc0\ud654\ub97c \uc0b4\ud3b4\ubd24\uc73c\ub2c8 \uc2e4\uc81c runtime\uc5d0\uc11c \ub3d9\uc791 \uc2dc\ucf30\uc744 \ub54c\uc758 \uc2dc\uac04\uc744 \uce21\uc815\ud574\ubd05\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Model_Scaling\/4.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc0ac\uc2e4 \uc774\ubbf8 RegNet \ub17c\ubb38\uc5d0\uc11c\ub3c4 FLOPS \ubcf4\ub2e4\ub294 Activations\uac00 \uc2e4\uc81c Runtime\uc5d0\uc11c\uc758 Inference Time\uacfc \ud6e8\uc52c \ub354 \ub192\uc740 Correlation\uc744 \uac00\uc9c0\ub294 \uac83\uc744 \ubc1d\ud614\uc5c8\ub294\ub370 \ubcf8 \ub17c\ubb38\uc5d0\uc11c\ub3c4 Params\uae4c\uc9c0 \uac19\uc774 \uace0\ub824\ud574\uc11c Y \ucd95\uc740 1 epoch\ub97c \ud559\uc2b5\uc2dc\ud0a4\ub294\ub370 \uc18c\uc694\ub418\ub294 \uc2dc\uac04\uc73c\ub85c \uc124\uc815\ud55c \ub4a4, X \ucd95\uc73c\ub85c FLOPS, Params, Acts\uc744 \ub450\uace0 \uadf8\ub798\ud504\ub97c \uadf8\ub824\ubd05\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Model_Scaling\/5.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc5ed\uc2dc\ub098 Params\uac00 \uac00\uc7a5 Epoch Time\uacfc Correlation\uc774 \uc801\uc5c8\uace0 Acts\ub294 \ub9e4\uc6b0 \ub192\uc740 Correlation\uc744 \uac00\uc9c0\uace0 \uc788\uc2b5\ub2c8\ub2e4. \uadf8\ub9bc\uc758 \uc6b0\uce21 \ud558\ub2e8\uc744 \ubcf4\uba74 \ub2e8\uc21c\ud788 EfficientNet \ubfd0\ub9cc \uc544\ub2c8\ub77c RegNet Z, RegNet Y\ub4f1 \ub2e4\ub978 network\uc5d0\uc11c\ub3c4 \ube44\uc2b7\ud55c \uacbd\ud5a5\uc744 \ubcf4\uc774\ub294 \uac83\uc744 \uc54c \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n<blockquote> Fast Compound Model Scaling <\/blockquote>\n<p>Runtime\uc5d0 \uac00\uc7a5 \uc911\uc694\ud55c \uc601\ud5a5\uc744 \uc8fc\ub294 \uc694\uc18c\uac00 Activations\uc784\uc744 \uc54c\uc544\ub0c8\uc73c\ub2c8 \ubaa8\ub378\uc758 \ud06c\uae30\ub97c \ud0a4\uc6cc\uc904 \ub54c Activations\uc758 \uc99d\uac00\ub97c \ucd5c\uc18c\ud55c\uc73c\ub85c \ud558\ub3c4\ub85d \uc124\uacc4\ub97c \ud558\ub294 \uac83\uc774 \uc911\uc694\ud558\uaca0\uc8e0?<\/p>\n\n<p>\uc544\uae4c \uc124\uba85 \ub4dc\ub838\ub358 Table 1, 2\ub97c \ubcf4\uba74 width, depth, resolution \uc911\uc5d0\uc11c width\ub97c \uc99d\uac00\uc2dc\ud0a4\uba74\uc11c network\ub97c scaling\ud558\ub294 \uac83\uc774 Activations\ub97c \uac00\uc7a5 \uc801\uac8c \uc99d\uac00\uc2dc\ud0a4\ub294 \uac83\uc744 \uc54c \uc218 \uc788\uc5c8\uc2b5\ub2c8\ub2e4. \uadf8\ub807\ub2e4\uace0 width\ub9cc \uc99d\uac00\uc2dc\ud0a4\uba74 \ucd5c\uc801\uc758 \uc815\ud655\ub3c4\ub97c \uc5bb\uae30 \ud798\ub4e4\uaca0\uc8e0? \uadf8\ub798\uc11c width\ub97c \uc911\uc810\uc801\uc73c\ub85c \uc99d\uac00\uc2dc\ud0a4\ub418, depth\uc640 resolution\ub3c4 \ub3d9\uc2dc\uc5d0 \uc99d\uac00\uc2dc\ud0a4\ub294 \ubc29\ubc95\uc744 \uc81c\uc548\ud569\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Model_Scaling\/6.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc5ec\uae30\uc11c\ub294 \uac04\ub2e8\ud558\uac8c width\uc5d0 Alpha \ub9cc\ud07c \uac00\uc911\uce58\ub97c \ub450\uace0, depth\uc640 resolution\uc5d0\ub294 \uac01\uac01 (1- Alpha)\/2 \ub9cc\ud07c \uac00\uc911\uce58\ub97c \ub450\ub3c4\ub85d parameterize \uc2dc\ud0b5\ub2c8\ub2e4. \uc5ec\uae30\uc11c Alpha\uac00 0\uc774\uba74 depth\uc640 resolution\ub9cc scaling \ud574\uc8fc\ub294 \uc148\uc774\uace0, Alpha\uac00 1\/3\uc774\uba74 3\uac00\uc9c0 \uc694\uc18c\ub97c \ub3d9\ub4f1\ud558\uac8c \uace0\ub824\ud558\ub294 \uc148\uc774\uaca0\uc8e0? Alpha\ub97c 1\/3\ubcf4\ub2e4 \ud070 \uac12\uc744 \uc0ac\uc6a9\ud558\uba74 width\uc5d0 \ub354 \ub9ce\uc740 \uac00\uc911\uce58\ub97c \uc904 \uc218 \uc788\uac8c \ub429\ub2c8\ub2e4. \ub17c\ubb38\uc5d0\uc11c\ub294 Alpha\uc758 default \uac12\uc73c\ub85c 0.8\uc744 \uc0ac\uc6a9\ud558\uc600\uace0 \uc774\ub97c <strong>dWr scaling<\/strong> \uc774\ub77c\uace0 \ud45c\ud604\ud569\ub2c8\ub2e4. \ub610\ud55c \uc774\ub807\uac8c Width\uc5d0 \ub192\uc740 \uac00\uc911\uce58\ub97c \uc8fc\ub294 scaling \ubc29\uc2dd\uc744 <strong>Fast Scaling<\/strong> \uc774\ub77c\uace0 \ud45c\ud604\ud569\ub2c8\ub2e4.<\/p>\n\n<blockquote> \uc2e4\ud5d8 \uacb0\uacfc <\/blockquote>\n<p>\ub17c\ubb38\uc5d0\uc11c\ub294 Baseline Network\ub85c EfficientNet, RegNet Y, RegNet Z\uc744 \uc0ac\uc6a9\ud558\uc600\uace0 \uac01\uac01 Network\uc5d0 \ub300\ud55c Details\uc740 \ub2e4\uc74c\uacfc \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Model_Scaling\/7.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub610\ud55c \uacf5\uc815\ud558\uba74\uc11c\ub3c4 \uc7ac\ud604 \uac00\ub2a5\ud55c \uacb0\uacfc\ub97c \uc5bb\uae30 \uc704\ud574 simple &amp; weak optimization setup\uacfc difficult &amp; strong setup\uc744 \ub3d9\uc2dc\uc5d0 \uace0\ub824\ud558\uc5ec \ud559\uc2b5\uc744 \uc2dc\ucf30\uc2b5\ub2c8\ub2e4. \ud559\uc2b5\uc5d0 \ub300\ud55c Details\uc740 \ub2e4\uc74c\uacfc \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n<figure>\n\t<img src=\"\/assets\/img\/Model_Scaling\/8.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub17c\ubb38\uc5d0\uc120 \uc2e4\ud5d8 \uacb0\uacfc\uac00 \uad49\uc7a5\ud788 \ub2e4\uc591\ud558\uac8c \uc81c\uc2dc\uac00 \ub418\uc5b4\uc788\ub294\ub370 \uc800\ub294 \ud575\uc2ec\ub9cc \uac04\ub2e8\ud788 \uc124\uba85 \ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4. \uc790\uc138\ud55c \uacb0\uacfc\uac00 \uad81\uae08\ud558\uc2e0 \ubd84\ub4e4\uc740 \ub17c\ubb38\uc758 7\ud398\uc774\uc9c0 ~ 9\ud398\uc774\uc9c0\ub97c \ucc38\uace0\ud558\uc2dc\uba74 \uc88b\uc744 \uac83 \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Model_Scaling\/9.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc6b0\uc120 Figure 4\ub294 \uc800\ud76c\uac00 \uc798 \uc54c\uace0 \uc788\ub294 EfficientNet\uc758 Compound scaling\uc5d0 \ub300\ud55c \uc2e4\ud5d8 \uacb0\uacfc\uc785\ub2c8\ub2e4. Figure 4\uc758 \uc88c\uce21 \uadf8\ub9bc\uc744 \ubcf4\uba74 \uc5ed\uc2dc depth, width, resolution\uc744 \uac01\uac01 scaling \ud574\uc904 \ub54c \ubcf4\ub2e4 \ub3d9\uc2dc\uc5d0 (dwr, orig) \ud0a4\uc6cc\uc8fc\ub294 \uac83\uc774 \ub354 \ub0ae\uc740 error rate\ub97c \uc5bb\uc744 \uc218 \uc788\uc74c\uc744 \ubcf4\uc5ec\uc8fc\uace0 \uc788\uc2b5\ub2c8\ub2e4. EfficientNet \ub17c\ubb38\uc5d0\uc11c\ub294 depth, width, resolution\uc744 non-uniform \ud558\uac8c \ud0a4\uc6cc\uc92c\ub294\ub370 (orig), depth, width, resolution\uc744 uniform\ud558\uac8c \ud0a4\uc6cc\uc918\ub3c4 (dwr) \ube44\uc2b7\ud55c error rate\ub97c \uc5bb\uc744 \uc218 \uc788\uc5c8\ub2e4\uace0 \ud569\ub2c8\ub2e4. \ub610\ud55c Figure 4\uc758 \uc624\ub978\ucabd \uadf8\ub9bc\uc740 Runtime\uc744 \ub098\ud0c0\ub0b4\uace0 \uc788\ub294\ub370 \uc5ed\uc2dc width\ub9cc \ud0a4\uc6cc\uc8fc\ub294 \ubc29\uc2dd\uc774 Activations\ub97c \uc801\uac8c \uc99d\uac00\uc2dc\ucf1c\uc11c Runtime\uc774 \uac00\uc7a5 \ube68\ub790\uace0 \uadf8 \ub4a4\ub97c dwr, orig\uc774 \ub530\ub974\uace0 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>Figure 6\uc740 \ubc29\uae08 \uc124\uba85 \ub4dc\ub9b0 Compound Scaling (dwr)\uc5d0\uc11c width\uc758 \uac00\uc911\uce58\ub97c \ud0a4\uc6cc\uc900 Fast Scaling\uc758 \uc2e4\ud5d8 \uacb0\uacfc\ub97c \ubcf4\uc5ec\uc8fc\uace0 \uc788\uc2b5\ub2c8\ub2e4. Alpha \uac12\uc744 \ud0a4\uc6cc\uc918\ub3c4 \ube44\uc2b7\ud55c error rate\ub97c \uc5bb\uc744 \uc218 \uc788\uc5c8\uc9c0\ub9cc Runtime\uc5d0\uc11c\ub294 Alpha\ub97c \ud0a4\uc6cc\uc904\uc218\ub85d Runtime\uc774 \ub354 \uc904\uc5b4\ub4dc\ub294 \uac83\uc744 \uc54c \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc989, Compound Scaling\uc5d0\uc11c Width\uc5d0 \uac00\uc911\uce58\ub97c \ub354 \ud0a4\uc6cc\uc8fc\ub294 \ubc29\uc2dd\uc774 \uc815\ud655\ub3c4\ub294 \ube44\uc2b7\ud55c\ub370 \ub354 \ube60\ub974\uac8c \ub3d9\uc791\ud560 \uc218 \uc788\ub2e4\ub294 \ub73b\uc785\ub2c8\ub2e4. \uc774\ub7ec\ud55c \uacbd\ud5a5\uc740 EfficientNet \ubfd0\ub9cc \uc544\ub2c8\ub77c RegNet Y, RegNet Z\uc5d0\uc11c\ub3c4 \uad00\ucc30\uc774 \ub429\ub2c8\ub2e4.<\/p>\n\n<blockquote> \uacb0\ub860 <\/blockquote>\n<p>\uc624\ub298 \uc18c\uac1c \ub4dc\ub9b0 <a href=\" https:\/\/arxiv.org\/abs\/2103.06877v1 \" target=\"_blank\"><b> \u201cFast and Accurate Model Scaling\u201d <\/b><\/a> \ub17c\ubb38\uc740 \ubaa8\ub378\uc758 \ud06c\uae30\ub97c \ud0a4\uc6cc\uc8fc\ub294 Model Scaling \uae30\ubc95\uc744 \ub2e4\ub8ec \ub17c\ubb38\uc785\ub2c8\ub2e4.\n\uae30\uc874\uc5d0 EfficietNet\uc5d0\uc11c \uc81c\uc548\ub418\uc5c8\ub358 Compound Scaling \uae30\ubc95\uc774 \uc8fc\ub97c \uc774\ub8e8\uace0 \uc788\uc5c8\ub294\ub370 \uc2e4\uc81c Runtime\uc5d0\uc11c \ube60\ub974\uac8c \ub3d9\uc791\ud558\uae30 \uc704\ud574\uc120 FLOPS, Parameters\uac00 \uc544\ub2c8\ub77c Activations\ub97c \uace0\ub824\ud574\uc57c\ud568\uc744 \uc8fc\uc7a5\ud558\uba70, Activations\ub97c \uac00\uc7a5 \uc801\uac8c \uc99d\uac00\uc2dc\ud0a4\ub294 \ubc29\ubc95\uc740 Width\ub97c \ud0a4\uc6cc\uc8fc\ub294 \ubc29\uc2dd\uc784\uc744 \ubc1d\ud788\uba70 \uc774\ub97c \uc774\uc6a9\ud55c Fast Scaling \uae30\ubc95\uc744 \uc81c\uc548\ud558\uc600\uc2b5\ub2c8\ub2e4. \n\ub17c\ubb38 \uc790\uccb4\uac00 \uad49\uc7a5\ud788 \uc27d\uac8c \uc77d\ud788\uace0 \uc804\ub2ec\ud558\uace0\uc790 \ud558\ub294 \uba54\uc2dc\uc9c0\ub3c4 \uba85\ud655\ud558\uc9c0\ub9cc \uc758\ubbf8\uac00 \ud070 \uac83 \uac19\uc544\uc11c \uc88b\uc740 \ub17c\ubb38\uc778 \uac83 \uac19\uc2b5\ub2c8\ub2e4. \uae34 \uae00 \uc77d\uc5b4 \uc8fc\uc154\uc11c \uac10\uc0ac\ud569\ub2c8\ub2e4.<\/p>\n","<p>\uc548\ub155\ud558\uc138\uc694, \uc624\ub298\uc740 CVPR 2021\uc5d0\uc11c \ubc1c\ud45c \uc608\uc815\uc778 Facebook Research\uc758 <a href=\" https:\/\/arxiv.org\/abs\/2103.06877v1 \" target=\"_blank\"><b> \u201cFast and Accurate Model Scaling\u201d <\/b><\/a> \ub17c\ubb38\uc744 \uc18c\uac1c\ub4dc\ub9b4 \uc608\uc815\uc785\ub2c8\ub2e4. \uc81c\ubaa9\uc5d0\uc11c \uc720\ucd94\uac00 \uac00\ub2a5\ud558 \ub4ef \ubaa8\ub378\uc758 \ud06c\uae30\ub97c \uc870\uc808\ud574\uc8fc\ub294 (Scaling) \ubc29\ubc95\uc744 \ub2e4\ub8ec \ub17c\ubb38\uc774\uba70, \ud56d\uc0c1 \ubaa8\ub378\uc758 \ud06c\uae30\ub97c \ud0a4\uc6cc\uc8fc\uba74 \uc815\ud655\ub3c4\uac00 \uc88b\uc544\uc9c0\uc9c0\ub9cc \uadf8\uc5d0 \ub530\ub77c\uc11c \ucc98\ub9ac \uc18d\ub3c4\uac00 \ub290\ub824 \uc9c0\ub294 Trade-off \uad00\uacc4\ub97c \uac00\uc9c0\ub294\ub370 \uc774\ub97c \uc798 \ud0c0\uac1c\ud558\uae30 \uc704\ud55c \ubc29\ubc95\uc744 \uc81c\uc548\ud55c \ub17c\ubb38\uc785\ub2c8\ub2e4.<\/p>\n\n"],"pubDate":"Wed, 31 Mar 2021 00:00:00 +0000","link":"https:\/\/hoya012.github.io\/\/blog\/Fast-Scaling\/","guid":"https:\/\/hoya012.github.io\/\/blog\/Fast-Scaling\/"},{"title":"Unsupervised Anomaly Detection Using Style Distillation \ub9ac\ubdf0","description":["<p>\uc548\ub155\ud558\uc138\uc694, \uc624\ub298\uc740 \uc81c\uac00 \uc7ac\uc9c1 \uc911\uc778 Cognex\uc758 \uc5f0\uad6c\ud300 \ub3d9\ub8cc\ubd84\ub4e4\uc774 IEEE Access\uc5d0 \uc81c\ucd9c\ud558\uc2e0 <a href=\"https:\/\/ieeexplore.ieee.org\/document\/9288772\" target=\"_blank\"><b> \u201cUnsupervised Anomaly Detection Using Style Distillation\u201d <\/b><\/a> \ub17c\ubb38\uc744 \uc18c\uac1c\ub4dc\ub9b4 \uc608\uc815\uc785\ub2c8\ub2e4.<\/p>\n\n<p>Anomaly Detection\uc740 \uc81c \uc8fc \uad00\uc2ec\ubd84\uc57c\uc774\uc9c0\ub9cc \ube14\ub85c\uadf8\uc5d0\uc11c \uc18c\uac1c\ub4dc\ub9b0 \uc801\uc740 \ub9ce\uc774 \uc5c6\ub294 \uac83 \uac19\ub124\uc694. \uc774 \uae00\uc744 \ubcf4\uc2dc\uae30 \uc804\uc5d0 \uba3c\uc800 \uc544\ub798\uc758 \uae00\ub4e4\uc744 \uc77d\uace0 \uc624\uc2dc\uba74 \ub354 \uc774\ud574\uac00 \uc218\uc6d4\ud558\uc2e4 \uac83 \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/hoya012.github.io\/blog\/anomaly-detection-overview-1\/\" target=\"_blank\"><b> \u201cAnomaly Detection \uac1c\uc694\uff1a [1] \uc774\uc0c1\uce58 \ud0d0\uc9c0 \ubd84\uc57c\uc5d0 \ub300\ud55c \uc18c\uac1c \ubc0f \uc8fc\uc694 \ubb38\uc81c\uc640 \ud575\uc2ec \uc6a9\uc5b4, \uc0b0\uc5c5 \ud604\uc7a5 \uc801\uc6a9 \uc0ac\ub840 \uc815\ub9ac\u201d <\/b><\/a><\/li>\n  <li><a href=\"https:\/\/hoya012.github.io\/blog\/anomaly-detection-overview-2\/\" target=\"_blank\"><b> \u201cAnomaly Detection \uac1c\uc694\uff1a [2] Out-of-distribution(OOD) Detection \ubb38\uc81c \uc18c\uac1c \ubc0f \ud575\uc2ec \ub17c\ubb38 \ub9ac\ubdf0\u201d <\/b><\/a><\/li>\n  <li><a href=\"https:\/\/hoya012.github.io\/blog\/MVTec-AD\/\" target=\"_blank\"><b> \u201cMVTec AD\u2014A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection \ub9ac\ubdf0\u201d <\/b><\/a><\/li>\n<\/ul>\n\n<blockquote> Related Works <\/blockquote>\n\n<p>Unsupervised Anomaly Detection \ubb38\uc81c\ub97c \ud574\uacb0\ud558\uae30 \uc704\ud574 \ub2e4\uc591\ud55c \uc811\uadfc \ubc29\ubc95\uc774 \uc788\uc5c8\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"generative-adversarial-networks-\uae30\ubc18-\uc5f0\uad6c\">Generative Adversarial Networks \uae30\ubc18 \uc5f0\uad6c<\/h3>\n<p>\ub300\ud45c\uc801\uc73c\ub85c Generative Adversarial Networks \ubc29\ubc95\uc774 \uc788\ub294\ub370\uc694, \uc704\uc5d0\uc11c \uc18c\uac1c\ub4dc\ub838\ub358 <a href=\"https:\/\/hoya012.github.io\/blog\/MVTec-AD\/\" target=\"_blank\"><b> \u201cMVTec AD\u2014A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection \ub9ac\ubdf0\u201d <\/b><\/a> \uae00\uc744 \ubcf4\uc2dc\uba74 AnoGAN\uc5d0 \ub300\ud574 \uac04\ub2e8\ud788 \uc18c\uac1c\ub4dc\ub9b0 \uc801 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/OE-SDN\/1.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc815\uc0c1 sample\ub4e4\ub85c GAN\uc744 \ud559\uc2b5\uc2dc\ud0a8 \ub4a4, Generator\uc640 Discriminator\ub97c \uace0\uc815\uc2dc\ud0a4\uace0 test image\ub97c \uac00\uc7a5 \uc798 \ud45c\ud604\ud560 \uc218 \uc788\ub294 latent variable\uc744 optimization\uc744 \ud1b5\ud574 \ucc3e\uac8c \ub429\ub2c8\ub2e4. Test image\uac00 \uc815\uc0c1 sample\uc778 \uacbd\uc6b0 \ud559\uc2b5\uc5d0\uc11c \ubc30\uc6b4 \uc801\uc774 \uc788\uae30 \ub54c\ubb38\uc5d0 Generator\uac00 input\uacfc \ube44\uc2b7\ud55c image\ub97c \uc798 \uc0dd\uc131\ud558\uac8c \ub429\ub2c8\ub2e4. \ubc18\ub300\ub85c, test image\uac00 \ube44\uc815\uc0c1 sample\uc778 \uacbd\uc6b0 Generator\ub294 \uc798 \uc0dd\uc131\ud558\uc9c0 \ubabb\ud558\uac8c \ub418\uace0, \uacb0\uacfc\uc801\uc73c\ub85c Discriminator\ub97c \uc18d\uc774\uc9c0 \ubabb\ud569\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/OE-SDN\/2.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub2e4\ub9cc \uc774\ub7f0 \ubc29\ubc95\uc740 GAN\uc758 \uace0\uc9c8\uc801\uc778 \ubb38\uc81c\uc778 \ud559\uc2b5 \ubd88\uc548\uc815\uc131\uacfc Mode Collapse\ub97c \uc790\uc8fc \uacaa\uac8c \ub418\uace0, \ub370\uc774\ud130 \uc14b\uc774 \ub2e8\uc21c\ud558\uba74 \uc798 \ub418\uc9c0\ub9cc \ub370\uc774\ud130 \uc14b\uc774 \ubcf5\uc7a1\ud55c \uacbd\uc6b0 \uc131\ub2a5\uc774 \ub9e4\uc6b0 \ub5a8\uc5b4\uc9c0\ub294 \ubb38\uc81c\uac00 \ubc1c\uc0dd\ud569\ub2c8\ub2e4. \ub610\ud55c 1\uc7a5\uc529 optimization\uc744 \uc218\ud589\ud558\ub294 \ubc29\uc2dd\uc774\ub77c test \ud558\ub294 \ub370 \ub9ce\uc740 \uc2dc\uac04\uc774 \uc18c\uc694\ub418\ub294 \ubb38\uc81c\ub3c4 \uc788\uc2b5\ub2c8\ub2e4. \uc774\ub7ec\ud55c \uc810\ub4e4\uc744 \ud574\uacb0\ud558\uae30 \uc704\ud574 AnoGAN \uc774\ud6c4 \ub2e4\uc591\ud55c \uc5f0\uad6c\ub4e4\uc774 \uc9c4\ud589\ub418\uc5c8\uc2b5\ub2c8\ub2e4.<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/arxiv.org\/abs\/1805.06725\" target=\"_blank\"><b> \u201cGANomaly: Semi-Supervised Anomaly Detection via Adversarial Training\u201d, 2018 ACCV <\/b><\/a><\/li>\n  <li><a href=\"https:\/\/arxiv.org\/abs\/1901.08954\" target=\"_blank\"><b> \u201cSkip-GANomaly: Skip Connected and Adversarially Trained Encoder-Decoder Anomaly Detection\u201d, 2019 IJCNN <\/b><\/a><\/li>\n  <li><a href=\"https:\/\/www.sciencedirect.com\/science\/article\/abs\/pii\/S1361841518302640\" target=\"_blank\"><b> \u201cf-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks\u201d, 2019 MIA <\/b><\/a><\/li>\n<\/ul>\n\n<h3 id=\"deep-convolutional-autoencoders-and-variational-autoencoders-vae-\uae30\ubc18-\uc5f0\uad6c\">Deep Convolutional Autoencoders and Variational Autoencoders (VAE) \uae30\ubc18 \uc5f0\uad6c<\/h3>\n<p>\ub2e4\uc74c\uc740 \ucd5c\uadfc \uc8fc\ub85c \uc0ac\uc6a9\ub418\uace0 \uc788\ub294 Convolutional Autoencoder \uae30\ubc18 \ubc29\ubc95\ub4e4\uc785\ub2c8\ub2e4. <a href=\"https:\/\/hoya012.github.io\/blog\/anomaly-detection-overview-1\/\" target=\"_blank\"><b> \u201cAnomaly Detection \uac1c\uc694\uff1a [1] \uc774\uc0c1\uce58 \ud0d0\uc9c0 \ubd84\uc57c\uc5d0 \ub300\ud55c \uc18c\uac1c \ubc0f \uc8fc\uc694 \ubb38\uc81c\uc640 \ud575\uc2ec \uc6a9\uc5b4, \uc0b0\uc5c5 \ud604\uc7a5 \uc801\uc6a9 \uc0ac\ub840 \uc815\ub9ac\u201d <\/b><\/a> \uae00\uc5d0\uc11c \uc7a0\uc2dc \uc18c\uac1c\ub4dc\ub838\ub4ef\uc774 Autoencoder\ub97c \uc815\uc0c1 sample\ub4e4\ub85c \ud559\uc2b5\uc744 \uc2dc\ud0b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/OE-SDN\/3.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ubc29\ubc95\uc740 \ub2e8\uc21c\ud569\ub2c8\ub2e4. Input\uc744 Autoencoder\uc5d0 \ub123\uc5b4\uc11c Output\uc73c\ub85c Input\uacfc \uac19\uc740 image\ub97c \uc0dd\uc131\ud558\ub3c4\ub85d loss function\uc744 \uc124\uacc4\ud558\uc5ec \ud559\uc2b5\uc744 \uc2dc\ud0b5\ub2c8\ub2e4. \ub2e4\ub9cc \uc774\ub807\uac8c \ub418\uba74 \ub2e8\uc21c\ud788 Identity function\uc744 \ubc30\uc6b8 \uc218 \uc788\uae30 \ub54c\ubb38\uc5d0 Bottleneck \uad6c\uc870\ub97c \ucc28\uc6a9\ud558\uc5ec input\uc758 \ub370\uc774\ud130\ub97c \ucd95\uc18c\uc2dc\ud0a8 \ub4a4 \ub2e4\uc2dc \ub298\ub824\uc8fc\ub294 \ubc29\uc2dd\uc744 \ud1b5\ud574 input\uc744 \uc678\uc6b0\ub294 \uac83\uc744 \ubc29\uc9c0\ud569\ub2c8\ub2e4. \uc774\ub807\uac8c \ub418\uba74 Autoencoder\ub294 \uc815\uc0c1 sample\uc744 \ub123\uc5b4\uc8fc\uba74 \uc798 \ubcf5\uc6d0\ud558\uac8c \ud559\uc2b5\uc774 \ub429\ub2c8\ub2e4. \uc774 \ub54c test image\ub85c \ube44\uc815\uc0c1 sample\uc744 \ub123\uc5b4\uc8fc\uba74 Autoencoder\ub294 \ubc30\uc6e0\ub358 \ub300\ub85c \uc815\uc0c1 sample\ub85c \ubcf5\uc6d0\ud558\uac8c \ub418\uba70, Input\uacfc Output\uc758 \ucc28\uc774\ub97c \uad6c\ud558\uba74 \uadf8 \uc601\uc5ed\uc774 \ubc14\ub85c \uacb0\ud568 \uc601\uc5ed\uc774 \ub418\ub294 \ubc29\uc2dd\uc785\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/OE-SDN\/4.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub2e4\ub9cc Bottleneck Autoencoder\uc758 \uad6c\uc870\uc801\uc778 \ud2b9\uc9d5\uacfc, Loss function\uc73c\ub85c \ubcf4\ud1b5 Distance \uae30\ubc18\uc758 L2 loss\ub4f1\uc744 \uc0ac\uc6a9\ud558\uae30 \ub54c\ubb38\uc5d0 \uc804\uccb4\uc801\uc778 \ud2c0\uc740 \uc798 \ubcf5\uc6d0\ud558\uc9c0\ub9cc \uc138\ubd80\uc801\uc778 \ubd80\ubd84, \ud2b9\ud788 edge\uc640 \uac19\uc740 high-frequency \uc601\uc5ed\uc740 blurry\ud558\uac8c \ubcf5\uc6d0\ud558\ub294 \ud55c\uacc4\uac00 \uc874\uc7ac\ud569\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/OE-SDN\/5.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc774\ub97c \ud574\uacb0\ud558\uae30 \uc704\ud574 <a href=\"https:\/\/arxiv.org\/abs\/1807.02011\" target=\"_blank\"><b> \u201cImproving Unsupervised Defect Segmentation by Applying Structural Similarity to Autoencoders\u201d <\/b><\/a> \ub17c\ubb38\uc5d0\uc11c\ub294 Structural Similarity(SSIM) Loss\ub97c \uc0ac\uc6a9\ud558\uc5ec Autoencoder\ub97c \ud559\uc2b5\uc2dc\ud0a4\ub294 \ubc29\ubc95\uc744 \uc81c\uc548\ud558\uc600\uc2b5\ub2c8\ub2e4. \uac80\ucd9c \uc131\ub2a5\uc740 \uc57d\uac04 \uc88b\uc544\uc84c\uc9c0\ub9cc \uc5ec\uc804\ud788 blurry\ud55c output\uc744 \ub0b4\ub294 \ubb38\uc81c\ub294 \ub0a8\uc544\uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/OE-SDN\/6.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc774\ub97c \ud574\uacb0\ud558\uae30 \uc704\ud574 Autoencoder\uc758 Latent Variable\uc758 length, \ub2e4\ub978 \ub9d0\ub85c\ub294 Code size\ub97c \uc870\uc808\ud558\ub294 \ubc29\ubc95\uc774 \uc788\uc2b5\ub2c8\ub2e4. Code size\ub97c \ud0a4\uc6cc\uc8fc\uba74 \uc804\ubc18\uc801\uc778 \ubcf5\uc6d0 \uc131\ub2a5\uc774 \uc88b\uc544\uc9c0\uc9c0\ub9cc \uacb0\ud568 \uc601\uc5ed\ub3c4 \uadf8\ub300\ub85c \ud1b5\uacfc\uc2dc\ud0a4\ub294 \ubb38\uc81c\uac00 \ubc1c\uc0dd\ud558\uace0, \ubc18\ub300\ub85c Code size\ub97c \uc904\uc5ec\uc8fc\uba74 \uacb0\ud568 \uc601\uc5ed\uc740 \uc798 \ubb49\uac1c\ubc84\ub9ac\uc9c0\ub9cc \uc815\uc0c1 \uc601\uc5ed\ub3c4 bluury\ud558\uac8c \ubcf5\uc6d0\uc2dc\ud0a4\ub294 \ubb38\uc81c\uac00 \ubc1c\uc0dd\ud569\ub2c8\ub2e4. \uc989, \ub370\uc774\ud130 \uc14b\uc774 \ubc14\ub014 \ub54c \ub9c8\ub2e4 \ucd5c\uc801\uc758 Code Size\ub97c \ubc14\uafd4\uc8fc\ub294 \ubc29\ubc95\uc740 \ud559\uc2b5 \uc2dc\uac04\ub3c4 \uc624\ub798 \uac78\ub9ac\uace0 \uadf8\ub2e4\uc9c0 \ud6a8\uc728\uc801\uc774\uc9c4 \uc54a\uc2b5\ub2c8\ub2e4.<\/p>\n\n<blockquote> Unsupervised Anomaly Detection Using Style Distillation<\/blockquote>\n\n<figure>\n\t<img src=\"\/assets\/img\/OE-SDN\/7.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>Autoencoder\uac00 Blurry\ud558\uac8c Output\uc744 \ub0b4\ub294 \uac83\uc744 \uace0\uce58\ub294\uac8c \uc5b4\ub824\uc6b0\ub2c8 \uadf8\uac74 \uadf8\ub300\ub85c \ub194\ub450\uace0, Autoencoder\uac00 Blurry\ud558\uac8c Output\uc744 \ub0b4\ub294 \uac83\uc744 \ud749\ub0b4\ub0b4\ub294 \ub610 \ud558\ub098\uc758 Network\ub97c \uc124\uacc4\ud55c \ub4a4, Autoencoder\uacfc \uadf8 Network\uc758 Output \uac04\uc758 \ucc28\uc774\ub97c \ud1b5\ud574 Anomaly Detection\uc744 \ud558\ub294 \ubc29\uc2dd\uc744 \uc81c\uc548\ud558\uc600\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/OE-SDN\/8.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc6b0\uc120 Autoencoder\ub294 \uc6d0\ub798 \ud558\ub358 \ub300\ub85c \ud559\uc2b5\uc744 \uc2dc\ud0b5\ub2c8\ub2e4. \uadf8 \ub4a4, Autoencoder\ubcf4\ub2e4 Network Capacity\uac00 \ud070 \uc0c8\ub85c\uc6b4 Network (Style Distillation Network, \uc774\ud558 SDN)\ub97c \uac00\uc838\uc628 \ub4a4, SDN\uc740 Input\uc740 \uac19\uc740 training image\ub97c \uc0ac\uc6a9\ud558\uc9c0\ub9cc, \ubcf5\uc6d0\ud574\uc57c \ud558\ub294 Target\uc744 \uc6d0\ubcf8\uc774 \uc544\ub2cc Autoencoder\uc758 Output\uc73c\ub85c \uc0ac\uc6a9\ud569\ub2c8\ub2e4. \uc989, \uadf8\ub9bc\uc758 \uc544\ub798\uc758 SDN\uc740 Autoencoder\uac00 \ubcf5\uc6d0\ud558\ub294 \uac83\uc744 \ub530\ub77c\ud558\ub3c4\ub85d \ud559\uc2b5\ud558\uac8c \ub429\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/OE-SDN\/9.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub2e4\ub9cc \uc774\ub807\uac8c \ub418\uba74 \uc800\ud76c\ub294 SDN\uc740 Autoencoder\uc758 Blurry\ud55c Output\uc744 \ub0b4\ub294 \uac83\ub9cc \ubc30\uc6b0\uae38 \uc6d0\ud558\ub294\ub370 \ud559\uc2b5 \ub370\uc774\ud130\uc758 \ubd84\ud3ec\uac00 \ub2e4\uc591\ud558\uc9c0 \uc54a\uc740 \uacbd\uc6b0 \uadf8\ub0e5 \ubb49\uac1c\uc9c4 training sample\ub4e4\uc744 \uc678\uc6cc\ubc84\ub9ac\uac8c \ub429\ub2c8\ub2e4. \uc608\ub97c \ub4e4\uc5b4 MNIST\uc5d0\uc11c 3\uc744 \uc815\uc0c1 class\ub85c \ud559\uc2b5\uc2dc\ud0a4\ub294 \uacbd\uc6b0, \uc22b\uc790 3\uc740 \uc0dd\uae40\uc0c8\uac00 \ub2e4 \ube44\uc2b7\ud558\uae30 \ub54c\ubb38\uc5d0 SDN\uc740 input\uc744 blurry\ud558\uac8c \ub9cc\ub4dc\ub294 \uac83\uc774 \uc544\ub2cc, <strong>\ubb49\uac1c\uc9c4 3\uc744 \ub9cc\ub4dc\ub294 \uac83<\/strong> \ub9cc \ubc30\uc6b0\uac8c \ub429\ub2c8\ub2e4. \uc774\ub807\uac8c SDN\uc774 \ub2e8\uc21c \uc554\uae30\ud558\ub294 \uac83\uc744 \ub9c9\uae30 \uc704\ud574 Regularization \uae30\ubc95\uc774 \ud544\uc694\ud569\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/OE-SDN\/10.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uadf8\ub798\uc11c <a href=\"https:\/\/arxiv.org\/abs\/1812.04606\" target=\"_blank\"><b> \u201cDeep Anomaly Detection with Outlier Exposure\u201d, 2019 ICLR <\/b><\/a> \ub17c\ubb38\uc5d0\uc11c \uc81c\uc548\ud55c <strong>Outlier Exposure<\/strong> \uae30\ubc95\uc744 \uc774\uc6a9\ud558\uc600\uc2b5\ub2c8\ub2e4. \uc774 \ubc29\ubc95\uc740 Anomaly Detection Network\uc758 \ub354 \ub098\uc740 Representation\uc744 \uc704\ud574 \uc678\ubd80 \ub370\uc774\ud130 \uc14b\uc744 \uc0ac\uc6a9\ud558\ub294 \ubc29\ubc95\uc778\ub370\uc694, \uc800\uc790\ub4e4\uc740 \uc774 \uc678\ubd80 \ub370\uc774\ud130 \uc14b (Auxiliary \ub370\uc774\ud130 \uc14b)\uc744 \uc0dd\uc131\ud558\uae30 \uc704\ud574 \ub2e4\uc591\ud55c image \ubcc0\ud658 \uae30\ubc95\ub4e4\uc744 \uc0ac\uc6a9\ud558\uc600\uace0, \uadf8 \uc911\uc5d0 Rotation\uc774 \uac00\uc7a5 \uc131\ub2a5\uc774 \uc88b\uc544\uc11c \uc774\ub97c \ucc44\ud0dd\ud588\uc2b5\ub2c8\ub2e4. \uc989 Training \ub370\uc774\ud130\uc5d0 Rotation\uc744 \uac00\ud55c \ub4a4 Input\uc73c\ub85c \ub123\uc5b4\uc8fc\uace0, SDN\uc758 Output\uc774 \uc774 Input\uacfc \uac19\uc544\uc9c0\ub3c4\ub85d \ud559\uc2b5\uc744 \uc2dc\ud0a4\ub294 \ubc29\uc2dd\uc785\ub2c8\ub2e4. \uadf8\ub798\uc11c \ucd5c\uc885 \ubaa8\ub378\uc740 Outlier-Exposed Style Distillation Network (OE-SDN)\uc774\ub77c \ubd80\ub974\uac8c \ub429\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/OE-SDN\/11.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ud559\uc2b5 \uacfc\uc815\uc740 \uc77c\ub2e8 Autoencoder\ub97c \uba3c\uc800 \ud559\uc2b5 \uc2dc\ud0a8 \ub4a4, AE\uc640 OE-SDN\uc758 output\uc774 \uac19\uc544\uc9c0\ub3c4\ub85d \ud574\uc8fc\ub294 KD(Knowledge Distillation) loss term\uacfc, OE-SDN\uc5d0 Auxiliary \ub370\uc774\ud130 \uc14b\uc744 \ub123\uc5b4\uc11c \ubcf5\uc6d0 \uc2dc\ud0a4\ub294 OER(Outlier Exposed Regularization) loss term\uc744 \ub354\ud558\uc5ec multi-task learning\uc744 \ud1b5\ud574 \ud559\uc2b5\uc744 \uc2dc\ud0a4\uba70, \ub450 loss\uac04\uc758 weight\ub97c lambda\ub85c \uc870\uc808\ud569\ub2c8\ub2e4. \ub17c\ubb38\uc5d0\uc11c\ub294 0.5 \uac12\uc744 \uc0ac\uc6a9\ud588\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/OE-SDN\/12.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc774\ub807\uac8c \ud559\uc2b5\uc744 \uc2dc\ucf1c\uc11c \uc6d0\ub798 Autoencoder\uc640 \uc0c8\ub85c\uc6b4 OE-SDN 2\uac1c\uc758 network\ub97c \uc5bb\uac8c \ub418\uba74, \uadf8 \ub4a4\uc5d0\ub294 input\uc744 \ub450 network\uc5d0 \ub3d9\uc2dc\uc5d0 \ub123\uc5b4\uc900 \ub4a4, \ub450 Output\uc758 \ucc28\uc774\ub97c \ud1b5\ud574 Anomaly Detection\uc744 \ud558\uac8c \ub429\ub2c8\ub2e4. \uc774\uc81c\ub294 \ub450 network\uac00 \ubaa8\ub450 blurry\ud558\uac8c output\uc744 \ub0b4\uae30 \ub54c\ubb38\uc5d0 \ucc28\uc774\ub97c \uad6c\ud558\uba74 blur\ud55c \uc601\uc5ed\uc5d0\uc11c \ucc28\uc774\uac00 \uc801\uac8c \ubc1c\uc0dd\ud558\uac8c \ub418\uace0, \uacb0\ud568 \uc601\uc5ed\ub9cc \ub3c4\ub4dc\ub77c\uc9c0\uac8c \uac78\ub7ec\ub0bc \uc218 \uc788\uac8c \ub429\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/OE-SDN\/13.PNG\" alt=\"\" \/> \n<\/figure>\n\n<figure>\n\t<img src=\"\/assets\/img\/OE-SDN\/14.PNG\" alt=\"\" \/> \n<\/figure>\n<p>\uc2e4\ud5d8 \uacb0\uacfc Classification, Segmentation \ubaa8\ub450 \uae30\uc874 \ubc29\ubc95\ub4e4\ubcf4\ub2e4 \ub354 \uc88b\uc740 \uc131\ub2a5\uc744 \ub2ec\uc131\ud560 \uc218 \uc788\uc5c8\uc2b5\ub2c8\ub2e4.<\/p>\n\n<blockquote> \uacb0\ub860 <\/blockquote>\n<p>\uc624\ub298\uc740 \uc81c\uac00 \uadfc\ubb34 \uc911\uc778 Cognex\uc758 \uc5f0\uad6c\ud300 \ub3d9\ub8cc\ubd84\ub4e4\uc774 IEEE Access\uc5d0 \uc81c\ucd9c\ud558\uc2e0 <a href=\"https:\/\/ieeexplore.ieee.org\/document\/9288772\" target=\"_blank\"><b> \u201cUnsupervised Anomaly Detection Using Style Distillation\u201d <\/b><\/a> \ub17c\ubb38\uc744 \uc18c\uac1c\ub4dc\ub838\uc2b5\ub2c8\ub2e4. \uba85\ud655\ud55c \ubb38\uc81c\uc810 (AE\uc758 Blurry\ud55c Output)\uc744 \uc9c1\uc811\uc801\uc73c\ub85c \ud574\uacb0\ud558\ub294 \ub300\uc2e0 \uac04\uc811\uc801\uc73c\ub85c \ud574\uacb0\ud558\ub294 \uc7ac\ubbf8\ub09c \ubc29\ubc95\uc744 \uc81c\uc548\ud558\uc600\ub294\ub370\uc694, \uc774 \ubd84\uc57c\ub294 \uc544\uc9c1\uae4c\uc9c0 \ub354 \ud574\ubcfc\ub9cc\ud55c \uc5ec\uc9c0\uac00 \ub9ce\uc774 \ub0a8\uc544 \uc788\ub294 \uac83 \uac19\uc544\uc11c \uc7ac\ubc0c\ub294 \uac83 \uac19\uc2b5\ub2c8\ub2e4. \uae34 \uae00 \uc77d\uc5b4\uc8fc\uc154\uc11c \uac10\uc0ac\ud569\ub2c8\ub2e4.<\/p>\n","<p>\uc548\ub155\ud558\uc138\uc694, \uc624\ub298\uc740 \uc81c\uac00 \uc7ac\uc9c1 \uc911\uc778 Cognex\uc758 \uc5f0\uad6c\ud300 \ub3d9\ub8cc\ubd84\ub4e4\uc774 IEEE Access\uc5d0 \uc81c\ucd9c\ud558\uc2e0 <a href=\"https:\/\/ieeexplore.ieee.org\/document\/9288772\" target=\"_blank\"><b> \u201cUnsupervised Anomaly Detection Using Style Distillation\u201d <\/b><\/a> \ub17c\ubb38\uc744 \uc18c\uac1c\ub4dc\ub9b4 \uc608\uc815\uc785\ub2c8\ub2e4.<\/p>\n\n"],"pubDate":"Wed, 03 Mar 2021 00:00:00 +0000","link":"https:\/\/hoya012.github.io\/\/blog\/OE-SDN\/","guid":"https:\/\/hoya012.github.io\/\/blog\/OE-SDN\/"},{"title":"Transformers in Vision\uff1a A Survey [1] Transformer \uc18c\uac1c & Transformers for Image Recognition","description":["<p>\uc548\ub155\ud558\uc138\uc694, \uc624\ub298\uc740 \uc790\uc5f0\uc5b4 \ucc98\ub9ac(NLP)\uc5d0\uc11c \uc555\ub3c4\uc801\uc778 \uc131\ub2a5\uc744 \ubcf4\uc5ec\uc8fc\uba70 \uc8fc\ub958\ub85c \uc790\ub9ac\uc7a1\uc740 Transformers \ubaa8\ub378\uc744 \ucef4\ud4e8\ud130 \ube44\uc804\uc5d0 \uc801\uc6a9\ud558\ub824\ub294 \uc2dc\ub3c4\ub4e4\uc744 \uc815\ub9ac\ud55c \uc11c\ubca0\uc774 \ub17c\ubb38\uc778 <a href=\"https:\/\/arxiv.org\/abs\/2101.01169\" target=\"_blank\"><b> \u201cTransformers in Vision: A Survey\u201d <\/b><\/a> \ub97c \uc77d\uace0 \uac04\ub2e8\ud788 \uc815\ub9ac\ud574\ubcfc \uc608\uc815\uc785\ub2c8\ub2e4. \ub17c\ubb38\uc758 \ubd84\ub7c9\uc774 \ub9ce\uc740 \ub9cc\ud07c \uc5ec\ub7ec \ud3b8\uc5d0 \uac78\uccd0\uc11c \uc18c\uac1c\ub4dc\ub9b4 \uc608\uc815\uc774\uba70, 1\ud3b8\uc5d0\uc11c\ub294 Transformer\uc5d0 \ub300\ud574 \uac04\ub2e8\ud788 \uc18c\uac1c \ub4dc\ub9ac\uace0, Transformer\ub97c Image Recognition\uc5d0 \uc801\uc6a9\ud55c \ub300\ud45c \uc5f0\uad6c\ub4e4\uc744 \uc18c\uac1c\ub4dc\ub9b4 \uc608\uc815\uc785\ub2c8\ub2e4.<\/p>\n\n<blockquote> What is a Transformer? <\/blockquote>\n<p>\uc81c\uac00 Deep Learning\uc5d0 \uc785\ubb38\ud588\uc744 \ub54c\ub9cc \ud574\ub3c4 \uc790\uc5f0\uc5b4 \ucc98\ub9ac \ubd84\uc57c\uc5d0\uc11c\ub294 RNN\uc744 \uc8fc\ub85c \uc0ac\uc6a9\ud558\uace0, RNN\uc758 long term dependency \ubb38\uc81c\ub97c \ud574\uacb0\ud558\uae30 \uc704\ud574 \uace0\uc548\ub41c LSTM \ubc29\ubc95\ub860\uc774 \uc788\ub2e4! \uc815\ub3c4\ub9cc \uacf5\ubd80\ud558\uace0 \uc81c \uad00\uc2ec \ubd84\uc57c\uac00 \uc544\ub2c8\ub77c\uc11c \uacf5\ubd80\ub97c \uc548\ud588\ub294\ub370 \uac01\uc885 \ucee4\ubba4\ub2c8\ud2f0\uc5d0 Transformer, BERT, GPT \ub4f1 \uc0dd\uc18c\ud55c \uc6a9\uc5b4\ub4e4\uc774 \uc790\uc8fc \ubcf4\uc774\uae30 \uc2dc\uc791\ud588\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\uadf8\ub798\uc11c \uac04\ub2e8\ud788 \uacf5\ubd80\ub97c \ud574\ubcf4\ub2c8 <a href=\"https:\/\/arxiv.org\/abs\/1706.03762\" target=\"_blank\"><b> \u201cAttention Is All You Need\u201d <\/b><\/a> \ub17c\ubb38\uc5d0\uc11c \uc81c\uc548\ud55c Transformer\ub77c\ub294 \ubaa8\ub378\uc774 \uc790\uc5f0\uc5b4 \ucc98\ub9ac\uc758 \ub2e4\uc591\ud55c \ubd84\uc57c\uc5d0\uc11c SOTA\ub97c \ud729\uc4f8\uace0 \uc788\uc5c8\uace0, \uc774\ub97c \uc798 \ud65c\uc6a9\ud558\uae30 \uc704\ud574 \ub300\uc6a9\ub7c9\uc758 Unlabeled \ub370\uc774\ud130\uc14b\uc744 \ud65c\uc6a9\ud558\uc5ec Self-Supervised Learning\uc73c\ub85c \ud559\uc2b5\uc744 \uc2dc\ud0a8 \ub4a4 Downstream task\uc5d0 Fine-tuning\uc744 \uc2dc\ud0a4\ub294 BERT(Bidirectional Encoder Representations from Transformers), GPT(Generative Pre-trained Transformer) \ub4f1\uc774 \ub4a4\ub530\ub77c\uc11c \ucd9c\ud604\ud588\uc74c\uc744 \uc54c\uac8c \ub418\uc5c8\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Visual_Transformer\/2.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>Transformers\uc758 \uc131\uacf5 \uc694\uc18c\ub294 \ud06c\uac8c <strong>Self-Supervision<\/strong> \uacfc <strong>Self-Attention<\/strong> \uc73c\ub85c \ub098\ub20c \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc138\uc0c1\uc5d4 \uad49\uc7a5\ud788 \ub2e4\uc591\ud55c \ub370\uc774\ud130\uac00 \uc874\uc7ac\ud558\uc9c0\ub9cc, Supervised Learning\uc73c\ub85c \ud559\uc2b5\uc744 \uc2dc\ud0a4\uae30 \uc704\ud574\uc120 \uc77c\uc77c\uc774 annotation\uc744 \ub9cc\ub4e4\uc5b4\uc918\uc57c \ud558\ub294\ub370, \ub300\uc2e0 \ubb34\uc218\ud788 \ub9ce\uc740 unlabeled \ub370\uc774\ud130\ub4e4\uc744 \uac00\uc9c0\uace0 \ubaa8\ub378\uc744 \ud559\uc2b5 \uc2dc\ud0a4\ub294 Self-Supervised Learning\uc744 \ud1b5\ud574 \ubaa8\ub378\uc744 \ud559\uc2b5 \uc2dc\ud0ac \uc218 \uc788\uc2b5\ub2c8\ub2e4. \ucef4\ud4e8\ud130 \ube44\uc804\uc758 Self-Supervised Learning \uc5f0\uad6c\ub4e4\uc740 <a href=\"https:\/\/hoya012.github.io\/blog\/Self-Supervised-Learning-Overview\/\" target=\"_blank\"><b> \u201cUnsupervised Visual Representation Learning Overview\uff1a Toward Self-Supervision\u201d <\/b><\/a> \uae00\uc5d0 \uc815\ub9ac\ud574 \ub450\uc5c8\uc73c\ub2c8 \uba3c\uc800 \uc77d\uace0 \uc624\uc2dc\ub294 \uac83\uc744 \uad8c\uc7a5 \ub4dc\ub9bd\ub2c8\ub2e4. \ubb34\ud2bc, \uc790\uc5f0\uc5b4 \ucc98\ub9ac\uc5d0\uc11c\ub3c4 Self-Supervised Learning\uc744 \ud1b5\ud574 \uc8fc\uc5b4\uc9c4 \ub9c9\ub300\ud55c \ub370\uc774\ud130 \uc14b\uc5d0\uc11c generalizable representations\uc744 \ubc30\uc6b8 \uc218 \uc788\uac8c \ub418\uba70, \uc774\ub807\uac8c pretraining\uc2dc\ud0a8 \ubaa8\ub378\uc744 downstream task\uc5d0 fine-tuning \uc2dc\ud0a4\uba74 \uc6b0\uc218\ud55c \uc131\ub2a5\uc744 \uac70\ub458 \uc218 \uc788\uac8c \ub429\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Visual_Transformer\/13.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub610 \ub2e4\ub978 \uc131\uacf5 \uc694\uc18c\uc778 Self-Attention\uc740 \ub9d0 \uadf8\ub300\ub85c \uc2a4\uc2a4\ub85c attention\uc744 \uacc4\uc0b0\ud558\ub294 \uac83\uc744 \uc758\ubbf8\ud558\uba70 CNN, RNN\uacfc \uac19\uc774 inductive bias\uac00 \ub9ce\uc774 \ub4e4\uc5b4\uac00 \uc788\ub294 \ubaa8\ub378\ub4e4\uacfc\ub294 \ub2e4\ub974\uac8c \ucd5c\uc18c\ud55c\uc758 inductive bias\ub97c \uac00\uc815\ud569\ub2c8\ub2e4. Self-Attention Layer\ub97c \ud1b5\ud574 \uc8fc\uc5b4\uc9c4 sequence\uc5d0\uc11c \uac01 token set elements(ex, words in language or patches in an image)\uac04\uc758 \uad00\uacc4\ub97c \ud559\uc2b5\ud558\uba74\uc11c \uad11\ubc94\uc704\ud55c context\ub97c \uace0\ub824\ud560 \uc218 \uc788\uac8c \ub429\ub2c8\ub2e4. \ub354 \uc790\uc138\ud55c \ub0b4\uc6a9\uc774 \uad81\uae08\ud558\uc2e0 \ubd84\ub4e4\uc740 Transformer \ub17c\ubb38\uc744 \uc9c1\uc811 \uc77d\uc5b4 \ubcf4\uc2dc\ub294 \uac83\uc744 \ucd94\ucc9c \ub4dc\ub9bd\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Visual_Transformer\/1.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc704\uc758 \uadf8\ub9bc\uc744 \ubcf4\uc2dc\uba74 \uc54c \uc218 \uc788\ub4ef\uc774 \ub9e4\ub144 Top-tier \ud559\ud68c, arxiv\uc5d0 Transformer \uad00\ub828 \uc5f0\uad6c\ub4e4\uc774 \ube60\ub978 \uc18d\ub3c4\ub85c \ub298\uc5b4\ub098\uace0 \uc788\uace0 \uc791\ub144(2020\ub144)\uc5d0\ub294 \uac70\uc758 \uc804\ub144 \ub300\ube44 2\ubc30 \uc774\uc0c1\uc758 \ub17c\ubb38\uc774 \uc81c\ucd9c\uc774 \ub418\uc5c8\uc2b5\ub2c8\ub2e4. \ubc14\uc57c\ud750\ub85c Transformer \uc2dc\ub300\uac00 \uc5f4\ub9b0 \uc148\uc774\uc8e0. \uadfc\ub370 \uc8fc\ubaa9\ud560\ub9cc\ud55c \uc810\uc740 Transformer\uac00 \uc790\uc5f0\uc5b4 \ucc98\ub9ac \ubfd0\ub9cc \uc544\ub2c8\ub77c \uac15\ud654 \ud559\uc2b5, \uc74c\uc131 \uc778\uc2dd, \ucef4\ud4e8\ud130 \ube44\uc804 \ub4f1 \ub2e4\ub978 task\uc5d0\ub3c4 \uc801\uc6a9\ud558\uae30 \uc704\ud55c \uc5f0\uad6c\ub4e4\uc774 \ud558\ub098 \ub458 \uc2dc\uc791\ub418\uace0 \uc788\ub2e4\ub294 \uc810\uc785\ub2c8\ub2e4. \uadf8\ub798\uc11c \uc624\ub298\uc740 \ucef4\ud4e8\ud130 \ube44\uc804\uc5d0 Transformer\ub97c \uc801\uc6a9\ud55c \uc5f0\uad6c\ub4e4\uc744 \uac04\ub7b5\ud788 \uc815\ub9ac\ud574\ubcfc \uc608\uc815\uc785\ub2c8\ub2e4.<\/p>\n\n<blockquote> Transformers &amp; Self-Attention in Vision <\/blockquote>\n\n<p>\ub17c\ubb38\uc5d0\uc11c\ub294 \ucef4\ud4e8\ud130 \ube44\uc804\uc5d0 Transformer\uc744 \uc801\uc6a9\uc2dc\ud0a8 \uc5f0\uad6c\ub4e4\uc744 \ud06c\uac8c 10\uac00\uc9c0 task\ub85c \ub098\ub220\uc11c \uc815\ub9ac\ub97c \ud574\ub450\uc5c8\uc2b5\ub2c8\ub2e4.<\/p>\n\n<ul>\n  <li>Image Recognition (Classification)<\/li>\n  <li>Object Detection<\/li>\n  <li>Segmentation<\/li>\n  <li>Image Generation<\/li>\n  <li>Low-level Vision<\/li>\n  <li>Multi-modal Tasks<\/li>\n  <li>Video Understanding<\/li>\n  <li>Low-shot Learning<\/li>\n  <li>Clustering<\/li>\n  <li>3D Analysis<\/li>\n<\/ul>\n\n<figure>\n\t<img src=\"\/assets\/img\/Visual_Transformer\/3.png\" alt=\"\" \/> \n<\/figure>\n\n<p>\ubd84\ub7c9\uc774 \ub108\ubb34 \ub9c9\ub300\ud55c \ub9cc\ud07c \ubaa8\ub4e0 task\ub97c \uc624\ub298 \ub2e4 \uc18c\uac1c \ub4dc\ub9ac\uae34 \uc5b4\ub824\uc6b8 \uac83 \uac19\uace0 \ucef4\ud4e8\ud130 \ube44\uc804\ud558\uba74 \uac00\uc7a5 \uba3c\uc800 \ub5a0\uc624\ub974\ub294 Image Recognition\uc744 \ub2e4\ub8ec \uc5f0\uad6c\ub4e4\uc744 \uc18c\uac1c \ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\ucef4\ud4e8\ud130 \ube44\uc804\uc5d0 Deep Learning\uc744 \uc801\uc6a9\ud558\ub294 \uc5f0\uad6c\ub4e4\uc744 \uc0dd\uac01\ud558\uba74 \uac00\uc7a5 \uba3c\uc800 \ub5a0\uc624\ub974\ub294 \uac83\uc774 \ubc14\ub85c Convolutional Neural Network (CNN) \uc785\ub2c8\ub2e4. AlexNet\uc744 \ud544\ub450\ub85c \uad49\uc7a5\ud788 \ub2e4\uc591\ud55c \ubc29\ubc95\ub4e4\uc774 \uc81c\uc548\ub418\uc5c8\uace0 \uc6b0\uc218\ud55c \uc131\ub2a5\uc744 \ubcf4\uc600\uc9c0\ub9cc \ub2e8\uc810\ub3c4 \uc874\uc7ac\ud569\ub2c8\ub2e4. \uc6b0\uc120 \uace0\uc815\ub41c size\uc758 convolution filter size (window size)\ub97c \uc0ac\uc6a9\ud558\uae30 \ub54c\ubb38\uc5d0 receptive field \ubc16\uc5d0 \uc788\ub294 pixel\uacfc\uc758 relation\uc744 \ubc30\uc6b8 \uc218 \uc5c6\uc2b5\ub2c8\ub2e4. \ub610\ud55c convolution filter\uc758 weight \uac12\ub4e4\uc740 \ud559\uc2b5\uc774 \ub05d\ub098\uba74 \uace0\uc815\ub41c \uac12\uc744 \uc0ac\uc6a9\ud558\uae30 \ub54c\ubb38\uc5d0 input\uc5d0 \uc57d\uac04\uc758 \ubcc0\ud654\uac00 \uc0dd\uaca8\ub3c4 dynamically \ubcc0\ud654\ud558\uc9c0 \ubabb\ud569\ub2c8\ub2e4. \uc774\ub7ec\ud55c \ub2e8\uc810\ub4e4\uc740 Self-Attention\uacfc Transformer\ub97c \uc0ac\uc6a9\ud558\uba74 \ud574\uacb0\ud560 \uc218 \uc788\ub294\ub370\uc694, \ub300\ud45c\uc801\uc778 \uc5f0\uad6c\ub4e4\uc744 \ud558\ub098\uc529 \uc18c\uac1c \ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"non-local-neural-networks\">Non-local Neural Networks<\/h3>\n<p>Non-local means \uc5f0\uc0b0\uc740 image denoising\uc5d0 \uc8fc\ub85c \uc0ac\uc6a9\ub418\ub358 \ubc29\ubc95\uc778\ub370, \ud575\uc2ec \uc544\uc774\ub514\uc5b4\ub97c \ubc14\ud0d5\uc73c\ub85c Neural Network\uc5d0 \uc801\uc6a9\uc2dc\ud0a8 \uc5f0\uad6c\uac00 \uc81c\uc548\ub418\uc5c8\uc2b5\ub2c8\ub2e4. \ub17c\ubb38\uc758 \uc81c\ubaa9\uc740 <a href=\"https:\/\/arxiv.org\/abs\/1711.07971\" target=\"_blank\"><b> \u201cNon-local Neural Networks\u201d <\/b><\/a> \uc774\uba70 2018 CVPR\uc5d0 \ubc1c\ud45c\ub418\uc5c8\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Visual_Transformer\/4.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>Non-local block\uc744 \ud1b5\ud574 spatial, temporal \ucd95\uc5d0\uc11c \ubaa8\ub450 long-range dependency\ub97c \ud655\ubcf4\ud560 \uc218 \uc788\uac8c \ub429\ub2c8\ub2e4. \uc989, input image (\ud639\uc740 feature map)\uc5d0\uc11c \ud2b9\uc815 pixel\uacfc \ub098\uba38\uc9c0 \ubaa8\ub4e0 pixel \ub4e4 \uac04\uc758 relation\uc744 weighted sum \ud615\ud0dc\ub85c \uacc4\uc0b0\ud558\uba74\uc11c relation\uc744 \ubc30\uc6b8 \uc218 \uc788\uac8c \ub429\ub2c8\ub2e4. \uc989, self-attention\uc758 \uc77c\uc885\uc774\ub77c\uace0 \ubcfc \uc218 \uc788\uc73c\uba70, CNN\uc5d0\uc11c\ub294 \uc8fc\uc5b4\uc9c4 \uac70\ub9ac \ubc16\uc5d0 \uc788\ub294 pixel\uacfc\ub294 \uc544\ubb34\ub7f0 relation\ub3c4 \ubc30\uc6b8 \uc218 \uc5c6\uc5c8\uc9c0\ub9cc Non-local Neural Network\ub294 \uadf8 \uac83\uc774 \uac00\ub2a5\ud574\uc9c0\ub294 \uc148\uc785\ub2c8\ub2e4. \ub17c\ubb38\uc5d0\uc11c\ub294 3D \ub370\uc774\ud130\uc778 Video\uc758 Classification\uc5d0 \uc801\uc6a9\uc744 \ud558\uc600\uc9c0\ub9cc 2D Image\uc5d0 \uc801\uc6a9\ud574\ub3c4 \uc131\ub2a5 \ud5a5\uc0c1\uc744 \uc5bb\uc744 \uc218 \uc788\ub294 \ubc29\ubc95\uc785\ub2c8\ub2e4.<\/p>\n\n<p>\uc81c\uac00 \ucc38\uc5ec\ud588\uc5c8\ub358 TensorFlow KR \ub17c\ubb38 \uc77d\uae30 \ubaa8\uc784 PR-12\uc758 \uae40\ud0dc\uc624\ub2d8\uaed8\uc11c \uc774 \ub17c\ubb38\uc744 \ud55c\uae00\ub85c \uc798 \uc124\uba85\ud574\uc8fc\uc2e0 \uc601\uc0c1 \uc790\ub8cc\uac00 \uc788\uc5b4\uc11c \uac19\uc774 \ucc38\uace0\ud558\uc2dc\uba74 \uc88b\uc744 \uac83 \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/youtu.be\/ZM153wo3baA\" target=\"_blank\"><b> \u201cPR-083: Non-local Neural Networks\u201d <\/b><\/a><\/li>\n<\/ul>\n\n<h3 id=\"criss-cross-attention\">Criss-Cross Attention<\/h3>\n<p>\ub2e4\uc74c\uc740 2019 ICCV\uc5d0\uc11c \ubc1c\ud45c\ub41c <a href=\"https:\/\/arxiv.org\/abs\/1811.11721\" target=\"_blank\"><b> \u201cCCNet: Criss-Cross Attention for Semantic Segmentation\u201d <\/b><\/a> \ub17c\ubb38\uc5d0\uc11c \uc81c\uc548\ud55c Criss-Cross Attention\uc785\ub2c8\ub2e4. \uc704\uc5d0\uc11c \uc124\uba85\ub4dc\ub9b0 Non-local block\uc744 \uc0ac\uc6a9\ud558\uba74 full-image contextual information\uc744 \ubaa8\ub378\ub9c1\ud560 \uc218 \uc788\uc9c0\ub9cc memory\uc640 computational cost\uac00 \ub9e4\uc6b0 \ud06c\ub2e4\ub294 \ud55c\uacc4\uac00 \uc788\uc2b5\ub2c8\ub2e4. \uc804\uccb4 feature map\uc5d0 \ub300\ud574 <strong>dense\ud55c attention map<\/strong> \uc744 \uacc4\uc0b0\ud574\uc57c\ud558\uae30 \ub54c\ubb38\uc778\ub370 \uc774\ub97c \uadf9\ubcf5\ud558\uae30 \uc704\ud574 <strong>sparse\ud558\uac8c attention map\uc744 \uacc4\uc0b0\ud558\ub294<\/strong> Criss-Cross Attention \ubc29\ubc95\uc744 \uc81c\uc548\ud569\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Visual_Transformer\/5.PNG\" alt=\"\" \/> \n<\/figure>\n<p>\uc774\ub7ec\ud55c \ubc29\ubc95\uc744 \ud1b5\ud574 \uc544\uc8fc \ubbf8\ubbf8\ud558\uac8c \uc815\ud655\ub3c4\uac00 \ub5a8\uc5b4\uc9c8 \uc21c \uc788\uc9c0\ub9cc \uacc4\uc0b0 \ubcf5\uc7a1\ub3c4\ub97c \ud06c\uac8c \uc904\uc77c \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\uc81c\uac00 \ucc38\uc5ec\ud588\uc5c8\ub358 TensorFlow KR \ub17c\ubb38 \uc77d\uae30 \ubaa8\uc784 PR-12\uc758 \uae40\ud0dc\uc624\ub2d8\uaed8\uc11c \uc774 \ub17c\ubb38\uc744 \ud3ec\ud568\ud558\uc5ec CNN\uc5d0 Attention\uc744 \uc801\uc6a9\ud55c \uc0ac\ub840\ub4e4\uc744 \ud55c\uae00\ub85c \uc798 \uc124\uba85\ud574\uc8fc\uc2e0 \uc601\uc0c1 \uc790\ub8cc\uac00 \uc788\uc5b4\uc11c \uac19\uc774 \ucc38\uace0\ud558\uc2dc\uba74 \uc88b\uc744 \uac83 \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/youtu.be\/Dvi5_YC8Yts\" target=\"_blank\"><b> \u201cPR-163: CNN Attention Networks\u201d <\/b><\/a><\/li>\n<\/ul>\n\n<h3 id=\"stand-alone-self-attention\">Stand-alone Self-Attention<\/h3>\n<p>\ub2e4\uc74c\uc740 2019 NeurIPS\uc5d0\uc11c \ubc1c\ud45c\ub41c <a href=\"https:\/\/arxiv.org\/abs\/1906.05909\" target=\"_blank\"><b> \u201cStand-Alone Self-Attention in Vision Models\u201d <\/b><\/a> \ub17c\ubb38\uc5d0\uc11c \uc81c\uc548\ud55c Stand-Alone Self-Attention\uc785\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Visual_Transformer\/6.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc774 \ub17c\ubb38\uc5d0\uc11c\ub294 \ubaa8\ub4e0 Convolutional Layer\ub97c Local Self-Attention Layer\ub85c \ub300\uccb4\ud558\ub294 \ubc29\ubc95\uc744 \uc81c\uc548\ud569\ub2c8\ub2e4. \uc774 Local Self-Attention Layer\ub97c ResNet-50\uc5d0 \uc801\uc6a9\ud558\uba74 \ub354 \uc801\uc740 \uc218\uc758 \ud30c\ub77c\ubbf8\ud130\uc640 \uc5f0\uc0b0\ub7c9\uc73c\ub85c \ub354 \ub192\uc740 \uc815\ud655\ub3c4\ub97c \ub2ec\uc131\ud560 \uc218 \uc788\ub2e4\uace0 \ud569\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"local-relation-networks\">Local Relation Networks<\/h3>\n<p>\ub2e4\uc74c\uc740 2019 ICCV\uc5d0\uc11c \ubc1c\ud45c\ub41c <a href=\"https:\/\/arxiv.org\/abs\/1904.11491\" target=\"_blank\"><b> \u201cLocal Relation Networks for Image Recognition\u201d <\/b><\/a> \ub17c\ubb38\uc5d0\uc11c \uc81c\uc548\ud55c Local Relation Network\uc785\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Visual_Transformer\/7.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc55e\uc11c \ub9d0\uc500 \ub4dc\ub838\ub4ef CNN\uc740 \ud559\uc2b5\uc774 \ub05d\ub09c \ub4a4\uc5d0\ub294 weight\uac00 \uace0\uc815\ub418\uae30 \ub54c\ubb38\uc5d0 input\uc758 \ubcc0\ud654\uc5d0 \ub530\ub77c weight\ub97c adaptive\ud558\uac8c \uc218\uc815\ud558\uc9c0 \ubabb\ud558\ub294 \ub2e8\uc810\uc744 \uac00\uc9c0\uace0 \uc788\uc5c8\uc2b5\ub2c8\ub2e4. \uc774\ub7ec\ud55c \uc810\uc5d0 \uc8fc\ubaa9\ud558\uc5ec \ubbf8\ubd84 \uac00\ub2a5\ud55c Local Relation Layer\ub97c \uc81c\uc548\ud558\uc600\uc2b5\ub2c8\ub2e4. \uc774 Layer\ub294 \uac19\uc740 window\uc5d0 \uc788\ub294 pixel\ub4e4\uac04\uc758 compositional relation\uc5d0 \uae30\ubc18\ud558\uc5ec adaptive\ud558\uac8c weight aggregation\uc744 \uc218\ud589\ud558\ub294 \ubc29\ubc95\uc744 \uc81c\uc548\ud569\ub2c8\ub2e4. \uc704\uc758 \uadf8\ub9bc\uacfc \uac19\uc774 \uc0c8\uc758 \ub208\uacfc \ubd80\ub9ac\uc758 spatial variability\ub97c \ud45c\ud604\ud558\uae30 \uc704\ud574 \uae30\uc874 CNN\uc740 3\uac1c\uc758 \ucc44\ub110\uc774 \ud544\uc694\ud588\ub2e4\uba74, Local Relation Layer\ub294 \uc624\ub85c\uc9c0 1\uac1c\uc758 \ucc44\ub110\ub85c \uc774\ub97c \ud45c\ud604\ud560 \uc218 \uc788\uac8c \ub418\ub294 \uc148\uc785\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Visual_Transformer\/8.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc790\uc138\ud55c Layer \ub3d9\uc791 \ubc29\uc2dd\uc740 \uc704\uc758 \uadf8\ub9bc\uacfc \uac19\uc73c\uba70 \uc790\uc138\ud55c \ub0b4\uc6a9\uc740 \ub17c\ubb38\uc744 \ucc38\uace0\ud558\uc2dc\uba74 \uc88b\uc744 \uac83 \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"attention-augmented-convolutional-networks\">Attention Augmented Convolutional Networks<\/h3>\n<p>\ub2e4\uc74c\uc740 2019 ICCV\uc5d0\uc11c \ubc1c\ud45c\ub41c <a href=\"https:\/\/arxiv.org\/abs\/1904.09925\" target=\"_blank\"><b> \u201cAttention Augmented Convolutional Networks\u201d <\/b><\/a> \ub17c\ubb38\uc5d0\uc11c \uc81c\uc548\ud55c Attention Augmented Convolutional Networks\uc785\ub2c8\ub2e4. CNN\uc758 translation equivariance (\uc785\ub825\uc758 \uc704\uce58\uac00 \ubcc0\ud558\uba74 \ucd9c\ub825\ub3c4 \ub3d9\uc77c\ud558\uac8c \uc704\uce58\uac00 \ubcc0\ud558\ub294 \uc131\uc9c8)\uc740 \uc720\uc9c0\ud558\uba74\uc11c Self-Attention \uba54\ucee4\ub2c8\uc998\uc744 \uc801\uc6a9\ud558\uae30 \uc704\ud55c Relative Position Encoding \uae30\ubc18\uc758 \uc5f0\uc0b0\uc744 \uc81c\uc548\ud569\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Visual_Transformer\/9.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ubaa8\ub4e0 Convolution \uc5f0\uc0b0\uc744 Self-Attention \uc5f0\uc0b0\uc73c\ub85c \ub300\uccb4\ud558\uba74 \uc5f0\uc0b0 \ud6a8\uc728\uc744 \ub192\uc77c \uc218 \uc788\uc9c0\ub9cc \ucd5c\uace0 \uc131\ub2a5\uc740 Convolution \uc5f0\uc0b0\uacfc \uac19\uc774 \uc4f8 \ub54c \ub2ec\uc131\ud560 \uc218 \uc788\uc73c\uba70, \uc704\uc758 \uadf8\ub9bc\uacfc \uac19\uc774 Self-Attention\uc744 \ud1b5\ud574 \ub098\uc628 Output\uacfc \uc77c\ubc18 Convolution \uc5f0\uc0b0\uc744 \ud1b5\ud574 \ub098\uc628 Output\uc744 concatenate \ud558\uc5ec \uc0ac\uc6a9\ud569\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Visual_Transformer\/10.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ube44\uc2b7\ud55c \ubc29\ubc95\uc744 \ub2e4\ub8ec \uc5f0\uad6c\uc778 Channel-wise\ub85c Attention Augmented \uc2dc\ud0a4\ub294 Squeeze-Excitation Network (SENet)\ubcf4\ub2e4 \ub354 \uc88b\uc740 \uc131\ub2a5\uc744 \ub2ec\uc131\ud560 \uc218 \uc788\uc5c8\ub2e4\uace0 \ud569\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"vectorized-self-attention\">Vectorized Self-Attention<\/h3>\n<p>\ub2e4\uc74c\uc740 2020 CVPR\uc5d0\uc11c \ubc1c\ud45c\ub41c <a href=\"https:\/\/arxiv.org\/abs\/2004.13621\" target=\"_blank\"><b> \u201cExploring Self-attention for Image Recognition\u201d <\/b><\/a> \ub17c\ubb38\uc5d0\uc11c \uc81c\uc548\ud55c Vectorized Self-Attention\uc785\ub2c8\ub2e4. \uc77c\ubc18 Convolution \uc5f0\uc0b0\uc740 feature aggregation\uacfc transformation (by activation function)\uc744 \uc5f0\ub2ec\uc544\uc11c \ucc98\ub9ac\ud558\ub294 \uac8c \uc77c\ubc18\uc801\uc785\ub2c8\ub2e4. \ub17c\ubb38\uc5d0\uc11c\ub294 Self-Attention\uc744 \uc0ac\uc6a9\ud558\uc5ec feature aggregation\uacfc transformation\uc744 \ubcc4\ub3c4\ub85c \uc218\ud589\ud558\uba70, transformation\uc5d0\ub294 element-wise perceptron layer\uac00 \uc0ac\uc6a9\ub429\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Visual_Transformer\/11.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub17c\ubb38\uc5d0\uc11c\ub294 feature aggregation\uc5d0 Pairwise Self-Attention\uacfc Patch-wise Self-Attention, 2\uac1c\uc758 Self-Attention \uae30\ubc95\uc744 \uc0ac\uc6a9\ud569\ub2c8\ub2e4. \ub450 \uc5f0\uc0b0 \ubaa8\ub450 spatial\uacfc channel dimension\uc5d0 \ub300\ud55c weight\ub97c \ud559\uc2b5\ud558\ub294 <strong>Vector Attention<\/strong> \uc744 \uc0ac\uc6a9\ud569\ub2c8\ub2e4. \uc774\ub807\uac8c \uad6c\uc131\ud55c Self-Attention Networks (SAN)\uc744 \ud1b5\ud574 \ub354 \uc801\uc740 \uc218\uc758 parameter\ub85c ImageNet \ub370\uc774\ud130 \uc14b\uc5d0\uc11c ResNet\ubcf4\ub2e4 \uc6b0\uc218\ud55c \uc131\ub2a5\uc744 \uac70\ub458 \uc218 \uc788\uc5c8\ub2e4\uace0 \ud569\ub2c8\ub2e4. \ub610\ud55c adversarial perturbation\uc5d0 robust\ud574\uc9c0\ub294 \ud6a8\uacfc\ub3c4 \uc5bb\uc744 \uc218 \uc788\uace0 test image\uc5d0 unseen transformations\uc774 \uc801\uc6a9\ub420 \ub54c\uc5d0\ub3c4 \uc77c\ubc18\ud654 \uc131\ub2a5\uc774 \uc88b\uc544\uc9c0\ub294 \ud6a8\uacfc\ub97c \uc5bb\uc744 \uc218 \uc788\uc5c8\ub2e4\uace0 \ud569\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"vision-transformer-vit\">Vision Transformer (ViT)<\/h3>\n<p>\ub2e4\uc74c\uc740 \ucc98\uc74c\uc73c\ub85c Large-scale \ucef4\ud4e8\ud130 \ube44\uc804 \ub370\uc774\ud130 \uc14b\uc5d0\uc11c CNN\uc5d0 \uacac\uc904 \ub9cc\ud55c \uc131\ub2a5\uc744 \ubcf4\uc5ec\uc900 Vision Transformer (ViT) \uc785\ub2c8\ub2e4. 2020\ub144 10\uc6d4 \ubc1c\ud45c\ub41c \ub530\ub048 \ub530\ub048\ud55c \ub17c\ubb38\uc774\uace0 \uc81c\ubaa9\uc740 <a href=\"https:\/\/arxiv.org\/abs\/2010.11929\" target=\"_blank\"><b> \u201cAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale\u201d <\/b><\/a> \uc785\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Visual_Transformer\/12.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>Transformer\ub97c \ud65c\uc6a9\ud558\uae30 \uc704\ud574 Input Image\ub97c \uc5ec\ub7ec \uac1c\uc758 patch\ub85c \ucabc\uac1c\uc11c CNN (ResNet)\uc5d0 \ub123\uc5b4\uc11c feature map\uc744 \ubf51\uc544\ub0b8 \ub4a4 flatten \uc2dc\ucf1c\uc11c Transformer encoder\uc5d0 \ub123\uc5b4\uc90d\ub2c8\ub2e4. \uadf8 \ub4a4 Classifier\ub97c \ubd99\uc5ec\uc11c \ud559\uc2b5\uc744 \uc2dc\ud0b5\ub2c8\ub2e4.<\/p>\n\n<p>\uc5ec\uae30\uc11c \uc911\uc694\ud55c \uc810\uc774 \uc788\ub294\ub370, Transformer \uae30\ubc18\uc758 \ubc29\ubc95\ub4e4\uc740 \ubb34\uc218\ud788 \ub9ce\uc740 \uc591\uc758 \ub370\uc774\ud130 \uc14b\uc73c\ub85c pre-training\uc744 \uc2dc\ud0a8 \ub4a4 downstream task (e.g. ImageNet)\uc5d0 fine-tuning\uc744 \uc2dc\ucf1c\uc57c \uc88b\uc740 \uc131\ub2a5\uc774 \ubcf4\uc7a5\ub429\ub2c8\ub2e4. \ud558\uc9c0\ub9cc \uc2e4\ud5d8\uc5d0\uc11c \uc0ac\uc6a9\ud55c \ub300\uc6a9\ub7c9\uc758 \ub370\uc774\ud130\uc14b\uc740 Google \ub0b4\ubd80\uc5d0\uc11c\ub9cc \uc0ac\uc6a9\ud558\uace0 \uc788\ub294 300 million image \ub370\uc774\ud130 \uc14b\uc778 JFT-300M\uc774\ub77c Google\uc774 \uc544\ub2cc \uc5f0\uad6c \uc9d1\ub2e8\uc5d0\uc11c\ub294 \uac19\uc740 \ubc29\ubc95\uc744 \uc801\uc6a9\ud574\ub3c4 \uc88b\uc740 \uc131\ub2a5\uc774 \ub098\uc62c \uc218 \uc5c6\ub2e4\ub294 \ub73b\uc785\ub2c8\ub2e4.<\/p>\n\n<p>CNN\uacfc Transformer\ub97c \ube44\uad50\ud574\ubcf4\uba74, CNN\uc740 translation equivariance \ub4f1 inductive bias\uac00 \ub9ce\uc774 \ub4e4\uc5b4\uac00 \uc788\ub294 \ubaa8\ub378\uc774\ub77c \ube44\uad50\uc801 \uc801\uc740 \uc218\uc758 \ub370\uc774\ud130\ub85c\ub3c4 \uc5b4\ub290\uc815\ub3c4 \uc131\ub2a5\uc774 \ubcf4\uc7a5\uc774 \ub418\ub294 \ubc18\uba74, Transformer\ub294 inductive bias\uac00 \uac70\uc758 \uc5c6\ub294 \ubaa8\ub378\uc774\ub77c \ub9ce\uc740 \uc218\uc758 \ub370\uc774\ud130\uac00 \uc788\uc5b4\uc57c \uc131\ub2a5\uc774 \ud5a5\uc0c1\ub429\ub2c8\ub2e4. \uc774 \uc810\uc774 Transformer\uc758 \uc7a5\uc810\uc774\uc790 \ub2e8\uc810\uc774 \ub420 \uc218 \uc788\ub294 \ubd80\ubd84\uc778\ub370 Google\uc5d0\uc11c\ub294 \ub9ce\uc740 \uc218\uc758 \ub370\uc774\ud130\ub97c \ud1b5\ud574 \uc7a5\uc810\uc73c\ub85c \uc2b9\ud654\uc2dc\ud0a8 \uc810\uc774 \uc778\uc0c1\uae4a\uc9c0\ub9cc \ub9ce\uc740 \uc218\uc758 \ub370\uc774\ud130\ub97c \ud655\ubcf4\ud558\uae30 \uc5b4\ub824\uc6b4 \ubd84\uc57c\uc5d0\uc11c\ub294 \uc801\uc6a9\ud558\uae30 \uc5b4\ub835\ub2e4\ub294 \ub2e8\uc810\ub3c4 \uc798 \ubcf4\uc5ec\uc8fc\ub294 \uac83 \uac19\uc2b5\ub2c8\ub2e4.\n\uc81c\uac00 \ucc38\uc5ec\ud588\uc5c8\ub358 TensorFlow KR \ub17c\ubb38 \uc77d\uae30 \ubaa8\uc784 PR-12\uc758 \uc774\uc724\uc131\ub2d8\uaed8\uc11c \uc774 \ub17c\ubb38\uc744 \ud55c\uae00\ub85c \uc798 \uc124\uba85\ud574\uc8fc\uc2e0 \uc601\uc0c1 \uc790\ub8cc\uac00 \uc788\uc5b4\uc11c \uac19\uc774 \ucc38\uace0\ud558\uc2dc\uba74 \uc88b\uc744 \uac83 \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/youtu.be\/D72_Cn-XV1g\" target=\"_blank\"><b> \u201cPR-281: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale\u201d <\/b><\/a><\/li>\n<\/ul>\n\n<h3 id=\"data-efficient-image-transformer-deit\">Data-efficient Image Transformer (DeiT)<\/h3>\n<p>\ub9c8\uc9c0\ub9c9\uc740 2020\ub144 12\uc6d4 \ubc1c\ud45c\ub41c <a href=\"https:\/\/arxiv.org\/abs\/2012.12877\" target=\"_blank\"><b> \u201cTraining data-efficient image transformers &amp; distillation through attention\u201d <\/b><\/a> \uc774\uba70 \uacf5\uac1c\ub41c\uc9c0 \ub450 \ub2ec\uc774 \ucc44 \ub418\uc9c0 \uc54a\uc740 \uc544\uc8fc \ub530\ub048\ub530\ub048\ud55c \ub17c\ubb38\uc785\ub2c8\ub2e4. ViT\ub294 Google\uc5d0\uc11c \ubc1c\ud45c\ud55c \ub17c\ubb38\uc774\uc5c8\ub2e4\uba74 DeiT\ub294 Facebook\uc5d0\uc11c \ubc1c\ud45c\ud55c \ub17c\ubb38\uc785\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Visual_Transformer\/14.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc704\uc758 \uadf8\ub9bc\uc5d0\uc11c \uc54c \uc218 \uc788\ub4ef\uc774 \uae30\uc874\uc758 ViT\ub294 \ubb3c\ub860\uc774\uace0 AutoML\ub85c \ucc3e\uc740 ImageNet\uc5d0 \ucd5c\uc801\ud654 \ub41c CNN architecture\uc778 EfficientNet\ubcf4\ub2e4\ub3c4 \ub354 \uc6b0\uc218\ud55c \uc131\ub2a5\uc744 \ubcf4\uc5ec\uc8fc\ub294 \uacb0\uacfc\ub97c \ubc1c\ud45c\ud558\uc5ec \ud070 \uc774\ubaa9\uc744 \ub04c\uc5c8\uc2b5\ub2c8\ub2e4. \uadf8\ub9bc\uc758 \ube68\uac04 \uc810\uc120\uc73c\ub85c \ub418\uc5b4\uc788\ub294 \uac83\uc740 \ub17c\ubb38\uc5d0\uc11c \uc81c\uc548\ud55c transformer-specific distillation \uae30\ubc95\uc744 \uc801\uc6a9\ud558\uc600\uc744 \ub54c\uc758 \uc131\ub2a5\uc744 \uc758\ubbf8\ud569\ub2c8\ub2e4.<\/p>\n\n<p>Transformer \uacc4\uc5f4 \ubc29\ubc95\uc73c\ub85c CNN\uc5d0 \uacac\uc904\ub9cc\ud55c \uc131\ub2a5\uc744 \ub0b8 \uc810\ub3c4 \uad49\uc7a5\ud788 \uc778\uc0c1 \uae4a\uc740\ub370 \ub354\uc6b1 \ub180\ub77c\uc6b4\uac74 ViT\uc758 \ud575\uc2ec \uc694\uc18c\uc600\ub358 JFT \uc640 \uac19\uc740 \uc5b4\ub9c8\uc5b4\ub9c8\ud55c \uaddc\ubaa8\uc758 \ub370\uc774\ud130\uc14b\uc73c\ub85c Pre-training\uc744 \uc2dc\ud0a4\ub294 \uacfc\uc815 \uc5c6\uc774 \uc88b\uc740 \uc131\ub2a5\uc744 \ub2ec\uc131\ud558\uc600\ub2e4\ub294 \uc810\uc785\ub2c8\ub2e4. \uc989, \ubb34\uc218\ud788 \ub9ce\uc740 \ub370\uc774\ud130\ub97c \uad6c\ucd95\ud558\uc9c0 \uc54a\uc544\ub3c4 \ub418\uace0, \uad49\uc7a5\ud788 \uae34 \uc2dc\uac04\uacfc \ube44\uc6a9\uc774 \uc18c\uc694\ub418\ub294 pre-training\uc774 \uc5c6\uc5b4\ub3c4 \ub41c\ub2e4\ub294 \ub73b\uc774\ub2c8 \ud6e8\uc52c \ubc94\uc6a9\uc131\uc774 \uc788\ub294 \ubaa8\ub378\uc744 \ub9cc\ub4e0 \uc148\uc785\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/Visual_Transformer\/15.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub300\ubd80\ubd84\uc758 \uad6c\uc870\ub294 ViT\uc758 Vision Transformer\uc5d0\uc11c \ucd9c\ubc1c\ud558\uc600\uace0 \ud070 \ubcc0\ud654\ub294 \uc5c6\uc2b5\ub2c8\ub2e4. \uc131\ub2a5 \ud5a5\uc0c1\uc5d0 \ud070 \uae30\uc5ec\ub97c \ud55c \uccab\ubc88\uc9f8 \uc694\uc778\uc740 CNN\uc5d0\uc11c \uc131\ub2a5 \ud5a5\uc0c1\uc5d0 \ud070 \uae30\uc5ec\ub97c \ud588\uc5c8\ub358 data augmentation, optimization, regularization \ub4f1\uc758 \uae30\ubc95\uc744 \uc801\uc808\ud558\uac8c \uc801\uc6a9\ud55c \uc810\uc785\ub2c8\ub2e4. \ub450\ubc88\uc9f8 \uc694\uc778\uc740 knowledge distillation\uc774\uba70 CNN\uc5d0\uc11c \uad04\ubaa9\ud560 \ub9cc\ud55c \uc131\uacfc\ub97c \ubcf4\uc5ec\uc900 RegNet\uc744 teacher model\ub85c \uc0ac\uc6a9\ud558\uc5ec teacher model\uc758 output\uc744 \ud65c\uc6a9\ud558\uace0, class token\uc5d0 distillation token\uc744 \ucd94\uac00\ud558\uc5ec \ubaa8\ub378\uc744 \ud6a8\uacfc\uc801\uc73c\ub85c \ud559\uc2b5\uc2dc\ud0a4\ub294 \ub370 \uc131\uacf5\ud569\ub2c8\ub2e4.<\/p>\n\n<p>\uc81c\uac00 \ucc38\uc5ec\ud588\uc5c8\ub358 TensorFlow KR \ub17c\ubb38 \uc77d\uae30 \ubaa8\uc784 PR-12\uc758 \uc774\uc9c4\uc6d0\ub2d8\uaed8\uc11c \uc774 \ub17c\ubb38\uc744 \ud55c\uae00\ub85c \uc798 \uc124\uba85\ud574\uc8fc\uc2e0 \uc601\uc0c1 \uc790\ub8cc\uac00 \uc788\uc5b4\uc11c \uac19\uc774 \ucc38\uace0\ud558\uc2dc\uba74 \uc88b\uc744 \uac83 \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/youtu.be\/DjEvzeiWBTo\" target=\"_blank\"><b> \u201cPR-297: Training Data-efficient Image Transformers &amp; Distillation through Attention (DeiT)\u201d <\/b><\/a><\/li>\n<\/ul>\n\n<blockquote> \uacb0\ub860 <\/blockquote>\n<p>\uc624\ub298\uc740 \uc790\uc5f0\uc5b4 \ucc98\ub9ac\uc5d0\uc11c \uc2dc\uc791\ud574\uc11c \uc774\uc81c\ub294 \ub2e4\uc591\ud55c \ubd84\uc57c\uc5d0\uc11c \uc6b0\uc218\ud55c \uc131\ub2a5\uc744 \ubcf4\uc5ec\uc8fc\uace0 \uc788\ub294 Transformer\uc5d0 \ub300\ud574 \uac04\ub2e8\ud788 \uc54c\uc544\ubcf4\uace0, \ucef4\ud4e8\ud130 \ube44\uc804\uc758 \uac00\uc7a5 \ub300\ud45c\uc801\uc778 \ubd84\uc57c\uc778 Image Recognition\uc5d0 Self-Attention\uacfc Transformer\uac00 \uc801\uc6a9\ub41c \uc0ac\ub840\ub4e4\uc744 \uc54c\uc544\ubd24\uc2b5\ub2c8\ub2e4. \ub2e4\uc74c \ud3b8\uc5d0\uc11c\ub294 Object Detection, Segmentation \ub4f1\uc5d0 Transformer\ub97c \uc801\uc6a9\ud55c \uc0ac\ub840\ub4e4\uc744 \uc18c\uac1c\ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n","<p>\uc548\ub155\ud558\uc138\uc694, \uc624\ub298\uc740 \uc790\uc5f0\uc5b4 \ucc98\ub9ac(NLP)\uc5d0\uc11c \uc555\ub3c4\uc801\uc778 \uc131\ub2a5\uc744 \ubcf4\uc5ec\uc8fc\uba70 \uc8fc\ub958\ub85c \uc790\ub9ac\uc7a1\uc740 Transformers \ubaa8\ub378\uc744 \ucef4\ud4e8\ud130 \ube44\uc804\uc5d0 \uc801\uc6a9\ud558\ub824\ub294 \uc2dc\ub3c4\ub4e4\uc744 \uc815\ub9ac\ud55c \uc11c\ubca0\uc774 \ub17c\ubb38\uc778 <a href=\"https:\/\/arxiv.org\/abs\/2101.01169\" target=\"_blank\"><b> \u201cTransformers in Vision: A Survey\u201d <\/b><\/a> \ub97c \uc77d\uace0 \uac04\ub2e8\ud788 \uc815\ub9ac\ud574\ubcfc \uc608\uc815\uc785\ub2c8\ub2e4. \ub17c\ubb38\uc758 \ubd84\ub7c9\uc774 \ub9ce\uc740 \ub9cc\ud07c \uc5ec\ub7ec \ud3b8\uc5d0 \uac78\uccd0\uc11c \uc18c\uac1c\ub4dc\ub9b4 \uc608\uc815\uc774\uba70, 1\ud3b8\uc5d0\uc11c\ub294 Transformer\uc5d0 \ub300\ud574 \uac04\ub2e8\ud788 \uc18c\uac1c \ub4dc\ub9ac\uace0, Transformer\ub97c Image Recognition\uc5d0 \uc801\uc6a9\ud55c \ub300\ud45c \uc5f0\uad6c\ub4e4\uc744 \uc18c\uac1c\ub4dc\ub9b4 \uc608\uc815\uc785\ub2c8\ub2e4.<\/p>\n\n"],"pubDate":"Thu, 18 Feb 2021 00:00:00 +0000","link":"https:\/\/hoya012.github.io\/\/blog\/Vision-Transformer-1\/","guid":"https:\/\/hoya012.github.io\/\/blog\/Vision-Transformer-1\/"},{"title":"Hoya012 Blog 2020\ub144\uc744 \ub9c8\uce58\uba70!","description":["<p>\uc548\ub155\ud558\uc138\uc694, \uc791\ub144 12\uc6d4 \uc791\uc131\ud588\ub358 <a href=\"https:\/\/hoya012.github.io\/blog\/hoya012-2019-review\/\" target=\"_blank\"><b> 2019\ub144 \ube14\ub85c\uadf8 \uc18c\ud68c <\/b><\/a> \uae00\uc5d0 \uc774\uc5b4\uc11c \uc62c\ud574\ub3c4 2020\ub144\uc744 \ub9c8\ubb34\ub9ac\ud558\uba70 1\ub144\uac04 \uc81c \ube14\ub85c\uadf8\uc5d0 \uc791\uc131\ud588\ub358 \uae00\ub4e4\uc5d0 \ub300\ud55c \uc18c\ud68c\uc640 \uac04\ub2e8\ud55c \ubd84\uc11d\uc744 \ud574\ubcf4\uc558\uc2b5\ub2c8\ub2e4. \uc62c \ud55c\ud574\ub3c4 \uc81c \ube14\ub85c\uadf8\ub97c \ucc3e\uc544\uc640\uc8fc\uc2e0 \ub9ce\uc740 \ubd84\ub4e4 \uc9c4\uc2ec\uc73c\ub85c \uac10\uc0ac\ub4dc\ub9bd\ub2c8\ub2e4.<\/p>\n\n<blockquote> 2020\ub144 \uc791\uc131\ud55c \ube14\ub85c\uadf8 \uae00 24\ud3b8! <\/blockquote>\n<p>\uc6b0\uc120 \uc791\ub144\uae4c\uc9c0 \ucd1d 33\ud3b8\uc758 \uae00\uc744 \uc791\uc131\ud588\uc5c8\ub294\ub370 \uc62c\ud574 24\ud3b8\uc744 \ucd94\uac00\ud558\uace0, \uc774 \uae00\uae4c\uc9c0 \ud3ec\ud568\ud558\uba74 \ucd1d 58\ud3b8\uc758 \uae00\uc744 \uc791\uc131\ud558\uac8c \ub418\uc5c8\ub124\uc694! \uc62c\ud574 \ubaa9\ud45c\uac00 50\ud3b8\uc744 \ucc44\uc6b0\ub294 \uac83\uc774\uc5c8\ub294\ub370 \ubaa9\ud45c\ub97c \ucd08\uacfc \ub2ec\uc131\ud574\uc11c \ubfcc\ub4ef\ud569\ub2c8\ub2e4.<\/p>\n\n<p>\uc791\ub144\uae4c\uc9c0\ub294 \uc8fc\ub85c Computer Vision \ubd84\uc57c\uc5d0 \ub300\ud55c \uc774\ub860\uc744 \ub2e4\ub8ec \ub17c\ubb38 \uc18c\uac1c, \ud559\ud68c \ubbf8\ub9ac\ubcf4\uae30 \ub4f1\uc758 Research\uc5d0 \uc9d1\uc911\ud55c \uae00\uc744 \uc8fc\ub85c \uc791\uc131\ud574\uc654\uc5c8\ub294\ub370, \uc62c\ud574 \ucd08 \ube14\ub85c\uadf8\uc758 \uc131\uc7a5\uacfc \ub2e4\uc591\uc131\uc744 \ud0a4\uc6cc\uc8fc\uae30 \uc704\ud574 \uc2dc\uc57c\ub97c \ub113\ud788\uac8c \ub418\uc5c8\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/2020_review\/1.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc6b0\uc120 \uac00\uc7a5 \ud070 \ubcc0\ud654\ub294 Deep Learning \ud504\ub85c\uadf8\ub798\ubc0d\uc744 \ub3c4\uc640\uc8fc\ub294 \uc8fc\uc694 Library\uc5d0 \ub300\ud55c \uc18c\uac1c, \uc8fc\uc694 \ubc29\ubc95\ub860\uc744 \uad6c\ud604\ud558\uc5ec Tutorial \ud615\ud0dc\ub85c \ucf54\ub4dc\uc640 \uae00\uc744 \uac19\uc774 \uc791\uc131\ud558\ub294 \ub4f1 Engineering \uc131\uaca9\uc758 \uae00\ub3c4 \uc791\uc131\uc744 \ud558\uae30 \uc2dc\uc791\ud55c \uc810\uc785\ub2c8\ub2e4.<\/p>\n\n<p>\uadf8\ub9ac\uace0 \uc62c\ud574 \uac00\uc7a5 \ub9ce\uc740 \uc2dc\uac04\uc744 \uc3df\uc558\ub358 \u201cDeep Learning Classification Guidebook\u201d \uc2dc\ub9ac\uc988\ub294 2018\ub144 \ub9d0\ubd80\ud130 2019\ub144 \ucd08\uae4c\uc9c0 \ud65c\ubc1c\ud558\uac8c \uc5f0\uc7ac\ud588\ub358 \u201cTutorials of Object Detection using Deep Learning\u201d \uc640 \uac19\uc774 Classification\uc758 \uae30\ubcf8\uc744 \uacf5\ubd80\ud558\uc2dc\ub294 \ubd84\ub4e4\uc5d0\uac8c \ub3c4\uc6c0\uc774 \ub420 \uc218 \uc788\ub3c4\ub85d 4\ud3b8\uc5d0 \uac78\uccd0\uc11c \uc8fc\uc694 CNN architecture\ub4e4\uc744 \uc18c\uac1c\ub4dc\ub838\uc2b5\ub2c8\ub2e4. \uc774 CNN architecture \uc2dc\ub9ac\uc988 \ub355\ubd84\uc5d0 \uc804\uc561 \uae30\ubd80 \ud589\uc0ac\uc5d0\uc11c \uc5f0\uc0ac\ub85c \ucc38\uc5ec\ud558\uc5ec \ubc1c\ud45c\ub97c \ud558\uae30\ub3c4 \ud558\uc600\uc73c\uba70, \ubc1c\ud45c\uc790\ub8cc\ub294 <a href=\"https:\/\/www.slideshare.net\/HoseongLee6\/cnn-architecture-a-to-z\" target=\"_blank\"><b> Slideshare <\/b><\/a>\ub97c \ud1b5\ud574 \uacf5\uc720\ud55c \ubc14 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\ub610\ud55c \uc81c\uac00 \ubcf8\uc5c5\uc73c\ub85c \ub2e4\ub8e8\uace0 \uc788\ub294 Anomaly Detection\uc5d0 \ub300\ud55c \uc18c\uac1c\ub3c4 2\ud3b8\uc5d0 \uac78\uccd0 \uc18c\uac1c\ub97c \ub4dc\ub9b4 \uc218 \uc788\uc5b4\uc11c \uae30\ubee4\uc2b5\ub2c8\ub2e4.<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/hoya012.github.io\/blog\/anomaly-detection-overview-1\/\" target=\"_blank\"><b> Anomaly Detection \uac1c\uc694\uff1a [1] \uc774\uc0c1\uce58 \ud0d0\uc9c0 \ubd84\uc57c\uc5d0 \ub300\ud55c \uc18c\uac1c \ubc0f \uc8fc\uc694 \ubb38\uc81c\uc640 \ud575\uc2ec \uc6a9\uc5b4, \uc0b0\uc5c5 \ud604\uc7a5 \uc801\uc6a9 \uc0ac\ub840 \uc815\ub9ac <\/b><\/a><\/li>\n  <li><a href=\"https:\/\/hoya012.github.io\/blog\/anomaly-detection-overview-2\/\" target=\"_blank\"><b> Anomaly Detection \uac1c\uc694\uff1a [2] Out-of-distribution(OOD) Detection \ubb38\uc81c \uc18c\uac1c \ubc0f \ud575\uc2ec \ub17c\ubb38 \ub9ac\ubdf0<\/b><\/a><\/li>\n<\/ul>\n\n<p>\ub9c8\uc9c0\ub9c9\uc73c\ub85c \uc62c\ud574 hoya012 blog\uc5d0 research\uac00 \uc544\ub2cc \uc77c\uc0c1\uc801\uc778 \uc8fc\uc81c\ub97c \ub2f4\uc740 \uae00\ub3c4 \ucc28\uadfc\ucc28\uadfc \uc791\uc131\ud574\ubcf4\ub824\uace0 \uacc4\ud68d\uc744 \uc138\uc6e0\uc5c8\ub294\ub370 \uc774 \uacc4\ud68d\uc740 \ud544\uc790\uc758 \uac8c\uc73c\ub984\uc73c\ub85c \uc778\ud574 \ub2e8 1\ud3b8 \ubc16\uc5d0 \uc791\uc131\ud558\uc9c0 \ubabb\ud574\uc11c \uc720\uc77c\ud55c \ubc18\uc131\uac70\ub9ac\uc778 \uac83 \uac19\uc2b5\ub2c8\ub2e4. \uc62c\ud574 \uccab \uc0bd\uc744 \ub72c \uae00\uc774 \ubc14\ub85c <a href=\"https:\/\/hoya012.github.io\/blog\/bartender_certificate\/\" target=\"_blank\"><b> \ub3c5\ud559\uc73c\ub85c \uc870\uc8fc\uae30\ub2a5\uc0ac(\ubc14\ud150\ub354) \uc790\uaca9\uc99d \ub530\uae30! [\ud569\uaca9 \ud6c4\uae30] <\/b><\/a> \uae00 \uc785\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/2020_review\/2.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ubc14\ud150\ub354 \uc790\uaca9\uc99d\uc5d0 \ub300\ud55c \uc18c\uac1c\uc640, \ub3c4\uc804\ud558\uba74\uc11c \uacaa\uc5c8\ub358 \uacfc\uc815\ub4e4\uc744 \uae00\ub85c \uc815\ub9ac\ud574\ubd24\ub294\ub370, \ud3c9\uc18c\uc5d0 \ub17c\ubb38\uc744 \uc77d\uace0 \uc815\ub9ac\ud574\uc11c \uae00\uc744 \uc791\uc131\ud558\ub294 \uac83\ubcf4\ub2e4 \uc2dc\uac04\uc774 \uba87 \ubc30\ub294 \ub9ce\uc774 \ub4e4\uc5c8\uc2b5\ub2c8\ub2e4. \uc5ed\uc2dc \uc0ac\ub78c\uc740 \uc775\uc219\ud55c \uc77c\uc744 \ud560 \ub54c\uc640 \uadf8\ub807\uc9c0 \uc54a\uc740 \uc77c\uc744 \ud560 \ub54c \ub2a5\ub960\uc774 \ub108\ubb34 \ub2e4\ub978 \uac83 \uac19\uc2b5\ub2c8\ub2e4. \ud5c8\ud5c8\ud5c8.. \uc9d1\uc5d0\uc11c \uc190\uc27d\uac8c \uce75\ud14c\uc77c\uc744 \ub9cc\ub4e4\uc5b4 \ub9c8\uc2e4 \uc218 \uc788\ub294 \ud648\ud150\ub529 \uad00\ub828 \uae00\uc740 \ub0b4\ub144\uc5d0\ub294 \uaf2d \uc791\uc131\ud558\ub294\uac8c \ubaa9\ud45c\ub78d\ub2c8\ub2e4! \u314b\u314b\u314b<\/p>\n\n<blockquote> 2020\ub144\uc758 \uc131\uc7a5 \uacfc\uc815 <\/blockquote>\n\n<p>\uc62c\ud574 \uc81c \ube14\ub85c\uadf8\uc758 \uc131\uc7a5\uc744 \uc0b4\ud3b4\ubcf4\uae30 \uc704\ud574 \uc791\ub144\uacfc \ub9c8\ucc2c\uac00\uc9c0\ub85c <a href=\"https:\/\/analytics.google.com\/\" target=\"_blank\"><b> \uad6c\uae00 \uc560\ub110\ub9ac\ud2f1\uc2a4<\/b><\/a>\ub97c \ud65c\uc6a9\ud588\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/2020_review\/3.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc791\ub144\uacfc \ube44\uad50\ud558\uc600\uc744 \ub54c \uc0ac\uc6a9\uc790 \uc218\uac00 \uac70\uc758 3\ubc30\ub85c \ub298\uc5c8\ub2e4\ub294 \uc810\uc774 \uac00\uc7a5 \uace0\ubb34\uc801\uc774\uba70, \uc9c0\ub09c 3\uc6d4 \ucee4\ubba4\ub2c8\ud2f0\ub97c \ub728\uac81\uac8c \ub2ec\uad9c\ub358 <a href=\"https:\/\/hoya012.github.io\/blog\/automl-zero-review\/\" target=\"_blank\"><b> AutoML-Zero\uff1aEvolving Machine Learning Algorithms From Scratch Review <\/b><\/a> \uae00\uc5d0\uc11c \uc77c\uc77c \ubc29\ubb38\uc790\uc218 \uc2e0\uae30\ub85d\uc744 \ub2ec\uc131\ud558\ub294 \ucf8c\uac70\ub97c \uc774\ub8e8\uc5c8\uc2b5\ub2c8\ub2e4. \uadf8 \ub4a4\ub85c \uafb8\uc900\ud788 PyTorch Tutorial \uae00\ub4e4\uc744 \uc790\uc8fc \uc62c\ub824\uc11c \uadf8\ub7f0\uc9c0 7\uc6d4\uc744 \uae30\uc810\uc73c\ub85c \uafb8\uc900\ud788 \uc8fc\ub9d0\uc744 \uc81c\uc678\ud558\uba74 \ub9e4\uc77c 500\uba85 \uc774\uc0c1 \ucc3e\uc544\uc624\ub294 \ud750\ub984\uc744 \ud0c0\uac8c \ub418\uc5c8\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/2020_review\/4.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub2e4\uc74c\uc740 \uc81c \ube14\ub85c\uadf8 \ubc29\ubb38\uc790 \uc720\uc785 \ud2b8\ub798\ud53d\uc5d0 \ub300\ud55c \ubd84\uc11d\uc785\ub2c8\ub2e4. \uc791\ub144\uae4c\uc9c0\ub294 \uc81c \uae00\uc744 Facebook \uadf8\ub8f9\uc5d0 \ud64d\ubcf4\ub97c \ud574\uc11c \uc720\uc785\ub418\ub294 <strong>Social<\/strong> \uc720\uc785\uc774 \ub192\uc740 \ube44\uc728\uc744 \ucc28\uc9c0\ud588\uc5c8\ub294\ub370, \uc62c\ud574\ub294 \uc81c\uac00 \ubc14\ub77c\ub358 \ub300\ub85c \uc0ac\ub78c\ub4e4\uc774 Google \ub4f1 \uac80\uc0c9 \uc5d4\uc9c4\uc744 \ud1b5\ud574 \uac80\uc0c9\ud558\uc5ec \uc720\uc785\ub418\ub294 <strong>Organic Search<\/strong> \uc774 75%\uae4c\uc9c0 \uc62c\ub77c\uc654\uc2b5\ub2c8\ub2e4. \uc62c\ud574 \ucc3e\uc544\uc640\uc8fc\uc2e0 86,005\ubd84\uc758 \ub3c5\uc790\ubd84\ub4e4 \ubaa8\ub450 \ud658\uc601\ud569\ub2c8\ub2e4!<\/p>\n\n<blockquote> \uc870\ud68c\uc218 Top 5 \uae00 &amp; \uc8fc\uc694 \uc720\uc785 \ud0a4\uc6cc\ub4dc \ubd84\uc11d <\/blockquote>\n\n<p>\ub9c8\uc9c0\ub9c9\uc73c\ub85c 2020\ub144 \ud55c \ud574\ub3d9\uc548 \uac00\uc7a5 \uc778\uae30\ub97c \ub04c\uc5c8\ub358 \uc0c1\uc704 5\uac1c\uc758 \uae00\uc744 \uc18c\uac1c\ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4. \uac8c\uc2dc\uae00\uc758 \uc870\ud68c\uc218\ub294 Google Analytics [\ud589\ub3d9] \u2013 [\uc0ac\uc774\ud2b8 \ucf58\ud150\uce20] \u2013 [\ubaa8\ub4e0 \ud398\uc774\uc9c0] \ud0ed\uc744 \ud1b5\ud574 \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<ul>\n  <li>5\uc704: <a href=\"https:\/\/hoya012.github.io\/blog\/SIngle-Image-Super-Resolution-Overview\/\" target=\"_blank\"><b> \u201cSingle Image Super Resolution using Deep Learning Overview\u201d <\/b><\/a> : 23,033\ud68c<\/li>\n  <li>4\uc704: <a href=\"https:\/\/hoya012.github.io\/blog\/yolov4\/\" target=\"_blank\"><b> \u201cYOLOv4\uff1aOptimal Speed and Accuracy of Object Detection Review\u201d <\/b><\/a> : 24,634\ud68c<\/li>\n  <li>3\uc704: <a href=\"https:\/\/hoya012.github.io\/blog\/deeplearning-classification-guidebook-1\/\" target=\"_blank\"><b> \u201cDeep Learning Image Classification Guidebook [1] LeNet, AlexNet, ZFNet, VGG, GoogLeNet, ResNet\u201d <\/b><\/a> : 25,073\ud68c<\/li>\n  <li>2\uc704: <a href=\"https:\/\/hoya012.github.io\/blog\/anomaly-detection-overview-1\/\" target=\"_blank\"><b> \u201cAnomaly Detection \uac1c\uc694\uff1a [1] \uc774\uc0c1\uce58 \ud0d0\uc9c0 \ubd84\uc57c\uc5d0 \ub300\ud55c \uc18c\uac1c \ubc0f \uc8fc\uc694 \ubb38\uc81c\uc640 \ud575\uc2ec \uc6a9\uc5b4, \uc0b0\uc5c5 \ud604\uc7a5 \uc801\uc6a9 \uc0ac\ub840 \uc815\ub9ac\u201d <\/b><\/a> : 26,225\ud68c<\/li>\n  <li>1\uc704: <a href=\"https:\/\/hoya012.github.io\/blog\/EfficientNet-review\/\" target=\"_blank\"><b> \u201cEfficientNet\uff1a Rethinking Model Scaling for Convolutional Neural Networks \ub9ac\ubdf0\u201d <\/b><\/a> : 30,348\ud68c<\/li>\n<\/ul>\n\n<p>\uc5ed\uc2dc\ub098 \ud55c \ubd84\uc57c\uc5d0 \uac1c\uc694\ub97c \ub2e4\ub8ec \uc8fc\uc81c\ub4e4\uc774 \ub300\uccb4\ub85c \ub192\uc740 \uc870\ud68c\uc218\ub97c \ubcf4\uc600\uace0, Computer Vision \ucabd\uc5d0\uc11c\ub294 \ud544\uc218\ub85c \uc77d\uc5b4\uc57c \ud558\ub294 \ub17c\ubb38\uc778 YOLO\uc640 Efficientnet \ub3c4 \ud070 \uc778\uae30\ub97c \ub04c\uc5c8\uc2b5\ub2c8\ub2e4. \ube44\uc2b7\ud55c \uc774\uc720\ub85c 6\uc704\ub294 \uad49\uc7a5\ud788 \ub9ce\uc740 \uc2dc\uac04\uc744 \ub4e4\uc5ec\uc11c \uc791\uc131\ud55c \u201cSelf-Supervised Learning Overview\u201d \uae00\uc774 \ucc28\uc9c0\ud558\uc600\uace0, 7\uc704\ub294 EfficientDet \ub17c\ubb38\uc774 \ucc28\uc9c0\ud558\uc600\ub2f5\ub2c8\ub2e4. \uc804\uccb4 \uc870\ud68c\uc218\ub294 \ucd1d 587,681\ud68c\ub85c \uc791\ub144\uae4c\uc9c0 \ucd1d 18\ub9cc\ud68c \uc600\ub358\uac70\uc5d0 \ube44\ud558\uba74 \uc5c4\uccad\ub09c \uc131\uc7a5\uc744 \uc774\ub918\uc2b5\ub2c8\ub2e4. \ub0b4\ub144\uc5d0\ub294 \uc774\ub9cc\ud07c \uc131\uc7a5\uc744 \uc774\ub8e8\uae30\ub294 \uc27d\uc9c0 \uc54a\uc744 \uac83 \uac19\ub124\uc694!<\/p>\n\n<p>\uc5ec\ub2f4\uc774\uc9c0\ub9cc \uc5f0\uad6c\uc640 \ubb34\uad00\ud55c \uc870\uc8fc\uae30\ub2a5\uc0ac \uc790\uaca9\uc99d \ud569\uaca9 \ud6c4\uae30 \uae00\ub3c4 \uc804\uccb4 58\ud3b8 \uc911\uc5d0 \uc870\ud68c\uc218\ub294 19\ub4f1\uc73c\ub85c \uc0c1\uc704\uad8c\uc5d0 \uc18d\ud558\uace0 \uc788\uc2b5\ub2c8\ub2e4. \u314b\u314b\u314b<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/2020_review\/5.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub9c8\uc9c0\ub9c9\uc73c\ub85c Search Console\uc744 \ud1b5\ud574 \uc81c \ube14\ub85c\uadf8\uc5d0 \uc720\uc785\ub41c \uac80\uc0c9\uc5b4\ub97c \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. EfficientNet, YOLOv4, EfficientDet, DenseNet, ResNet \ub4f1 Computer Vision \ud544\ub3c5 \ub17c\ubb38\ub4e4\uc774 \uc778\uae30\uac00 \ub9ce\uc558\uace0, \uc5f0\uad6c \ubd84\uc57c\uc778 Anomaly Detection, Object Detection, Self-Supervised Learning, Super Resolution\uc73c\ub85c\ub3c4 \uc720\uc785\uc774 \ub9ce\uc774 \ub41c \uc810\uc774 \uc800\ub294 \uad49\uc7a5\ud788 \ub9cc\uc871\uc2a4\ub7fd\uac8c \uc0dd\uac01\ud569\ub2c8\ub2e4. \uc2e4\uc81c\ub85c \ud574\ub2f9 \ud0a4\uc6cc\ub4dc\ub4e4\uc744 \uad6c\uae00\uc5d0 \uac80\uc0c9\ud574\ubcf4\uba74 \uccab \ud398\uc774\uc9c0\uc5d0 \uc81c \ube14\ub85c\uadf8 \uae00\uc774 \ub178\ucd9c\ub418\uace0 \uc788\uc5b4\uc11c \uc2e0\uae30\ud558\uae30\ub3c4 \ud558\uace0 \ubfcc\ub4ef\ud558\uae30\ub3c4 \ud558\ub124\uc694!<\/p>\n\n<blockquote> 2020\ub144\uc744 \ub9c8\uce58\uba70.. <\/blockquote>\n<p>\uc790 \uc774\ub807\uac8c \uc624\ub298\uc740 \uac04\ub2e8\ud558\uac8c 2020\ub144 \uc81c \ube14\ub85c\uadf8\uc5d0\uc11c \uc788\uc5c8\ub358 \uc77c\ub4e4\uc744 \uc815\ub9ac\ud574\ubcf4\uc558\ub294\ub370\uc694, 1\ub144\uc774 \ucc38 \ube60\ub974\ub2e4\ub294 \uc0dd\uac01\uc774 \ub4dc\ub124\uc694. \ub0b4\ub144\uc5d0\ub3c4 \uc62c\ud574 \uc791\uc131\ud588\ub358 \uac83\ucc98\ub7fc \uc5ec\ub7ec\uac00\uc9c0 \uc720\uc775\ud55c \uc8fc\uc81c\uc758 \uae00\ub4e4\ub85c \ucc3e\uc544\ubd50 \uc608\uc815\uc774\ub2c8 \uc55e\uc73c\ub85c\ub3c4 \ub9ce\uc740 \uad00\uc2ec \ubd80\ud0c1\ub4dc\ub9bd\ub2c8\ub2e4! 2021\ub144\uc5d0\ub3c4 \ubaa8\ub450 \ud558\uc2dc\ub294 \uc77c \ub2e4 \uc798 \ud480\ub9ac\uc2dc\uace0 \ud589\ubcf5\ud558\uc2dc\uae38 \uae30\uc6d0\ud558\uaca0\uc2b5\ub2c8\ub2e4. \uac10\uc0ac\ud569\ub2c8\ub2e4!<\/p>\n\n","<p>\uc548\ub155\ud558\uc138\uc694, \uc791\ub144 12\uc6d4 \uc791\uc131\ud588\ub358 <a href=\"https:\/\/hoya012.github.io\/blog\/hoya012-2019-review\/\" target=\"_blank\"><b> 2019\ub144 \ube14\ub85c\uadf8 \uc18c\ud68c <\/b><\/a> \uae00\uc5d0 \uc774\uc5b4\uc11c \uc62c\ud574\ub3c4 2020\ub144\uc744 \ub9c8\ubb34\ub9ac\ud558\uba70 1\ub144\uac04 \uc81c \ube14\ub85c\uadf8\uc5d0 \uc791\uc131\ud588\ub358 \uae00\ub4e4\uc5d0 \ub300\ud55c \uc18c\ud68c\uc640 \uac04\ub2e8\ud55c \ubd84\uc11d\uc744 \ud574\ubcf4\uc558\uc2b5\ub2c8\ub2e4. \uc62c \ud55c\ud574\ub3c4 \uc81c \ube14\ub85c\uadf8\ub97c \ucc3e\uc544\uc640\uc8fc\uc2e0 \ub9ce\uc740 \ubd84\ub4e4 \uc9c4\uc2ec\uc73c\ub85c \uac10\uc0ac\ub4dc\ub9bd\ub2c8\ub2e4.<\/p>\n\n"],"pubDate":"Wed, 30 Dec 2020 00:00:00 +0000","link":"https:\/\/hoya012.github.io\/\/blog\/hoya012-2020-review\/","guid":"https:\/\/hoya012.github.io\/\/blog\/hoya012-2020-review\/"},{"title":"Do Adversarially Robust ImageNet Models Transfer Better? \ub9ac\ubdf0","description":["<p>\uc548\ub155\ud558\uc138\uc694, \uc624\ub298\uc740 2020\ub144 NeurIPS \ud559\ud68c\uc5d0\uc11c \ubc1c\ud45c\ub41c <a href=\"https:\/\/arxiv.org\/pdf\/2007.08489.pdf\" target=\"_blank\"><b> \u201cDo Adversarially Robust ImageNet Models Transfer Better?\u201d <\/b><\/a> \ub17c\ubb38\uc744 \ub9ac\ubdf0\ud560 \uc608\uc815\uc785\ub2c8\ub2e4. Transfer Learning\uc744 \ub2e4\ub8ec \ub17c\ubb38\uc774\uba70 Transfer Learning\uc740 \ub525\ub7ec\ub2dd\uc5d0\uc11c \uad49\uc7a5\ud788 \uc790\uc8fc \uc0ac\uc6a9\ub418\ub294 \ud559\uc2b5 \ubc29\ubc95\uc774\uba70 \ucd5c\uadfc\uc5d0\ub294 \uac70\uc758 default\ub85c \uc0ac\uc6a9\uc774 \ub41c\ub2e4\uace0 \ud574\ub3c4 \uacfc\uc5b8\uc774 \uc544\ub2d9\ub2c8\ub2e4. \ub525\ub7ec\ub2dd\uc744 \uacf5\ubd80\ud574\ubcf4\uc2e0 \ubd84\ub4e4\uc774\ub77c\uba74 \ud544\uc218\uc801\uc73c\ub85c ImageNet Pretrained Model\uc744 \uac00\uc838\uc640\uc11c \uc0c8\ub85c\uc6b4 \ub370\uc774\ud130\uc14b\uc5d0 \ud559\uc2b5\uc744 \uc2dc\ucf1c \ubcf4\uc168\uc744 \uac83\uc785\ub2c8\ub2e4. \uc77c\ubc18\uc801\uc73c\ub85c \uc815\ud655\ub3c4\uac00 \ub192\uc558\ub358 pretrained model\uc5d0\uc11c transfer\ub97c \ud558\uba74 target model\uc5d0\uc11c\ub3c4 \ub192\uc740 \uc815\ud655\ub3c4\ub97c \uc5bb\ub294\ub2e4\uace0 \uc54c\ub824\uc838 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\uc2e4\uc81c\ub85c \uc774\ub7ec\ud55c \uacbd\ud5a5\uc744 \uc2e4\ud5d8\uc801\uc73c\ub85c \ubc1d\ud78c \ub17c\ubb38 \u201cDo Better ImageNet Models Transfer Better?\u201c \uc774 CVPR 2019\uc5d0\uc11c \ubc1c\ud45c \ub418\uc5c8\uc73c\uba70, ImageNet Top-1 Accuracy\uac00 \ub192\uc558\ub358 \ubaa8\ub378\uc77c\uc218\ub85d Transfer Accuracy\ub3c4 \uc0c1\ub300\uc801\uc73c\ub85c \ub192\uc544\uc9c4\ub2e4\ub294 \uacb0\uacfc\ub97c \uc81c\uc2dc\ud558\uc600\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/transfer-robust\/1.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub2e4\ub9cc Source \ub370\uc774\ud130\uc14b(ImageNet)\uc5d0\uc11c \ub192\uc740 \uc815\ud655\ub3c4\ub97c \ub2ec\uc131\ud558\ub294 \uac83\uc774 Transfer Accuracy\ub97c \ub192\uc774\ub294 \uc720\uc77c\ud55c \ubc29\ubc95\uc740 \uc544\ub2d0 \ud150\ub370, \uc544\uc9c1\uae4c\uc9c0 Transfer Accuracy\uc5d0 \uad00\uc5ec\ud558\ub294 \uc694\uc18c\uc5d0 \ub300\ud55c \uc790\uc138\ud55c \ubd84\uc11d\uc774 \uc798 \ub2e4\ub904\uc9c0\uc9c0 \uc54a\uc558\uc2b5\ub2c8\ub2e4. \uc624\ub298 \uc18c\uac1c\ub4dc\ub9b4 \ub17c\ubb38\uc740 Transfer Accuracy\ub97c \ub192\uc774\uae30 \uc704\ud574 Adversarial robustness\ub97c \uace0\ub824\ud574\uc57c \ud568\uc744 \uc81c\uc2dc\ud558\uace0 \uc788\ub294\ub370\uc694, \uc774\uc81c \ub17c\ubb38 \uc124\uba85\uc73c\ub85c \ub118\uc5b4\uac00\uaca0\uc2b5\ub2c8\ub2e4. \n\uc774 \ub17c\ubb38\uc744 \ub2e4\ub8ec \ubc1c\ud45c \uc790\ub8cc\uc640 \uc720\ud29c\ube0c \ubc1c\ud45c \uc601\uc0c1\ub3c4 \uc788\uc73c\ub2c8 \uac19\uc774 \ucc38\uace0\ud574\uc8fc\uc2dc\uba74 \uac10\uc0ac\ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n<ul>\n  <li><a href=\"https:\/\/www2.slideshare.net\/HoseongLee6\/do-adversarially-robust-image-net-models-transfer-better\" target=\"_blank\"><b> PPT Slide <\/b><\/a><\/li>\n  <li><a href=\"https:\/\/youtu.be\/x2L393xcL2M\" target=\"_blank\"><b> YouTube Video <\/b><\/a><\/li>\n<\/ul>\n\n<blockquote> Related Works <\/blockquote>\n<p>\uc6b0\uc120 \uc624\ub298 \uc18c\uac1c\ub4dc\ub9b4 \ub17c\ubb38\uacfc \uad00\ub828\uc774 \uc788\ub294 \uc120\ud589 \uc5f0\uad6c\ub4e4\uc744 \uc9da\uace0 \ub118\uc5b4\uac00\uaca0\uc2b5\ub2c8\ub2e4. \uc5ec\uae30\uc11c \uc5b8\uae09\ud558\ub294 \uc120\ud589\uc5f0\uad6c\ub4e4\uc740 \ubcf8 \ub17c\ubb38\uc5d0\uc11c \uc778\uc6a9\ud55c \ub17c\ubb38\ub4e4\uc785\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"transfer-learning-in-various-domain\">Transfer Learning in various domain<\/h3>\n<ul>\n  <li>\u201cComparison of deep transfer learning strategies for digital pathology\u201d, 2018 CVPRW<\/li>\n  <li>\u201cSenteval: An evaluation toolkit for universal sentence representations\u201d, 2018 arXiv<\/li>\n  <li>\u201cFaster r cnn : Towards real time object detection with region proposal networks\u201d, 2015 NIPS<\/li>\n  <li>\u201cR fcn : Object detection via region based fully convolutional networks\u201d, 2016 NIPS<\/li>\n  <li>\u201cSpeed\/accuracy trade offs for modern convolutional object detectors\u201d, 2017 CVPR<\/li>\n  <li>Deeplab : Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs \u201d, 2017 TPAMI<\/li>\n<\/ul>\n\n<p>\uc6b0\uc120 \ub2e4\uc591\ud55c domain\uc5d0 Transfer Learning\uc744 \uc801\uc6a9\ud558\ub824\ub294 \uc2dc\ub3c4\ub97c \ub2e4\ub8ec \ub17c\ubb38\uc740 \uad49\uc7a5\ud788 \ub9ce\uc2b5\ub2c8\ub2e4. \uadf8 \uc911\uc5d0\uc11c Medical Imaging, Language Modeling, Object Detection, Segmentation \ub4f1\uc5d0 Transfer Learning\uc744 \uc811\ubaa9\uc2dc\ud0a8 \ub300\ud45c\uc801\uc778 \ub17c\ubb38\uc774 \uc704\uc5d0 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"transfer-learning-with-fine-tuning-or-frozen-feature-based-methods\">Transfer Learning with fine-tuning or frozen feature-based methods<\/h3>\n<ul>\n  <li>\u201cAnalyzing the performance of multilayer neural networks for object recognition\u201d, 2014 ECCV<\/li>\n  <li>\u201cReturn of the devil in the details: Delving deep into convolutional nets\u201d, 2014 arXiv<\/li>\n  <li>\u201cRich feature hierarchies for accurate object detection and semantic seg-mentation\u201d,2014 CVPR<\/li>\n  <li>\u201cHow transferable are features in deep neural networks?\u201d,2014 NIPS<\/li>\n  <li>\u201cFactors of transferability for a generic convnet representation\u201d, 2015 TPAMI<\/li>\n  <li>\u201cBilinear cnn models for fine-grained visual recognition\u201d,2015 ICCV<\/li>\n  <li>\u201cWhat makes ImageNet good for transfer learning?\u201d, 2016 arXiv<\/li>\n  <li>\u201cBest practices for fine-tuning visual classifiers to new domains\u201d,2016 ECCV<\/li>\n<\/ul>\n\n<p>\ub2e4\uc74c\uc73c\ub860 Transfer Learning\uc744 \ud560 \ub54c feature extractor\ub97c freeze\ud560 \uc9c0, \uc544\ub2c8\uba74 \uc804\uccb4\ub97c fine-tuning\ud560 \uc9c0\ub97c \ubd84\uc11d\ud55c \uc5f0\uad6c\ub4e4\uc774 \uc704\uc5d0 \uc815\ub9ac\ub418\uc5b4 \uc788\uc2b5\ub2c8\ub2e4. \uc694\uc998\uc5d0\ub294 \ud6c4\uc790\uc778 fine-tuning\uc774 \ub300\uc138\uc774\uba70 \uc2e4\uc81c\ub85c \uc5ec\ub7ec \uc5f0\uad6c\uc5d0\uc11c\ub3c4 fine-tuning\uc774 \ub354 \uc88b\uc740 \uc131\ub2a5\uc744 \ubcf4\uc784\uc744 \uc81c\uc2dc\ud558\uace0 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"adversarial-robustness\">Adversarial robustness<\/h3>\n<ul>\n  <li>\u201cTowards deep learning models resistant to adversarial attacks\u201d, 2018 ICLR<\/li>\n  <li>\u201cVirtual adversarial training: a regularization method for supervised and semi-supervised learning\u201d,2018<\/li>\n  <li>\u201cProvably robust deep learning via adversarially trained smoothed classifier\u201d,2019NeurIPS<\/li>\n<\/ul>\n\n<figure>\n\t<img src=\"\/assets\/img\/transfer-robust\/2.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub2e4\uc74c\uc740 \uc6cc\ub099 \uc720\uba85\ud55c \ubd84\uc57c\uc8e0. Adversarial robustness\uc785\ub2c8\ub2e4. \uc704\uc758 \uadf8\ub9bc\uc740 \ub525\ub7ec\ub2dd\uc744 \uacf5\ubd80\ud574\ubcf4\uc2e0 \ubd84\ub4e4\uc774\ub77c\uba74 \ubc18\ub4dc\uc2dc \ud55c \ubc88\ucbe4\uc740 \ubcf4\uc168\uc744 \uadf8\ub9bc\uc785\ub2c8\ub2e4. Adversarial Attack\uc744 \ub2e4\ub8ec \ub17c\ubb38, Defense\ub97c \ub2e4\ub8ec \ub17c\ubb38 \ub4f1 \ub2e4\uc591\ud55c \ub17c\ubb38\ub4e4\uc774 \ub2e8\uae30\uac04\uc5d0 \uc3df\uc544\uc838 \ub098\uc654\uace0, \ucd5c\uadfc\uc5d0\ub294 Adversarially robust\ud558\uac8c network\ub97c \ud559\uc2b5\uc2dc\ucf30\uc744 \ub54c \uc5bb\uc5b4\uc9c0\ub294 feature\ub4e4\uc774 \uc5b4\ub5a4 \ud2b9\uc9d5\uc744 \uac00\uc9c0\uace0 \uc788\ub294\uc9c0\ub97c \ubd84\uc11d\ud558\ub824\ub294 \uc2dc\ub3c4\ub4e4\uc774 \uc81c\uc548\uc774 \ub418\uace0 \uc788\uc2b5\ub2c8\ub2e4. \uc77c\ubc18\uc801\uc73c\ub85c adversarial robustness\ub97c \uc99d\uac00\uc2dc\ud0a4\uba74, \uc77c\ubc18 test set\uc5d0\uc11c\uc758 accuracy\uac00 \uac10\uc18c\ud55c\ub2e4\uace0 \uc54c\ub824\uc838 \uc788\uc73c\uba70, \uc774 \ub458 \uac04\uc758 tradeoff\ub97c \uc774\ub860\uc801, \uc2e4\ud5d8\uc801\uc73c\ub85c \ubc1d\ud788\ub824\ub294 \uc5f0\uad6c\ub4e4\ub3c4 \uc9c4\ud589\uc774 \ub418\uace0 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"adversarial-robustness--transfer-learning\">Adversarial robustness &amp; Transfer Learning<\/h3>\n<ul>\n  <li>\u201cAdversarially robust transfer learning\u201d, 2019 arXiv<\/li>\n  <li>\u201cAdversarially-Trained Deep Nets Transfer Better\u201d, 2020 arXiv<\/li>\n<\/ul>\n\n<p>\ub9c8\uc9c0\ub9c9\uc73c\ub85c \uc624\ub298 \uc18c\uac1c \ub4dc\ub9ac\ub294 \ub17c\ubb38\uacfc \uac70\uc758 \ube44\uc2b7\ud55c \uad00\uc810\uc5d0\uc11c \uc9c4\ud589\ub41c \ub17c\ubb38\uc740 \ub450 \ud3b8\uc774 \uc788\uc2b5\ub2c8\ub2e4. \uc6b0\uc120 \uccab\ubc88\uc9f8 \ub17c\ubb38\uc740 \uc624\ub298 \ub17c\ubb38\uacfc\ub294 \ubc18\ub300\ub85c, Transfer Learning\uc744 \ud558\uba74 From scratch\ub85c \ud559\uc2b5\uc744 \uc2dc\ud0ac \ub54c \ubcf4\ub2e4 downstream task\uc5d0\uc11c adversarial robustness\uac00 \uc99d\uac00\ud55c\ub2e4\ub294 \uad00\ucc30\uc744 \ub2f4\uc740 \ub17c\ubb38\uc785\ub2c8\ub2e4. \ub450 \ubc88\uc9f8 \ub17c\ubb38\uc740 \uc624\ub298 \uc18c\uac1c \ub4dc\ub9ac\ub294 \ub17c\ubb38\uacfc \uac19\uc740 \uc774\uc57c\uae30\ub97c \ud558\uace0 \uc788\uc2b5\ub2c8\ub2e4. Adversarial robustness\ub97c \uc99d\uac00\uc2dc\ud0a4\ub294 \ubc29\ud5a5\uc73c\ub85c network\ub97c \ud559\uc2b5\uc2dc\ud0a4\uba74 Transfer Learning\uc774 \uc798 \ub41c\ub2e4\ub294 \uac83\uc744 \uad00\ucc30\uc740 \ud588\uc9c0\ub9cc \uc2e4\ud5d8\uacfc \ubd84\uc11d\uc774 \ub2e4\uc18c \ubd80\uc871\ud55c \uce21\uba74\uc774 \uc788\uc5b4\uc11c \uc774\ub97c \ubc1c\uc804\uc2dc\ud0a8 \ub17c\ubb38\uc774 \uc624\ub298 \ub2e4\ub8f0 \u201cDo Adversarially Robust ImageNet Models Transfer Better?\u201d \uc774\ub77c \ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.\n\u2003<\/p>\n<blockquote> Motivation: Feature Representation &amp; Transfer Learning <\/blockquote>\n<p>\uc790 \uc774\uc81c \ubcf8\ub860\uc73c\ub85c \ub4e4\uc5b4\uac00\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/transfer-robust\/3.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc704\uc758 \uadf8\ub9bc\uc740 \uc81c\uac00 \ub525\ub7ec\ub2dd\uc744 \uc785\ubb38\ud560 \ub54c \ub9ce\uc740 \ucc38\uace0\ub97c \ud588\ub358 \uc2a4\ud0e0\ud3ec\ub4dc\uc758 cs231n\uc758 lecture note\uc5d0\uc11c \ubc1c\ucdcc\ud55c \uadf8\ub9bc\uc785\ub2c8\ub2e4. \uc608\uc804\uc5d0\ub294 \uc704\uc758 \uadf8\ub9bc\uc758 \uac00\uc6b4\ub370\uc640 \uac19\uc774 feature extractor\ub294 freeze \uc2dc\ud0a4\uace0, \ub9c8\uc9c0\ub9c9\uc5d0 Fully-connected layer\ub9cc \ubd99\uc5ec\uc11c Transfer Learning\uc744 \ud558\ub294 \ubc29\uc2dd\uc744 \ub9ce\uc774 \uc0ac\uc6a9\ud588\uc2b5\ub2c8\ub2e4. \uadf8\ub807\uac8c \ub418\uba74 pretraining\uc744 \ud558\uc5ec \uc5bb\uc740 feature extractor\uc758 \ud488\uc9c8\uc5d0 \ub530\ub77c Transfer Learning\uc758 \uc131\ub2a5\uc774 \uc88c\uc6b0\ub418\uaca0\uc8e0?<\/p>\n\n<p>\uc5ec\uae30\uc11c \uad81\uae08\uc99d\uc774 \uc0dd\uae41\ub2c8\ub2e4. \uacfc\uc5f0 pretraining\uc5d0 \uc0ac\uc6a9\ud55c source dataset (e.g. ImageNet)\uc5d0 \ub300\ud574\uc11c \ub192\uc740 \uc815\ud655\ub3c4\ub97c \uc5bb\uc5c8\ub2e4\uace0 \ubc18\ub4dc\uc2dc feature extractor\uac00 \uc88b\ub2e4\uace0 \ud560 \uc218 \uc788\uc744\uae4c? (= Transfer Learning \uc131\ub2a5\uc774 \ub192\uac8c \ub098\uc62c \uc218 \uc788\uc744\uae4c?)<\/p>\n\n<p>\uc704\uc5d0 \uc9c8\ubb38\uc5d0\ub294 \uc0ac\uc2e4 \ub300\ub2f5\ud558\uae30 \uc5b4\ub835\uc2b5\ub2c8\ub2e4. Network\uac00 \ub192\uc740 \uc815\ud655\ub3c4\ub97c \ubcf4\uc720\ud55c\ub2e4\uba74, \uadf8\ub9cc\ud07c \uc591\uc9c8\uc758 feature\ub97c \ucd94\ucd9c\ud55c\ub2e4\uace0 \uc0dd\uac01\ud560 \uc218 \uc788\uae34 \ud558\uc9c0\ub9cc, \uadf8\ub807\ub2e4\uace0 \ud574\uc11c \ubb34\uc870\uac74 Transfer Learning\ub3c4 \uc798 \ub420 \uac83\uc774\ub77c\ub294 \ubcf4\uc7a5\uc740 \ud558\uae30 \uc5b4\ub835\uc2b5\ub2c8\ub2e4. Transfer Learning \uad00\uc810\uc5d0\uc11c \uc0dd\uac01\ud574\ubcf4\uba74 \ub2e8\uc21c source dataset\uc5d0 \ub300\ud55c accuracy \ubcf4\ub2e4\ub294 feature extractor\uc758 \ud488\uc9c8\uc774 \ub354 Transfer Learning\uc5d0 \uc88b\uc740 \uc601\ud5a5\uc744 \ubbf8\uce60 \uac83\uc774\ub77c\ub294 \uc0c1\uc0c1\uc744 \ud574\ubcfc \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc5ec\uae30\uc11c feature extractor\uc758 \ud488\uc9c8\uc744 \ub192\uc774\uae30 \uc704\ud574\uc11c\ub294 network architecture\ub97c \uac74\ub4dc\ub9ac\ub294 \ubc29\ubc95\ub3c4 \uc788\uace0, loss function\uc744 \ubc14\uafb8\ub294 \ubc29\ubc95\ub3c4 \uc788\uace0 \uac01\uc885 augmentation, regularization \uae30\ubc95\uc744 \uc801\uc6a9\ud558\ub294 \ubc29\ubc95\ub3c4 \uc788\uc2b5\ub2c8\ub2e4. \uc989, <strong>Transfer Learning\uc744 \uc798 \ud558\uae30 \uc704\ud574\uc120, \ub2e8\uc21c\ud788 source dataset\uc5d0 \ub300\ud55c accuracy\ub9cc \ubcf4\uc9c0 \ub9d0\uace0, feature extractor\ub97c \uc88b\uac8c \ub9cc\ub4dc\ub294 \uac83\uc5d0 \ucd08\uc810\uc744 \ub46c\uc57c \ud55c\ub2e4<\/strong> \uac00 \uc694\uc9c0\uc785\ub2c8\ub2e4.<\/p>\n\n<p>\uc11c\ub450\uac00 \uae38\uc5c8\ub294\ub370, \uc815\ub9ac\ud558\uc790\uba74 Transfer Learning\uc744 \uc798 \ud558\uae30 \uc704\ud574\uc120 \uc591\uc9c8\uc758 feature extractor\ub97c \uc5bb\uc5b4\uc57c \ud558\uace0, \uc591\uc9c8\uc758 feature extractor\ub97c \uc5bb\uae30 \uc704\ud574\uc120 \uac01\uc885 \uae30\ubc95\ub4e4\uc744 \uc801\uc6a9\ud560 \uc218 \uc788\ub294\ub370 \uadf8 \uc911 \ud55c\uac00\uc9c0 \uae30\ubc95\uc774 adversarial robustness\ub97c \uc99d\uac00\uc2dc\ud0a4\ub294 \ubc29\ubc95\uc785\ub2c8\ub2e4. \ud558\uc9c0\ub9cc \uc5ec\uae30\uc11c \uc57d\uac04\uc758 \ubaa8\uc21c\uc774 \uc0dd\uae41\ub2c8\ub2e4. Adversarially robust\ud558\uac8c network\ub97c \ud559\uc2b5\uc2dc\ud0a4\uba74 trade-off\ub85c \uc778\ud574 standard test set\uc5d0 \ub300\ud55c accuracy\uac00 \ub5a8\uc5b4\uc9c0\uac8c \ub429\ub2c8\ub2e4. \uc5ec\uae30\uc11c 2\uac00\uc9c0 \uc8fc\uc7a5\uc774 \uc0dd\uaca8\ub0a9\ub2c8\ub2e4.<\/p>\n<ul>\n  <li>Adversarially robust\ud558\uac8c \ud559\uc2b5\uc744 \uc2dc\ud0a4\uba74 standard accuracy\uac00 \ub5a8\uc5b4\uc9c0\ub2c8 Transfer Learning \uc131\ub2a5\ub3c4 \ub5a8\uc5b4\uc9c8 \uac83\uc774\ub2e4. (\uc608\uc804 \uc5f0\uad6c\ub4e4\uc758 \uc758\uacac)<\/li>\n  <li>Adversarially robust\ud558\uac8c \ud559\uc2b5\uc744 \uc2dc\ud0a4\uba74 feature representation\uc758 \ud488\uc9c8\uc774 \uc88b\uc544\uc9c0\ub2c8 Transfer Learning \uc131\ub2a5\uc774 \uc88b\uc544\uc9c8 \uac83\uc774\ub2e4. (\uc800\uc790\ub4e4\uc758 \uc8fc\uc7a5)<\/li>\n<\/ul>\n\n<figure>\n\t<img src=\"\/assets\/img\/transfer-robust\/4.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc2e4\uc81c\ub85c network\ub97c Adversarially robust\ud558\uac8c \ud559\uc2b5\uc744 \uc2dc\ud0a8 \ub4a4 feature \ub4e4\uc744 \ucd94\ucd9c\ud574\ubcf4\uba74 \uc77c\ubc18 \ubaa8\ub378\ub4e4 \ubcf4\ub2e4 \ub354 \uc2dc\uac01\uc801\uc778 \uc815\ubcf4\ub97c \ub9ce\uc774 \ub2f4\uace0 \uc788\uace0, representation\uc758 invertibility\uac00 \ub192\uc544\uc9c0\uace0, \ub354\uc6b1 specialized feature\ub97c \ubc30\uc6b4 \ub2e4\ub294 \uc5f0\uad6c\ub4e4\uc774 \ub9ce\uc774 \ubc1c\ud45c\uac00 \ub418\uc5c8\uc2b5\ub2c8\ub2e4. \uc800\uc790\ub4e4\uc740 \uc704\uc758 \uc0c1\ucda9\ub418\ub294 \ub450 \uac00\uc9c0 \uac00\uc124 \uc911\uc5d0 \ubb34\uc5c7\uc774 \uc815\ub2f5\uc778\uc9c0 \ud655\uc778\ud558\uae30 \uc704\ud574 \uc2e4\ud5d8\uc744 \uc124\uacc4\ud558\uace0 \uc9c4\ud589\ud558\uc600\uc2b5\ub2c8\ub2e4. \uc2e4\ud5d8\uc740 Fixed-Feature Transfer Learning \uc2e4\ud5d8\uacfc Full-Network Fine Tuning \ubaa8\ub450 \uc9c4\ud589\ud558\uc600\uc73c\uba70, \ub450 \uc2e4\ud5d8\uc758 \uacb0\uacfc\ub294 \uad49\uc7a5\ud788 \ub192\uc740 correlation\uc744 \uac00\uc9d1\ub2c8\ub2e4.<\/p>\n\n<blockquote> \uc2e4\ud5d8 \uacb0\uacfc <\/blockquote>\n<p>\uc790 \uc774\uc81c \uc2e4\ud5d8 \uc14b\ud305\uc744 \uc124\uba85 \ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4. \uc6b0\uc120 network\ub294 ResNet \uae30\ubc18\uc758 4\uac1c\uc758 architecture (ResNet-18, ResNet-50, WideResNet-50-x2, WideResNet-50-x4)\ub97c \uc0ac\uc6a9\ud558\uc600\uc73c\uba70, ResNet-50\uc5d0\uc11c channel \uac1c\uc218\ub97c 2\ubc30, 4\ubc30\uc529 \ud0a4\uc6cc\uc900 \uc774\uc720\ub294 \ub4a4\uc5d0\uc11c network\uc758 width\uc640 Transfer Learning\uc758 \uc131\ub2a5\uc744 \ube44\uad50\ud558\ub294 \uc2e4\ud5d8\uc5d0 \uc0ac\uc6a9\ub418\uae30 \ub54c\ubb38\uc785\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/transfer-robust\/5.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub2e4\uc74c\uc73c\ub85c \uc2e4\ud5d8\uc5d0 \uc0ac\uc6a9\ud55c \ub370\uc774\ud130 \uc14b\uc740 \uc77c\ubc18\uc801\uc778 Transfer Learning \uc5f0\uad6c\uc5d0\uc11c \uc8fc\ub85c \uc0ac\uc6a9\ub418\ub294 12\uac00\uc9c0\uc758 \ubca4\uce58\ub9c8\ud06c \ub370\uc774\ud130\uc14b\uc744 \uc0ac\uc6a9\ud558\uc600\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/transfer-robust\/6.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc2e4\ud5d8 \uacb0\uacfc \uc800\uc790\ub4e4\uc758 \uc8fc\uc7a5\ub300\ub85c Robust Model\uc744 \uc0ac\uc6a9\ud558\uc600\uc744 \ub54c transfer accuracy\uac00 \ub300\uccb4\ub85c \ub354 \ub192\uc740 \uac83\uc744 \ud655\uc778\ud560 \uc218 \uc788\uc5c8\uc2b5\ub2c8\ub2e4. \ubb3c\ub860 \ubaa8\ub4e0 \ub370\uc774\ud130\uc14b\uc5d0\uc11c \ub2e4 \uadf8\ub7f0 \uac74 \uc544\ub2c8\uc9c0\ub9cc \ub300\uccb4\ub85c \uadf8\ub7f0 \uacbd\ud5a5\uc744 \ubcf4\uc785\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/transfer-robust\/7.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub611\uac19\uc740 \uc2e4\ud5d8\uc744 Full-Network Fine tuning \uae30\ubc18\uc758 Transfer Learning\uc5d0\uc11c\ub3c4 \uc218\ud589\uc744 \ud558\uc600\uc2b5\ub2c8\ub2e4. \ub9ce\uc740 \uc120\ud589 \uc5f0\uad6c\ub4e4\uc5d0\uc11c Fixed-Feature transfer learning\uacfc Full-Network Fine Tuning\uc758 Transfer \uc131\ub2a5\uc774 \ub9e4\uc6b0 \ub192\uc740 correlation\uc744 \uac00\uc9c0\uace0 \uc788\uc74c\uc744 \ubcf4\uc600\uc5c8\ub294\ub370, Robust Model\uc744 \uc0ac\uc6a9\ud558\uc600\uc744 \ub54c\uc5d0\ub3c4 \uac19\uc740 \uacbd\ud5a5\uc744 \ubcf4\uc774\ub294 \uac83\uc744 \ud655\uc778\ud560 \uc218 \uc788\uc5c8\uace0, \ub9c8\ucc2c\uac00\uc9c0\ub85c Robust Model\uc744 \uc0ac\uc6a9\ud558\uc600\uc744 \ub54c \ub354 \uc88b\uc740 Transfer Accuracy\ub97c \ubcf4\uc600\uc2b5\ub2c8\ub2e4. \uc989, \uc800\uc790\ub4e4\uc758 \uc8fc\uc7a5\uc774 \ub9de\uc558\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/transfer-robust\/8.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub610\ud55c Classification \ubfd0\ub9cc \uc544\ub2c8\ub77c \ub2e4\ub978 Computer vision task\uc778 object detection, instance segmentation\uc5d0\ub3c4 Transfer Learning \uc2e4\ud5d8\uc744 \uc218\ud589\ud558\uc600\uace0 \ub9c8\ucc2c\uac00\uc9c0\ub85c Robust Model\uc744 \uc0ac\uc6a9\ud560 \ub54c \ub354 \uc88b\uc740 \uc815\ud655\ub3c4\ub97c \ub2ec\uc131\ud560 \uc218 \uc788\uc74c\uc744 \ubcf4\uc5ec\uc8fc\uace0 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<blockquote> Analysis &amp; Discussion <\/blockquote>\n<p>\ub2e4\uc74c\uc740 \ucd94\uac00\uc801\uc778 \uc2e4\ud5d8\uc744 \ud1b5\ud574 Adversarially Robust Network\uc758 \ud589\ub3d9\uc744 \ubd84\uc11d\ud55c \ub0b4\uc6a9\ub4e4\uc744 \uac04\ub7b5\ud558\uac8c \uc124\uba85\ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"41-imagenet-accuracy-and-transfer-performance\">4.1 ImageNet accuracy and Transfer performance<\/h3>\n<p>\uc6b0\uc120 robust network\uc640 \uc77c\ubc18 standard network\uc5d0 transfer learning\uc744 \uc801\uc6a9\ud558\uc600\uc744 \ub54c\uc758 \uacf5\ud1b5\uc810\uacfc \ucc28\uc774\uc810\uc744 \uc0b4\ud3b4\ubcf4\uc558\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/transfer-robust\/9.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc704\uc758 \uc2e4\ud5d8 \uacb0\uacfc\ub294 \ub2e4\uc591\ud55c adversarial robust revel\uc5d0 \ub530\ub978 standard accuracy (x\ucd95)\uc640 transfer accuracy (y\ucd95)\uc744 \ub098\ud0c0\ub0b8 \uadf8\ub798\ud504\uc774\uace0, \ube68\uac04\uc0c9 \uc810\uc120\uc774 standard network\uc758 transfer accuracy\ub97c \uc758\ubbf8\ud569\ub2c8\ub2e4. \uacb0\uacfc\ub97c \uc0b4\ud3b4\ubcf4\uba74 \ub300\uccb4\ub85c x\ucd95\uc774 \ucee4\uc9c8\uc218\ub85d y\ucd95\ub3c4 \ucee4\uc9c0\ub294 \uacbd\ud5a5\uc744 \ubcf4\uc774\uc9c0\ub9cc \uadf8\ub807\uc9c0 \uc54a\uc740 \uacbd\uc6b0 (ex, CIFAR-10, CIFAR-100, Caltech-101)\ub3c4 \uc874\uc7ac\ud558\ub294 \uac83\uc744 \uc54c \uc218 \uc788\uace0, Robust Model\uc744 \uc0ac\uc6a9\ud560 \ub54c\uac00 \uadf8\ub807\uc9c0 \uc54a\uc744 \ub54c(\ube68\uac04 \uc810\uc120)\ubcf4\ub2e4 \ud56d\uc0c1 \uc704\uc5d0 \uc874\uc7ac\ud560 \uc218 \uc788\uc74c\uc744 \ubcf4\uc5ec\uc8fc\uace0 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\uc774\ub7ec\ud55c \uc2e4\ud5d8\uc744 \ud1b5\ud574 standard accuracy (source \ub370\uc774\ud130 \uc14b\uc5d0\uc11c\uc758 accuracy)\uc640 transfer accuracy\ub294 \uc5b4\ub290 \uc815\ub3c4 \uc591\uc758 \uc0c1\uad00\uad00\uacc4\ub97c \uac00\uc9c0\uc9c0\ub9cc, adversarial robustness\ub97c \uace0\ub824\ud558\uba74 \uadf8\ub807\uc9c0 \uc54a\uc740 \uacbd\uc6b0\uac00 \ubc1c\uc0dd\ud558\uba70, \uc774\ub97c \ud1b5\ud574 robustness\uc640 standard accuracy\ub294 \ubd84\ub9ac\ud574\uc11c \uc0dd\uac01\ud574\uc57c \ud568\uc744 \uc2dc\uc0ac\ud558\uace0 \uc788\uc2b5\ub2c8\ub2e4. \uc989, \uace0\uc815\ub41c robustness \uac12\uc5d0\uc11c\ub294 \ub192\uc740 standard accuracy\uc77c \ub54c \ub354 \uc88b\uc740 transfer accuracy\ub97c \uac00\uc9c0\uace0, \uace0\uc815\ub41c standard accuracy \uc5d0\uc11c\ub294 \ub192\uc740 robustness\uc77c \ub54c \ub354 \uc88b\uc740 transfer accuracy\ub97c \uac00\uc9c0\ub294 \uac83\uc744 \uc758\ubbf8\ud569\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/transfer-robust\/10.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc774\ub97c \ub354 \uc790\uc138\ud788 \ud655\uc778\ud558\uae30 \uc704\ud574 robustness level\uc744 0\uacfc 3\uc73c\ub85c \uace0\uc815\ud574\ub450\uace0, \ub2e4\uc591\ud55c architecture\uc5d0 \ub300\ud574\uc11c \uc2e4\ud5d8\uc744 \uc218\ud589\ud55c \uacb0\uacfc\uac00 \uc704\uc758 \ud45c\uc5d0 \uc815\ub9ac\uac00 \ub418\uc5b4\uc788\uc2b5\ub2c8\ub2e4. Standard Model\uc744 \uc0ac\uc6a9\ud558\uc600\uc744 \ub54c\uc5d0 source dataset\uc758 accuracy(ImageNet)\uacfc transfer accuracy(CIFAR-10)\uc758 correlation\uc740 0.79\uc778 \ubc18\uba74, Robust Model\uc744 \uc0ac\uc6a9\ud558\uba74 \ub458 \uac04\uc758 correlation\uc774 0.98\ub85c \ub9e4\uc6b0 \ucee4\uc9c0\ub294 \uac83\ub3c4 \ubc1c\uacac\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"42-robust-models-improve-with-width\">4.2 Robust Models improve with width<\/h3>\n<p>\ub2e4\uc74c\uc73c\ub860 \uae30\uc874 \uc5f0\uad6c\ub4e4\uc5d0\uc11c\ub294 network\uc758 depth (layer \uac1c\uc218)\ub97c \ud0a4\uc6cc\uc8fc\ub294 \uac83\uc740 transfer accuracy\uc5d0 \uc88b\uc740 \uc601\ud5a5\uc744 \uc900 \ubc18\uba74, network\uc758 width (channel \uac1c\uc218)\ub97c \ud0a4\uc6cc\uc8fc\ub294 \uac83\uc740 \uc624\ud788\ub824 transfer accuracy\uc5d0 \uc548 \uc88b\uc740 \uc601\ud5a5\uc744 \uc92c\ub2e4\uace0 \ud569\ub2c8\ub2e4. \uc774\ub7ec\ud55c \uacbd\ud5a5\uc774 Robust Model\uc5d0\uc11c\ub3c4 \uad00\ucc30\ub418\ub294\uc9c0 \ud655\uc778\ud558\uae30 \uc704\ud574 \uc2e4\ud5d8\uc744 \uc218\ud589\ud558\uc600\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/transfer-robust\/11.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc2e4\ud5d8 \uacb0\uacfc Standard Model (\ube68\uac04 \uc810\uc120)\uc758 \uacbd\uc6b0 width\ub97c \ud0a4\uc6cc\uc904\uc218\ub85d Transfer accuracy\uac00 \ub5a8\uc5b4\uc9c0\uac70\ub098 \ube44\uc2b7\ud55c \uac12\uc744 \uac00\uc9c0\ub294 \uacbd\ud5a5\uc774 \uad00\ucc30\ub418\ub294 \ubc18\uba74, Robust Model\uc758 \uacbd\uc6b0 width\ub97c \ud0a4\uc6cc\uc8fc\uba74 Transfer accuracy\ub3c4 \uac19\uc774 \uc99d\uac00\ud558\ub294 \uacbd\ud5a5\uc744 \ubcf4\uc774\ub294 \uac83\uc744 \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"43-optimal-robustness-levels-for-downstream-tasks\">4.3 Optimal robustness levels for downstream tasks<\/h3>\n<p>\ub2e4\uc74c\uc73c\ub860 \uac01 \ub370\uc774\ud130 \uc14b \ub9c8\ub2e4 \uac00\uc7a5 Transfer \uc131\ub2a5\uc774 \uc88b\uc558\ub358 robustness parameter\uc5d0 \ub300\ud55c \uace0\ucc30\uc785\ub2c8\ub2e4. CIFAR-10, CIFAR-100\uc5d0\uc11c\ub294 \uac01\uac01 1, 3\uc77c \ub54c\uac00 \ucd5c\uc801\uc778 \ubc18\uba74, \ub098\uba38\uc9c0 10\uac1c\uc758 \ub370\uc774\ud130 \uc14b\uc5d0\uc11c\ub294 \uad49\uc7a5\ud788 \uc791\uc740 \uac12\uc758 \uc785\uc2e4\ub860\uc744 \uc0ac\uc6a9\ud560 \ub54c \uc131\ub2a5\uc774 \uc88b\uc558\ub2e4\uace0 \ud569\ub2c8\ub2e4. CIFAR\uc640 \ub098\uba38\uc9c0 \ub370\uc774\ud130\uc14b\uc758 \uac00\uc7a5 \ud070 \ucc28\uc774\ub294 input resolution\uc785\ub2c8\ub2e4. CIFAR\ub294 32x32\ub85c \uad49\uc7a5\ud788 \uc791\uc740 \ud574\uc0c1\ub3c4\uc758 \uc774\ubbf8\uc9c0\ub85c \uad6c\uc131\uc774 \ub418\uc5b4\uc788\ub294\ub370, input image resolution\uc744 dataset\uc758 granularity\ub77c\uace0 \uac00\uc815\uc744 \ud55c \ub4a4, dataset\uc758 granularity\uc640 \ucd5c\uc801\uc758 robustness parameter \uc785\uc2e4\ub860\uc774 \uad00\uacc4\uac00 \uc788\uc744 \uac83\uc774\ub77c\ub294 \uac00\uc124\uc744 \uc138\uc6c1\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/transfer-robust\/12.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc774\ub97c \uac80\uc99d\ud558\uae30 \uc704\ud574 \ub098\uba38\uc9c0 10\uac1c\uc758 \ub370\uc774\ud130\uc14b\ub3c4 CIFAR\ucc98\ub7fc 32x32\ub85c \uc904\uc778 \ub4a4 Transfer Learning \uc2e4\ud5d8\uc744 \uc218\ud589\ud558\uc5ec \ucd5c\uc801\uc758 \uc785\uc2e4\ub860 \uac12\uc744 \ucc3e\uc544\ubcf4\ub294 \uc2e4\ud5d8\uc744 \uc218\ud589\ud558\uc600\uace0, \uadf8 \uacb0\uacfc\uac00 \uc704\uc758 \uadf8\ub9bc\uc5d0 \ub098\uc640\uc788\uc2b5\ub2c8\ub2e4. \uc800\uc790\ub4e4\uc758 \uc608\uc0c1\ub300\ub85c input resolution\uc744 \ub9de\ucdb0\uc8fc\ub2c8 \ube44\uc2b7\ud55c \uacbd\ud5a5\uc774 \uad00\ucc30\ub418\ub294 \uac83\uc744 \ud655\uc778\ud560 \uc218 \uc788\uc5c8\uc2b5\ub2c8\ub2e4. \uc774\ub97c \ud1b5\ud574 Dataset\uc758 granularity\uac00 \ub192\uc744\uc218\ub85d (= image resolution\uc774 \ud074\uc218\ub85d) \ub354 \uc791\uc740 \uc785\uc2e4\ub860 \uac12\uc744 \uc0ac\uc6a9\ud558\ub294 \uac83\uc774 \uc720\ub9ac\ud558\ub2e4\ub294 \uc2e4\ud5d8\uc801\uc778 \uacb0\ub860\uc744 \ub0b4\ub9b4 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"44-comparing-adversarial-robustness-to-texture-robustness\">4.4 Comparing adversarial robustness to texture robustness<\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/transfer-robust\/13.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub9c8\uc9c0\ub9c9\uc73c\ub85c <a href=\"https:\/\/arxiv.org\/abs\/1811.12231\" target=\"_blank\"><b> \u201cImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness\u201d, 2019 ICLR <\/b><\/a> \ub17c\ubb38\uc5d0\uc11c \ub9cc\ub4e0 Stylized ImageNet \ub370\uc774\ud130\uc14b\uc73c\ub85c \ud559\uc2b5\uc2dc\ud0a8 texture-invariant model\uacfc \uc131\ub2a5\uc744 \ube44\uad50\ud558\uc600\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/transfer-robust\/14.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc2e4\ud5d8 \uacb0\uacfc, Adversarially robust model\uc774 texture-invariant model\ubcf4\ub2e4 \ub354 \uc88b\uc740 \uc131\ub2a5\uc744 \ub2ec\uc131\ud560 \uc218 \uc788\uc5c8\uc74c\uc744 \ubcf4\uc5ec\uc8fc\uace0 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<blockquote> \uacb0\ub860 <\/blockquote>\n<p>\uc624\ub298\uc740 NeurIPS 2020\uc5d0 \ubc1c\ud45c\ub41c \u201cDo Adversarially Robust ImageNet Models Transfer Better?\u201d \ub17c\ubb38\uc744 \ub9ac\ubdf0\ud574\ubd24\uc2b5\ub2c8\ub2e4. Robustness \ucabd\uc73c\ub85c \ud65c\ubc1c\ud558\uac8c \uc5f0\uad6c \uc911\uc778 MIT Madry Lab\uc5d0\uc11c \ub098\uc628 \ub17c\ubb38\uc774\ub77c \uae30\ub300\ud558\uba74\uc11c \uc77d\uc5c8\ub294\ub370 \uc804\ub2ec\ud558\uace0\uc790 \ud558\ub294 \uba54\uc2dc\uc9c0 \u201cAdversarial Robust Model\uc744 \uc4f0\uba74 Transfer Learning \uc131\ub2a5\uc774 \uc88b\uc544\uc9c4\ub2e4\u201d \ub294 \ud655\uc2e4\ud558\uac8c \uc640 \ub2ff\uc558\uc9c0\ub9cc, \uc65c \uc88b\uc544\uc9c0\ub294\ub370? \uc5d0 \ub300\ud574\uc11c\ub294 \uc544\uc9c1 \uba85\ud655\ud558\uac8c \ubc1d\ud600\uc9c0\uc9c0 \uc54a\uc544\uc11c \uadf8 \uc810\uc774 \uc880 \uc544\uc26c\uc6e0\uace0, \uc800\uc790\ub4e4\ub3c4 \u201cStill, future work is needed to confirm or refute such hypotheses, and more broadly, to understand what properties of pre-trained models are important for transfer learning.\u201d \ub77c\uace0 \uc5b8\uae09\ud558\uba70 \uc774 \uc810\uc744 future work\ub85c \ub0a8\uaca8\ub450\uace0 \uc788\uc2b5\ub2c8\ub2e4. \ub2e4\uc74c \uc5f0\uad6c\uac00 \uae30\ub300\uac00 \ub418\ub294 \ub17c\ubb38\uc774\uc5c8\uc2b5\ub2c8\ub2e4. \uae34 \uae00 \uc77d\uc5b4 \uc8fc\uc154\uc11c \uac10\uc0ac\ud569\ub2c8\ub2e4!<\/p>\n\n","<p>\uc548\ub155\ud558\uc138\uc694, \uc624\ub298\uc740 2020\ub144 NeurIPS \ud559\ud68c\uc5d0\uc11c \ubc1c\ud45c\ub41c <a href=\"https:\/\/arxiv.org\/pdf\/2007.08489.pdf\" target=\"_blank\"><b> \u201cDo Adversarially Robust ImageNet Models Transfer Better?\u201d <\/b><\/a> \ub17c\ubb38\uc744 \ub9ac\ubdf0\ud560 \uc608\uc815\uc785\ub2c8\ub2e4. Transfer Learning\uc744 \ub2e4\ub8ec \ub17c\ubb38\uc774\uba70 Transfer Learning\uc740 \ub525\ub7ec\ub2dd\uc5d0\uc11c \uad49\uc7a5\ud788 \uc790\uc8fc \uc0ac\uc6a9\ub418\ub294 \ud559\uc2b5 \ubc29\ubc95\uc774\uba70 \ucd5c\uadfc\uc5d0\ub294 \uac70\uc758 default\ub85c \uc0ac\uc6a9\uc774 \ub41c\ub2e4\uace0 \ud574\ub3c4 \uacfc\uc5b8\uc774 \uc544\ub2d9\ub2c8\ub2e4. \ub525\ub7ec\ub2dd\uc744 \uacf5\ubd80\ud574\ubcf4\uc2e0 \ubd84\ub4e4\uc774\ub77c\uba74 \ud544\uc218\uc801\uc73c\ub85c ImageNet Pretrained Model\uc744 \uac00\uc838\uc640\uc11c \uc0c8\ub85c\uc6b4 \ub370\uc774\ud130\uc14b\uc5d0 \ud559\uc2b5\uc744 \uc2dc\ucf1c \ubcf4\uc168\uc744 \uac83\uc785\ub2c8\ub2e4. \uc77c\ubc18\uc801\uc73c\ub85c \uc815\ud655\ub3c4\uac00 \ub192\uc558\ub358 pretrained model\uc5d0\uc11c transfer\ub97c \ud558\uba74 target model\uc5d0\uc11c\ub3c4 \ub192\uc740 \uc815\ud655\ub3c4\ub97c \uc5bb\ub294\ub2e4\uace0 \uc54c\ub824\uc838 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n"],"pubDate":"Mon, 07 Dec 2020 00:00:00 +0000","link":"https:\/\/hoya012.github.io\/\/blog\/Do-Adversarially-Robust-ImageNet-Models-Transfer-Better\/","guid":"https:\/\/hoya012.github.io\/\/blog\/Do-Adversarially-Robust-ImageNet-Models-Transfer-Better\/"},{"title":"Image Classification with Stochastic Weight Averaging PyTorch Tutorial","description":["<p>\uc548\ub155\ud558\uc138\uc694, \uc624\ub298\uc740 \uc9c0\ub09c <a href=\"https:\/\/hoya012.github.io\/blog\/SWA\/\" target=\"_blank\"><b> \u201cAveraging Weights Leads to Wider Optima and Better Generalization \ub9ac\ubdf0\u201d <\/b><\/a> \uae00\uc5d0 \uc774\uc5b4\uc11c, \uc774 \ub17c\ubb38\uc5d0\uc11c \uc81c\uc548\ud55c Stochastic Weight Averaging(SWA)\uc744 \uc774\uc6a9\ud55c Image Classification \ud29c\ud1a0\ub9ac\uc5bc\uc744 \uc900\ube44\ud588\uc2b5\ub2c8\ub2e4. \uc9c0\ub09c \uae00\uc5d0\uc11c\ub3c4 \ub9d0\uc500\ub4dc\ub838\uc9c0\ub9cc, PyTorch 1.6\uc5d0\uc11c SWA\ub97c \uacf5\uc2dd\uc801\uc73c\ub85c \uc9c0\uc6d0\ud558\uae30 \uc2dc\uc791\ud574\uc11c \uc774\uc81c \uc190 \uc27d\uac8c \uac00\uc838\ub2e4 \uc4f8 \uc218 \uc788\ub294\ub370\uc694, \uadf8\ub798\uc11c \uc624\ub298 \uae00\uc5d0\uc11c\ub294 \uc9e7\uac8c \ucf54\ub4dc\ub97c \uad6c\ud604\ud574\uc11c \uc124\uba85 \ub4dc\ub9ac\uace0 \uc2e4\ud5d8 \uacb0\uacfc\ub97c \uc18c\uac1c\ub4dc\ub9b4 \uc608\uc815\uc785\ub2c8\ub2e4.<\/p>\n\n<p>\uc2e4\ud5d8\uc5d0 \uc0ac\uc6a9\ud55c \ucf54\ub4dc\ub294 <a href=\"https:\/\/github.com\/hoya012\/swa-tutorials-pytorch\" target=\"_blank\"><b> \uc81c GitHub Repository<\/b><\/a> \uc5d0 \uc62c\ub824 \ub450\uc5c8\uc2b5\ub2c8\ub2e4.<\/p>\n\n<blockquote> PyTorch 1.6 \u2013 Stochastic Weight Averaging <\/blockquote>\n<p>\uc9c0\ub09c 2020\ub144 7\uc6d4 \ub9d0, PyTorch\uc758 \uc0c8\ub85c\uc6b4 \ubc84\uc804\uc778 1.6\uc774 \ub9b4\ub9ac\uc988 \ub418\uc5c8\uc2b5\ub2c8\ub2e4. <a href=\"https:\/\/hoya012.github.io\/blog\/SWA\/\" target=\"_blank\"><b> \ub9b4\ub9ac\uc988 \ub178\ud2b8 <\/b><\/a>\uc5d4 \ub2e4\uc591\ud55c \uae30\ub2a5\ub4e4\uc774 \ucd94\uac00\uac00 \ub418\uc5c8\ub2e4\uace0 \uc124\uba85 \ub418\uc5b4\uc788\uc9c0\ub9cc Stochastic Weight Averaging\uc5d0 \ub300\ud55c \ub0b4\uc6a9\uc774 \ube60\uc838 \uc788\uc5c8\ub294\ub370\uc694, \ubcc4\ub3c4\uc758 <a href=\"https:\/\/pytorch.org\/blog\/pytorch-1.6-now-includes-stochastic-weight-averaging\/\" target=\"_blank\"><b> \uacf5\uc2dd \ube14\ub85c\uadf8 \uae00 <\/b><\/a>\uc744 \ud1b5\ud574 SWA\uc758 native \uc9c0\uc6d0\uc744 \uc124\uba85\ud558\uace0 \uc788\uc2b5\ub2c8\ub2e4. \uc774 \uae00\uc744 \ubcf4\uc2dc\ub294 \ubd84\ub4e4\uc740 \uc6b0\uc120 \uacf5\uc2dd \ube14\ub85c\uadf8 \uae00\uc744 \ubcf4\uace0 \uc624\uc2dc\uba74 \ub354 \uc774\ud574\uac00 \uc218\uc6d4\ud558\uc2e4 \uac83\uc774\ub77c \uc0dd\uac01\ub429\ub2c8\ub2e4. \uacf5\uc2dd \ube14\ub85c\uadf8 \uae00\uc5d0\uc11c \uc815\ub9ac\ud574\ub454 \ub0b4\uc6a9\uc744 \ud55c\uae00\ub85c \ubc88\uc5ed\ud558\uba74 \ub2e4\uc74c\uacfc \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\" alt=\"\" \/> \n<\/figure>\n\n<ul>\n  <li>SWA\ub294 standard training (SGD)\uc640 \ube44\uad50\ud588\uc744 \ub54c \ub2e4\uc591\ud55c computer vision task\uc5d0\uc11c \uc131\ub2a5 \ud5a5\uc0c1\uc774 \uac00\ub2a5\ud558\ub2e4.<\/li>\n  <li>SWA\ub294 Semi-Supervised Learning, Domain Adaptation\uc758 \uc8fc\uc694 benchmark\uc5d0\uc11c SOTA \uc131\ub2a5\uc744 \ub2ec\uc131\ud558\uac8c \ub3c4\uc640\uc900\ub2e4.<\/li>\n  <li>SWA\ub294 language modeling\uacfc \uac15\ud654\ud559\uc2b5(policy gradient method)\uc5d0\uc11c\ub3c4 \uc131\ub2a5 \ud5a5\uc0c1\uc774 \uac00\ub2a5\ud558\ub2e4.<\/li>\n  <li>SWA\uc758 \ud6c4\uc18d \uc5f0\uad6c\uc778 SWAG\uc740 Bayesian model averaging\uc744 approximate\ud560 \uc218 \uc788\uace0, uncertainty calibration\uc5d0\uc11c SOTA \uc131\ub2a5\uc744 \ub2ec\uc131\ud560 \uc218 \uc788\ub2e4. \ub610\ud55c MultiSWAG\uacfc Subspace Inference \ub4f1 \ud6c4\uc18d \uc5f0\uad6c\ub4e4\ub3c4 \uc88b\uc740 \uc131\ub2a5\uc744 \ubcf4\uc778\ub2e4.<\/li>\n  <li>SWA\uc758 Low Precision Training \uae30\ubc95\uc744 \uc801\uc6a9\ud55c SWALP\ub3c4 Full-precision SGD training\uc5d0 \uc900\ud558\ub294 \uc131\ub2a5\uc744 \uc5bb\uc744 \uc218 \uc788\ub2e4.<\/li>\n  <li>SWA\uc758 parallel \ubc84\uc804\uc778 SWAP\ub294 \ud070 batch size\uc640 \ud568\uaed8 Neural Network\ub97c \ud559\uc2b5\uc2dc\ucf1c \ud559\uc2b5 \uc18d\ub3c4\ub97c \ube60\ub974\uac8c \ud560 \uc218 \uc788\uc73c\uba70, 27\ucd08\ub9cc\uc5d0 CIFAR-10\uc5d0\uc11c 94% \uc815\ud655\ub3c4\ub97c \uc5bb\uc744 \uc218 \uc788\ub2e4.<\/li>\n<\/ul>\n\n<p>\uc774\ucc98\ub7fc SWA\ub294 \ube44\ub2e8 Computer Vision \ubfd0\ub9cc \uc544\ub2c8\ub77c \ub2e4\uc591\ud55c task\uc5d0\uc11c\ub3c4 \uc751\uc6a9\uc774 \ub420 \uc218 \uc788\uace0, \uc5ec\ub7ec \ub2e4\uc591\ud55c \ud559\uc2b5 \uae30\ubc95\uacfc\ub3c4 \uac19\uc774 \uc0ac\uc6a9\uc774 \ub420 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uacf5\uc2dd \ube14\ub85c\uadf8\ub97c \ucc38\uace0\ud558\uc2dc\uba74 \uc704\uc758 \ub0b4\uc6a9\ub4e4\uc744 \ud655\uc778\ud558\uc2e4 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<blockquote> Image Classification with SWA Tutorials <\/blockquote>\n<p>\uc790 \uc774\uc81c Image \ubd84\ub958 \ubb38\uc81c\uc5d0 \ub300\ud574 SWA\ub97c \uc801\uc6a9\ud574\ubcf4\ub294 \uc2dc\uac04\uc785\ub2c8\ub2e4. PyTorch 1.6 \ubc84\uc804\uc73c\ub85c \ucf54\ub4dc\ub97c \uad6c\ud604\ud558\uc600\uc2b5\ub2c8\ub2e4. \uc6b0\uc120 \uacf5\uc2dd \ube14\ub85c\uadf8 \uae00\uc5d0\uc11c \uc608\uc81c\ub85c \uc62c\ub824 \ub454 \ucf54\ub4dc\ub294 \ub2e4\uc74c\uacfc \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">torch.optim.swa_utils<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">AveragedModel<\/span><span class=\"p\">,<\/span> <span class=\"n\">SWALR<\/span>\n<span class=\"kn\">from<\/span> <span class=\"nn\">torch.optim.lr_scheduler<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">CosineAnnealingLR<\/span>\n\n<span class=\"n\">loader<\/span><span class=\"p\">,<\/span> <span class=\"n\">optimizer<\/span><span class=\"p\">,<\/span> <span class=\"n\">model<\/span><span class=\"p\">,<\/span> <span class=\"n\">loss_fn<\/span> <span class=\"o\">=<\/span> <span class=\"p\">...<\/span>\n<span class=\"n\">swa_model<\/span> <span class=\"o\">=<\/span> <span class=\"n\">AveragedModel<\/span><span class=\"p\">(<\/span><span class=\"n\">model<\/span><span class=\"p\">)<\/span>\n<span class=\"n\">scheduler<\/span> <span class=\"o\">=<\/span> <span class=\"n\">CosineAnnealingLR<\/span><span class=\"p\">(<\/span><span class=\"n\">optimizer<\/span><span class=\"p\">,<\/span> <span class=\"n\">T_max<\/span><span class=\"o\">=<\/span><span class=\"mi\">100<\/span><span class=\"p\">)<\/span>\n<span class=\"n\">swa_start<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">5<\/span>\n<span class=\"n\">swa_scheduler<\/span> <span class=\"o\">=<\/span> <span class=\"n\">SWALR<\/span><span class=\"p\">(<\/span><span class=\"n\">optimizer<\/span><span class=\"p\">,<\/span> <span class=\"n\">swa_lr<\/span><span class=\"o\">=<\/span><span class=\"mf\">0.05<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">for<\/span> <span class=\"n\">epoch<\/span> <span class=\"ow\">in<\/span> <span class=\"nb\">range<\/span><span class=\"p\">(<\/span><span class=\"mi\">100<\/span><span class=\"p\">):<\/span>\n      <span class=\"k\">for<\/span> <span class=\"nb\">input<\/span><span class=\"p\">,<\/span> <span class=\"n\">target<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">loader<\/span><span class=\"p\">:<\/span>\n          <span class=\"n\">optimizer<\/span><span class=\"p\">.<\/span><span class=\"n\">zero_grad<\/span><span class=\"p\">()<\/span>\n          <span class=\"n\">loss_fn<\/span><span class=\"p\">(<\/span><span class=\"n\">model<\/span><span class=\"p\">(<\/span><span class=\"nb\">input<\/span><span class=\"p\">),<\/span> <span class=\"n\">target<\/span><span class=\"p\">).<\/span><span class=\"n\">backward<\/span><span class=\"p\">()<\/span>\n          <span class=\"n\">optimizer<\/span><span class=\"p\">.<\/span><span class=\"n\">step<\/span><span class=\"p\">()<\/span>\n      <span class=\"k\">if<\/span> <span class=\"n\">epoch<\/span> <span class=\"o\">&gt;<\/span> <span class=\"n\">swa_start<\/span><span class=\"p\">:<\/span>\n          <span class=\"n\">swa_model<\/span><span class=\"p\">.<\/span><span class=\"n\">update_parameters<\/span><span class=\"p\">(<\/span><span class=\"n\">model<\/span><span class=\"p\">)<\/span>\n          <span class=\"n\">swa_scheduler<\/span><span class=\"p\">.<\/span><span class=\"n\">step<\/span><span class=\"p\">()<\/span>\n      <span class=\"k\">else<\/span><span class=\"p\">:<\/span>\n          <span class=\"n\">scheduler<\/span><span class=\"p\">.<\/span><span class=\"n\">step<\/span><span class=\"p\">()<\/span>\n\n<span class=\"c1\"># Update bn statistics for the swa_model at the end\n<\/span><span class=\"n\">torch<\/span><span class=\"p\">.<\/span><span class=\"n\">optim<\/span><span class=\"p\">.<\/span><span class=\"n\">swa_utils<\/span><span class=\"p\">.<\/span><span class=\"n\">update_bn<\/span><span class=\"p\">(<\/span><span class=\"n\">loader<\/span><span class=\"p\">,<\/span> <span class=\"n\">swa_model<\/span><span class=\"p\">)<\/span>\n<span class=\"c1\"># Use swa_model to make predictions on test data \n<\/span><span class=\"n\">preds<\/span> <span class=\"o\">=<\/span> <span class=\"n\">swa_model<\/span><span class=\"p\">(<\/span><span class=\"n\">test_input<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>\ucf54\ub4dc\uac00 \uad49\uc7a5\ud788 \ub2e8\uc21c\ud558\uc8e0? \uc6b0\uc120 \ud2b9\ubcc4\ud55c \ubd80\ubd84\uc740 model\uc744 AveragedModel\ub85c \ubb36\uc5b4\uc8fc\ub294 \ubd80\ubd84\uacfc, SWA scheduler\ub97c \uc0c8\ub86d\uac8c \uc120\uc5b8\ud574\uc8fc\ub294 \ubd80\ubd84\uc774 \uc788\uc2b5\ub2c8\ub2e4. \uadf8\ub9ac\uace0 epoch\ub85c \ubb36\uc5ec \uc788\ub294 for loop \uc548\uc5d0, epoch\uac00 swa_start\ubcf4\ub2e4 \ud06c\uba74 swa_model\uc758 parameter\ub97c update\ud574\uc8fc\uace0, swa_scheduler\ub97c \ud55c step \ub3cc\ub824\uc8fc\ub294 \ubd80\ubd84\uc774 \uc788\uc2b5\ub2c8\ub2e4. \ub9c8\uc9c0\ub9c9\uc73c\ub85c \ud559\uc2b5\uc774 \ub05d\ub09c \ub2e4\uc74c swa_model\uc758 batch normalization layer\ub97c update\ud574\uc8fc\ub294 \ubd80\ubd84\uc774 \ub4a4\ub530\ub985\ub2c8\ub2e4. Batch Normalization\uc758 statistics\ub97c update\ud574\uc8fc\ub294 \ubd80\ubd84\uc5d0 \ub300\ud55c \uc124\uba85\uc740 \ub17c\ubb38\uc5d0\uc11c \uc798 \uc124\uba85\uc774 \ub418\uc5b4\uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p><strong>If the DNN uses batch normal- ization [Ioffe and Szegedy, 2015], we run one additional pass over the data, as in Garipov et al. [2018], to compute the running mean and standard deviation of the activa- tions for each layer of the network with w_SWA weights after the training is finished, since these statistics are not collected during training.<\/strong><\/p>\n\n<p>\ub9c8\uc9c0\ub9c9\uc73c\ub85c torch.optim.swa_utils.py \ud30c\uc77c\uc744 \uc0b4\ud3b4\ubcf4\uc2dc\uba74 \uac01 Class\ub4e4\uc758 \ub3d9\uc791 \uc6d0\ub9ac\ub97c \ub354 \uc790\uc138\ud788 \ud655\uc778\ud558\uc2e4 \uc218 \uc788\uc73c\ub2c8 \ub9c1\ud06c\ub97c \ucca8\ubd80\ud569\ub2c8\ub2e4.<\/p>\n<ul>\n  <li>Torch.optim.swa_utils.py: https:\/\/github.com\/pytorch\/pytorch\/blob\/master\/torch\/optim\/swa_utils.py<\/li>\n<\/ul>\n\n<h3 id=\"0-experimental-setup\">0. Experimental Setup<\/h3>\n<p>\uc6b0\uc120 \uc11c\ub860\uc5d0\ub3c4 \ub9d0\uc500\ub4dc\ub838\uc9c0\ub9cc \uc81c GitHub Repository\uc5d0 \uc788\ub294 <a href=\"https:\/\/github.com\/hoya012\/swa-tutorials-pytorch\" target=\"_blank\"><b> \ucf54\ub4dc<\/b><\/a>\ub97c \ub2e4\uc6b4\ubc1b\uc73c\uc2e0 \ub4a4, \uc2e4\ud5d8\uc5d0 \ud544\uc694\ud55c \ub77c\uc774\ube0c\ub7ec\ub9ac\ub4e4\uc744 \uc124\uce58\ud574\uc90d\ub2c8\ub2e4.<\/p>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"n\">pip<\/span> <span class=\"n\">install<\/span> <span class=\"o\">-<\/span><span class=\"n\">r<\/span> <span class=\"n\">requirements<\/span><span class=\"p\">.<\/span><span class=\"n\">txt<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>\uadf8\ub9ac\uace0 \uc2e4\ud5d8\uc5d0 \uc0ac\uc6a9\ud560 \ub370\uc774\ud130\uc14b\ub3c4 \uc9c0\ub09c AMP Tutorial\uc5d0\uc11c \uc0ac\uc6a9\ud588\ub358 <strong>Intel Image Classification<\/strong> \ub370\uc774\ud130\uc14b\uc744 \ub610 \uc0ac\uc6a9\ud560 \uc608\uc815\uc785\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/mixed_precision\/10.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc774 \ub370\uc774\ud130\uc14b\uc740 \ube4c\ub529, \uc232, \ube59\ud558, \uc0b0, \ubc14\ub2e4, \uac70\ub9ac \ucd1d 6\uac00\uc9c0\uc758 class\ub85c \uad6c\uc131\ub418\uc5b4 \uc788\uace0, 150x150 \ud06c\uae30\uc758 image 25000\uc7a5\uc774 \uc81c\uacf5\ub429\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"1-baseline-training\">1. Baseline Training<\/h3>\n<p>\uc6b0\uc120 SWA\uc744 \uc0ac\uc6a9\ud558\uae30 \uc804\uc5d0 \uae30\uc874 \ubc29\uc2dd\uc73c\ub85c \ud559\uc2b5\uc744 \uc2dc\ucf1c\uc11c Baseline \uc131\ub2a5\uc744 \uce21\uc815\ud560 \uc608\uc815\uc785\ub2c8\ub2e4. Baseline \uc2e4\ud5d8 \uc14b\ud305\uc740 \ub2e4\uc74c\uacfc \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<ul>\n  <li>ImageNet Pretrained ResNet-18 from torchvision.models<\/li>\n  <li>Batch Size 256 \/ Epochs 120 \/ Initial Learning Rate 0.0001<\/li>\n  <li>Training Augmentation: Resize((256, 256)), RandomHorizontalFlip()<\/li>\n  <li>Adam + Cosine Learning rate scheduling with warmup<\/li>\n<\/ul>\n\n<p>\ucd5c\ub300\ud55c \ub2e8\uc21c\ud558\uba74\uc11c \uc790\uc8fc \uc0ac\uc6a9\ub418\ub294 \uae30\ubc95\ub4e4\uc744 \ucc44\ud0dd\ud558\uc600\uc2b5\ub2c8\ub2e4. Baseline Training\uc744 \ub3cc\ub9ac\uae30 \uc704\ud55c \ucee4\ub9e8\ub4dc \ub77c\uc778 \uba85\ub839\uc5b4\ub294 \ub2e4\uc74c\uacfc \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"n\">Python<\/span> <span class=\"n\">main<\/span><span class=\"p\">.<\/span><span class=\"n\">py<\/span> <span class=\"o\">--<\/span><span class=\"n\">checkpoint_name<\/span> <span class=\"n\">baseline<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<h3 id=\"2-stochastic-weight-averaging-training\">2. Stochastic Weight Averaging Training<\/h3>\n<p>\ub2e4\uc74c\uc740 PyTorch 1.6\uc758 SWA \uae30\ub2a5\uc744 \uae30\uc874\uc5d0 \uc0ac\uc6a9\ud558\ub358 Image Classification \ucf54\ub4dc \ubca0\uc774\uc2a4\uc5d0 \uc801\uc6a9\ud558\ub294 \uacfc\uc815\uc744 \uc124\uba85 \ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"kn\">from<\/span> <span class=\"nn\">torch.optim.swa_utils<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">SWALR<\/span>\n<span class=\"s\">\"\"\" define model and learning rate scheduler for stochastic weight averaging \"\"\"<\/span>\n<span class=\"n\">swa_model<\/span> <span class=\"o\">=<\/span> <span class=\"n\">torch<\/span><span class=\"p\">.<\/span><span class=\"n\">optim<\/span><span class=\"p\">.<\/span><span class=\"n\">swa_utils<\/span><span class=\"p\">.<\/span><span class=\"n\">AveragedModel<\/span><span class=\"p\">(<\/span><span class=\"n\">model<\/span><span class=\"p\">)<\/span>\n<span class=\"n\">swa_scheduler<\/span> <span class=\"o\">=<\/span> <span class=\"n\">SWALR<\/span><span class=\"p\">(<\/span><span class=\"n\">optimizer<\/span><span class=\"p\">,<\/span> <span class=\"n\">swa_lr<\/span><span class=\"o\">=<\/span><span class=\"n\">args<\/span><span class=\"p\">.<\/span><span class=\"n\">swa_lr<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>\uc6b0\uc120 <strong>main.py<\/strong> \uc5d0\uc11c \uc704\uc758 PyTorch \uacf5\uc2dd \ube14\ub85c\uadf8 \uae00\uc758 \uc608\uc81c\uc640 \uac19\uc774 model\uc744 AveragedModel\ub85c \ubb36\uc5b4\uc8fc\uace0, SWALR scheduler\ub97c \uc120\uc5b8\ud574\uc90d\ub2c8\ub2e4.<\/p>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"k\">for<\/span> <span class=\"n\">batch_idx<\/span><span class=\"p\">,<\/span> <span class=\"p\">(<\/span><span class=\"n\">inputs<\/span><span class=\"p\">,<\/span> <span class=\"n\">labels<\/span><span class=\"p\">)<\/span> <span class=\"ow\">in<\/span> <span class=\"nb\">enumerate<\/span><span class=\"p\">(<\/span><span class=\"n\">data_loader<\/span><span class=\"p\">):<\/span>\n  <span class=\"k\">if<\/span> <span class=\"ow\">not<\/span> <span class=\"n\">args<\/span><span class=\"p\">.<\/span><span class=\"n\">decay_type<\/span> <span class=\"o\">==<\/span> <span class=\"s\">'swa'<\/span><span class=\"p\">:<\/span>\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">scheduler<\/span><span class=\"p\">.<\/span><span class=\"n\">step<\/span><span class=\"p\">()<\/span>\n  <span class=\"k\">else<\/span><span class=\"p\">:<\/span>\n      <span class=\"k\">if<\/span> <span class=\"n\">epoch<\/span> <span class=\"o\">&lt;=<\/span> <span class=\"n\">args<\/span><span class=\"p\">.<\/span><span class=\"n\">swa_start<\/span><span class=\"p\">:<\/span>\n          <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">scheduler<\/span><span class=\"p\">.<\/span><span class=\"n\">step<\/span><span class=\"p\">()<\/span>\n\n<span class=\"k\">if<\/span> <span class=\"n\">epoch<\/span> <span class=\"o\">&gt;<\/span> <span class=\"n\">args<\/span><span class=\"p\">.<\/span><span class=\"n\">swa_start<\/span> <span class=\"ow\">and<\/span> <span class=\"n\">args<\/span><span class=\"p\">.<\/span><span class=\"n\">decay_type<\/span> <span class=\"o\">==<\/span> <span class=\"s\">'swa'<\/span><span class=\"p\">:<\/span>\n  <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">swa_model<\/span><span class=\"p\">.<\/span><span class=\"n\">update_parameters<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">model<\/span><span class=\"p\">)<\/span>\n  <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">swa_scheduler<\/span><span class=\"p\">.<\/span><span class=\"n\">step<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>\uadf8 \ub4a4, <strong>learning\/trainer.py<\/strong> \uc5d0 \uc788\ub294 Data Loader\ub97c enumerate\ud558\ub294 for loop\uc548\uc5d0 SWA\ub97c \ucd94\uac00\ud574\uc90d\ub2c8\ub2e4. \uc6b0\uc120 \ud604\uc7ac\uc758 epoch\uc774 SWA\ub97c \uc5b8\uc81c\ubd80\ud130 \uc2dc\uc791\ud560 \uc9c0 \uc54c\ub824\uc8fc\ub294 <strong>args.swa_start<\/strong> \ubcf4\ub2e4 \ucee4\uc9c0\uace0, decay type\uc774 swa\uba74 swa_model\uc758 parameter\uc640 swa_scheduler\ub97c update\ud574\uc8fc\uba74 \ub429\ub2c8\ub2e4. \uac70\uc758 \ub611\uac19\uc774 \uad6c\ud604\uc744 \ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"n\">swa_model<\/span> <span class=\"o\">=<\/span> <span class=\"n\">swa_model<\/span><span class=\"p\">.<\/span><span class=\"n\">cpu<\/span><span class=\"p\">()<\/span>\n<span class=\"n\">torch<\/span><span class=\"p\">.<\/span><span class=\"n\">optim<\/span><span class=\"p\">.<\/span><span class=\"n\">swa_utils<\/span><span class=\"p\">.<\/span><span class=\"n\">update_bn<\/span><span class=\"p\">(<\/span><span class=\"n\">train_loader<\/span><span class=\"p\">,<\/span> <span class=\"n\">swa_model<\/span><span class=\"p\">)<\/span>\n<span class=\"n\">swa_model<\/span> <span class=\"o\">=<\/span> <span class=\"n\">swa_model<\/span><span class=\"p\">.<\/span><span class=\"n\">cuda<\/span><span class=\"p\">()<\/span> \n<\/code><\/pre><\/div><\/div>\n\n<p>\ub9c8\uc9c0\ub9c9\uc73c\ub85c \ub2e4\uc2dc <strong>main.py<\/strong>\ub85c \ub3cc\uc544\uc640\uc11c swa_model\uc758 batch normalization parameter\ub4e4\uc744 update\ud574\uc8fc\uba74 \ub429\ub2c8\ub2e4. \uc81c \uad6c\ud604\uccb4\uc5d0\uc11c\ub294 data loader\uac00 CPU\ub85c \ubc1b\uc544\uc628 \ub4a4, training \ud639\uc740 evaluation loop\uc5d0\uc11c cuda tensor\ub85c \ubcc0\ud658\uc744 \ud574\uc8fc\uae30 \ub54c\ubb38\uc5d0 \uc704\uc640 \uac19\uc774 cpu\uc640 gpu\ub97c \uc654\ub2e4\uac14\ub2e4\ud558\ub294 \ubc88\uac70\ub85c\uc6c0\uc774 \uc788\uc2b5\ub2c8\ub2e4. \ub180\ub78d\uac8c\ub3c4 \uc774\uac8c \ub05d\uc785\ub2c8\ub2e4. \uac70\uc758 \uc774\ubc88\uc5d0\ub3c4 10\uc904 \uc815\ub3c4\ub9cc \ucd94\uac00\ud588\ub294\ub370 \ub05d\uc774 \ub0ac\uc8e0? \uc815\ub9d0 \uac04\ub2e8\ud558\uac8c \uae30\uc874 \ucf54\ub4dc\uc5d0 SWA\ub97c \ucd94\uac00\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\uc8fc\uc694 hyper-parameter\ub294 \uc5b8\uc81c\ubd80\ud130 SWA\ub97c \uc2dc\uc791\ud560 \uc9c0 \uacb0\uc815\ud558\ub294 <strong>args.swa_start<\/strong> \uc640, SWA\uc758 Learning Rate\uc778 <strong>args.swa_lr<\/strong>  \uc774\uba70, \uc774 2\uac00\uc9c0 \uac12\uc744 \uc801\ub2f9\ud788 \uc870\uc808\ud574\uac00\uba70 \uc2e4\ud5d8\uc744 \ud574\ubcfc \uc608\uc815\uc785\ub2c8\ub2e4.<\/p>\n\n<p>SWA\ub97c \uc801\uc6a9\ud558\uc5ec \uc2e4\ud5d8\ud558\uae30 \uc704\ud55c \ucee4\ub9e8\ub4dc \ub77c\uc778 \uba85\ub839\uc5b4\ub294 \ub2e4\uc74c\uacfc \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"n\">python<\/span> <span class=\"n\">main<\/span><span class=\"p\">.<\/span><span class=\"n\">py<\/span> <span class=\"o\">--<\/span><span class=\"n\">checkpoint_name<\/span> <span class=\"n\">swa<\/span> <span class=\"o\">--<\/span><span class=\"n\">decay_type<\/span> <span class=\"n\">swa<\/span> <span class=\"o\">--<\/span><span class=\"n\">swa_start<\/span> <span class=\"mi\">90<\/span> <span class=\"o\">--<\/span><span class=\"n\">swa_lr<\/span> <span class=\"mf\">5e-5<\/span><span class=\"p\">;<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<h3 id=\"3-\uc2e4\ud5d8-\uacb0\uacfc\">3. \uc2e4\ud5d8 \uacb0\uacfc<\/h3>\n<p>\uc774\uc81c Baseline Training\uacfc SWA Training\uc758 \uc2e4\ud5d8 \uacb0\uacfc\ub97c \uc124\uba85 \ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n\n<table>\n  <thead>\n    <tr>\n      <th style=\"text-align: center\">Algorithm<\/th>\n      <th style=\"text-align: center\">Test Accuracy<\/th>\n    <\/tr>\n  <\/thead>\n  <tbody>\n    <tr>\n      <td style=\"text-align: center\">Baseline<\/td>\n      <td style=\"text-align: center\">94.10<\/td>\n    <\/tr>\n    <tr>\n      <td style=\"text-align: center\">SWA_90_0.05<\/td>\n      <td style=\"text-align: center\">80.53<\/td>\n    <\/tr>\n    <tr>\n      <td style=\"text-align: center\">SWA_90_5e-4<\/td>\n      <td style=\"text-align: center\">93.87<\/td>\n    <\/tr>\n    <tr>\n      <td style=\"text-align: center\">SWA_90_1e-4<\/td>\n      <td style=\"text-align: center\">94.20<\/td>\n    <\/tr>\n    <tr>\n      <td style=\"text-align: center\">SWA_90_5e-5<\/td>\n      <td style=\"text-align: center\"><strong>94.57<\/strong><\/td>\n    <\/tr>\n    <tr>\n      <td style=\"text-align: center\">SWA_90_1e-5<\/td>\n      <td style=\"text-align: center\">94.23<\/td>\n    <\/tr>\n    <tr>\n      <td style=\"text-align: center\">SWA_75_5e-5<\/td>\n      <td style=\"text-align: center\">94.27<\/td>\n    <\/tr>\n    <tr>\n      <td style=\"text-align: center\">SWA_60_5e-5<\/td>\n      <td style=\"text-align: center\">94.33<\/td>\n    <\/tr>\n  <\/tbody>\n<\/table>\n\n<p>\uc2e4\ud5d8 \uacb0\uacfc, \uc6d0\ub798 \ub17c\ubb38\uc5d0\uc11c\ub294 SGD\uc5d0 \ud070 \uac12\uc758 initial learning rate\uc744 \uc0ac\uc6a9\ud588\uc5c8\ub294\ub370 \uc800\ub294 \uc774\ubc88\uc5d0 Adam\uc744 \uc0ac\uc6a9\ud558\uba74\uc11c 1e-4\uc758 \uc791\uc740 initial learning rate \uac12\uc744 \uc0ac\uc6a9\ud588\uc2b5\ub2c8\ub2e4. \uc774 \ub54c, \ub108\ubb34 \ud070 swa_lr\uc744 \uc0ac\uc6a9\ud558\uba74 \ud559\uc2b5\uc774 \uc81c\ub300\ub85c \ub418\uc9c0 \uc54a\ub294\ub2e4\ub294, \uc5b4\ub5bb\uac8c \ubcf4\uba74 \ub2f9\uc5f0\ud558\uc9c0\ub9cc \ud574\ubd10\uc57c \uc544\ub294 \uc2e4\ud5d8\uc744 \uc9c4\ud589\ud574\ubd24\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\uadf8 \ub4a4\ub85c\ub294 swa_lr \uac12\uc744 5e-4 \ubd80\ud130 \uc808\ubc18\uc529 \uc904\uc5ec\uac00\uba70 1e-5\uae4c\uc9c0 \uc2e4\ud5d8\uc744 \ud574\ubd24\uace0, \uadf8 \uc911\uc5d0 5e-5 \uc77c \ub54c\uac00 \uac00\uc7a5 test accuracy\uac00 \ub192\uc558\uc2b5\ub2c8\ub2e4. \uadf8\ub9ac\uace0, 1e-4\ubcf4\ub2e4 \uc791\uc740 swa_lr\uc744 \uc0ac\uc6a9\ud558\uba74 \ub2e4 Baseline \ubcf4\ub2e4 \uc131\ub2a5\uc774 \uc88b\uc558\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\ub9c8\uc9c0\ub9c9\uc73c\ub860, SWA\uc744 \uc2dc\uc791\ud558\ub294 epoch\uc744 \uc804\uccb4 120 epoch \uc911\uc5d0 \uae30\uc874\uc5d0\ub294 90 epoch\ubd80\ud130 \uc2dc\uc791\uc744 \ud588\ub294\ub370, \uc2dc\uc791 \uc2dc\uc810\uc744 75, 60 epoch\uc73c\ub85c \ubc14\uafe8\ub354\ub2c8 Baseline\ubcf4\ub2e4 \uc131\ub2a5\uc774 \uc88b\uae34 \ud588\uc9c0\ub9cc, 90 epoch\uc744 \uc0ac\uc6a9\ud558\uc600\uc744 \ub54c\ubcf4\ub2e8 \uc88b\uc9c0 \uc54a\uc740 \uacb0\uacfc\ub97c \ubcf4\uc600\uc2b5\ub2c8\ub2e4.<\/p>\n\n<blockquote> \uacb0\ub860 <\/blockquote>\n<p>\uc624\ub298\uc740 \uc9c0\ub09c \uae00\uc5d0 \uc774\uc5b4\uc11c, PyTorch 1.6\uc5d0 \uacf5\uc2dd\uc801\uc73c\ub85c \uc9c0\uc6d0\ub418\uae30 \uc2dc\uc791\ud55c Stochastic Weight Averaging(SWA) \uae30\ub2a5\uc744 Image Classification Codebase\uc5d0 \uad6c\ud604\ud558\uc5ec \uc2e4\ud5d8\uc744 \uc9c4\ud589\ud558\uace0, \uc2e4\ud5d8 \uacb0\uacfc\ub97c \uacf5\uc720 \ub4dc\ub838\uc2b5\ub2c8\ub2e4. \uc774\ubc88\uc5d0\ub3c4 PyTorch\uc5d0\uc11c \uc798 \uad6c\ud604\uc744 \ud574\uc900 \ub355\ubd84\uc5d0 10\uc904 \ub0b4\uc678\uc758 \ucf54\ub4dc\ub9cc \ucd94\uac00\ud558\uba74 \uc27d\uac8c \uc0ac\uc6a9\ud560 \uc218 \uc788\uc5c8\uc73c\uba70, SGD Training \ubfd0\ub9cc \uc544\ub2c8\ub77c Adam Optimizer\ub97c \uc0ac\uc6a9\ud558\uc5ec\ub3c4 SWA\ub97c \uc798 \uc0ac\uc6a9\ud558\uba74 Test Accuracy\ub97c \ub192\uc77c \uc218 \uc788\ub2e4\ub294 \uc2e4\ud5d8 \uacb0\uacfc\ub3c4 \ubcf4\uc5ec\ub4dc\ub838\uc2b5\ub2c8\ub2e4. \uc800\ub294 \uc774 \uae30\ub2a5\ub3c4 \uc774\uc81c \uc790\uc8fc \uc0ac\uc6a9\ud558\uac8c \ub420 \uac83 \uac19\ub124\uc694! \uae34 \uae00 \uc77d\uc5b4 \uc8fc\uc154\uc11c \uac10\uc0ac\ub4dc\ub9ac\uace0, \uad81\uae08\ud55c \uc810 \uc788\uc73c\uba74 \uc5b8\uc81c\ub4e0 \ub313\uae00 \ub0a8\uaca8\uc8fc\uc138\uc694!<\/p>\n\n","<p>\uc548\ub155\ud558\uc138\uc694, \uc624\ub298\uc740 \uc9c0\ub09c <a href=\"https:\/\/hoya012.github.io\/blog\/SWA\/\" target=\"_blank\"><b> \u201cAveraging Weights Leads to Wider Optima and Better Generalization \ub9ac\ubdf0\u201d <\/b><\/a> \uae00\uc5d0 \uc774\uc5b4\uc11c, \uc774 \ub17c\ubb38\uc5d0\uc11c \uc81c\uc548\ud55c Stochastic Weight Averaging(SWA)\uc744 \uc774\uc6a9\ud55c Image Classification \ud29c\ud1a0\ub9ac\uc5bc\uc744 \uc900\ube44\ud588\uc2b5\ub2c8\ub2e4. \uc9c0\ub09c \uae00\uc5d0\uc11c\ub3c4 \ub9d0\uc500\ub4dc\ub838\uc9c0\ub9cc, PyTorch 1.6\uc5d0\uc11c SWA\ub97c \uacf5\uc2dd\uc801\uc73c\ub85c \uc9c0\uc6d0\ud558\uae30 \uc2dc\uc791\ud574\uc11c \uc774\uc81c \uc190 \uc27d\uac8c \uac00\uc838\ub2e4 \uc4f8 \uc218 \uc788\ub294\ub370\uc694, \uadf8\ub798\uc11c \uc624\ub298 \uae00\uc5d0\uc11c\ub294 \uc9e7\uac8c \ucf54\ub4dc\ub97c \uad6c\ud604\ud574\uc11c \uc124\uba85 \ub4dc\ub9ac\uace0 \uc2e4\ud5d8 \uacb0\uacfc\ub97c \uc18c\uac1c\ub4dc\ub9b4 \uc608\uc815\uc785\ub2c8\ub2e4.<\/p>\n\n"],"pubDate":"Thu, 22 Oct 2020 00:00:00 +0000","link":"https:\/\/hoya012.github.io\/\/blog\/Image-Classification-with-Stochastic-Weight-Averaging-Training-PyTorch-Tutorial\/","guid":"https:\/\/hoya012.github.io\/\/blog\/Image-Classification-with-Stochastic-Weight-Averaging-Training-PyTorch-Tutorial\/"},{"title":"Averaging Weights Leads to Wider Optima and Better Generalization \ub9ac\ubdf0","description":["<p>\uc548\ub155\ud558\uc138\uc694, \uc624\ub298\uc740 2018\ub144 UAI \ud559\ud68c\uc5d0\uc11c \ubc1c\ud45c\ub41c <a href=\"https:\/\/arxiv.org\/pdf\/1803.05407.pdf\" target=\"_blank\"><b> \u201cAveraging Weights Leads to Wider Optima and Better Generalization\u201d <\/b><\/a> \ub17c\ubb38\uc744 \ub9ac\ubdf0\ud560 \uc608\uc815\uc785\ub2c8\ub2e4. \uc774 \ub17c\ubb38\uc5d0\uc11c \uc81c\uc548\ud55c \ubc29\ubc95\uc778 Stochastic Weight Averaging(\uc774\ud558 SWA)\ub294 \ub525\ub7ec\ub2dd \ubaa8\ub378\uc758 \uc77c\ubc18\ud654 \uc131\ub2a5\uc744 \ub192\uc5ec\uc8fc\ub294 \uac04\ub2e8\ud55c \uae30\ubc95\uc774\uba70, PyTorch 1.6 \ubc84\uc804\uc5d0\uc11c \uacf5\uc2dd\uc801\uc73c\ub85c \uc9c0\uc6d0\ud558\uac8c \ub418\uc5c8\uc2b5\ub2c8\ub2e4. \ubc29\ubc95\uc774 \ub2e8\uc21c\ud55c\ub370 \uaf64 \ud6a8\uacfc\uc801\uc774\uc5b4\uc11c \uac15\ub825 \ucd94\ucc9c\ub4dc\ub9ac\uba70, PyTorch\ub97c \uc774\uc6a9\ud55c Tutorial\uc740 \ub2e4\uc74c \uae00\uc5d0\uc11c \uc790\uc138\ud788 \ub2e4\ub8e8\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n\n<blockquote> Related Works <\/blockquote>\n<p>\uc774 \ub17c\ubb38\uc744 \uc124\uba85 \ub4dc\ub9ac\uae30 \uc55e\uc11c, \uc774 \ub17c\ubb38\uc774 \uc791\uc131\ub418\uae30 \uc774\uc804\uc5d0 \ub2e4\ub904\uc84c\ub358 \uad00\ub828 \uc5f0\uad6c\ub4e4\uc744 \uba3c\uc800 \uc9e4\ub9c9\ud558\uac8c \uc18c\uac1c\ub4dc\ub9ac\uace0 \uc2dc\uc791\ud558\uaca0\uc2b5\ub2c8\ub2e4. \uc6b0\uc120 deep neural network\ub294 \uae30\ubcf8\uc801\uc73c\ub85c \uc5ec\ub7ec layer\ub97c \uc313\ub294 \uad6c\uc870\ub97c \ud0dd\ud558\ub294\ub370, \uc774\ub7ec\ud55c multilayer network\uc758 loss surface\ub97c \ubd84\uc11d\ud558\ub824\ub294 \uc5f0\uad6c\ub4e4\uc774 \uc9c4\ud589\ub418\uc5b4\uc654\uace0, \uc774\ub97c \uc774\uc6a9\ud574 \ud559\uc2b5\uc774 \ube68\ub9ac \uc218\ub834\ud558\ub294 \ubc29\ubc95, \ud559\uc2b5 \uc548\uc815\uc131\uc744 \ub192\uc774\ub294 \ubc29\ubc95, test set\uc5d0 \ub300\ud55c \uc815\ud655\ub3c4\ub97c \ub192\uc774\ub294 \ubc29\ubc95 \ub4f1 \ub2e4\uc591\ud55c \uc5f0\uad6c\ub4e4\uc774 \uc9c4\ud589\ub418\uc5b4\uc654\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\uc774 \ub17c\ubb38\uc5d0\uc11c\ub294 \uc5b8\uae09\ud558\uc9c0 \uc54a\uc558\uc9c0\ub9cc, \ub300\ud45c\uc801\uc73c\ub85c 2018 NeurIPS\uc5d0\uc11c \ubc1c\ud45c\ub41c <a href=\"https:\/\/papers.nips.cc\/paper\/7515-how-does-batch-normalization-help-optimization.pdf\" target=\"_blank\"><b> \u201cHow does batch normalization help optimization\u201d <\/b><\/a> \ub17c\ubb38\uc774 loss surface \ubd84\uc11d\uc744 \ud1b5\ud574 Batch Normalization\uc774 \uc5b4\ub5bb\uac8c \ud559\uc2b5\uc5d0 \ub3c4\uc6c0\uc744 \uc8fc\ub294\uc9c0\ub97c \uc774\ub860\uc801, \uc2e4\ud5d8\uc801\uc73c\ub85c \uc811\uadfc\ud558\uc600\uc2b5\ub2c8\ub2e4. \uc774 \ub17c\ubb38\uc744 \ub9ac\ubdf0\ud55c <a href=\"https:\/\/www.slideshare.net\/HoseongLee6\/how-does-batch-normalization-help-optimization-paper-review\" target=\"_blank\"><b> \ubc1c\ud45c \uc790\ub8cc<\/b><\/a> \ub3c4 \uac19\uc774 \ubcf4\uc2dc\uba74 \uc88b\uc744 \uac83 \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\ub610\ud55c \ub9c8\ucc2c\uac00\uc9c0\ub85c 2018 NeurIPS\uc5d0\uc11c \ubc1c\ud45c\ub41c <a href=\"https:\/\/papers.nips.cc\/paper\/8095-loss-surfaces-mode-connectivity-and-fast-ensembling-of-dnns.pdf\" target=\"_blank\"><b> \u201cLoss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs\u201d <\/b><\/a> \ub17c\ubb38\uc5d0\uc11c\ub294 Stochastic Gradient Descent(\uc774\ud558, SGD)\uc5d0 \uc758\ud574 \ubc1c\uacac\ub41c local optima\uac00 \uac04\ub2e8\ud55c \uc0c1\uc218\uc758 loss\uc758 curve\ub85c \uc5f0\uacb0\ub41c\ub2e4\ub294 \uac83\uc744 \ubc1d\ud614\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/swa\/1.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc774\ub7ec\ud55c \uad00\ucc30\uc5d0\uc11c Insight\ub97c \uc5bb\uc5b4\uc11c \ud574\ub2f9 \ub17c\ubb38\uc5d0\uc11c <strong>Fast Geometric Ensembling(\uc774\ud558, FGE)<\/strong> \ubc29\ubc95\uc744 \uc81c\uc548\ud558\uc600\uc2b5\ub2c8\ub2e4. \uc774 \ubc29\ubc95\uc740 \ud558\ub098\uc758 DNN\uc744 \ud559\uc2b5\uc2dc\ud0ac \ub54c weight space\uc5d0 \uc704\uce58\ud55c \uc5ec\ub7ec \uc778\uc811\ud55c point\ub4e4\uc744 sampling\ud558\uc5ec ensemble\ud558\ub294 \uae30\ubc95\uc744 \uc81c\uc548\ud558\uc600\uc73c\uba70, \ube44\uc2b7\ud55c \ubc29\ubc95\uc73c\ub85c\ub294 \uc120\ud589 \uc5f0\uad6c\uc778 <a href=\"https:\/\/arxiv.org\/pdf\/1704.00109.pdf\" target=\"_blank\"><b> \u201cSnapshot ensembles: Train 1, get m for free\u201d, 2017 ICLR <\/b><\/a> \uc774 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/swa\/2.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc704\uc758 \uadf8\ub9bc\uc740 Snapshot ensemble\uc744 \ud55c\uc7a5\uc73c\ub85c \uc124\uba85\ud558\ub294 \uadf8\ub9bc\uc785\ub2c8\ub2e4. \uc774\uc81c\ub294 \ub300\uc138\ub85c \uc790\ub9ac\uc7a1\uc740 Cosine Annealing Learning Rate Scheduling\uc5d0 restart\ub97c \uc8fc\ub294 <strong>Cyclic Cosine Annealing<\/strong>\uc744 \ud1b5\ud574 \uc77c\uc815 \uc8fc\uae30\ub9c8\ub2e4 Learning Rate\ub97c \ub2e4\uc2dc \ud06c\uac8c \ud0a4\uc6cc\uc8fc\uba74\uc11c, \ud574\ub2f9 \uc9c0\uc810\uc758 \ubaa8\ub378\uc744 \uc800\uc7a5\ud569\ub2c8\ub2e4. \uc791\uc740 Learning Rate\ub85c \ubbf8\uc138\ud558\uac8c \uc218\ub834\uc744 \ud558\ub2e4\uac00 \uac11\uc790\uae30 Learning Rate\uac00 \ucee4\uc9c0\uba74 loss surface\uc5d0\uc11c \ud06c\uac8c \ubc97\uc5b4\ub09c \uc9c0\uc810\uc73c\ub85c \uaed1\ucda9 \ub6f0\uac8c \ub418\uaca0\uc8e0? \uc774\ub7ec\ud55c \ubc29\uc2dd\uc744 \ud1b5\ud574 \uc2dc\uac04 \ucd95\uc73c\ub85c \uac01\uae30 \ub2e4\ub978 \ud2b9\uc9d5\uc744 \uac16\ub294 \ubaa8\ub378\uc744 \ud55c \ubc88\uc758 \ud559\uc2b5\uc73c\ub85c \uc5bb\uc744 \uc218 \uc788\uac8c \ub429\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/swa\/3.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>FGE\ub294 Snapshot Ensemble\uacfc \uc544\uc774\ub514\uc5b4\ub294 \ube44\uc2b7\ud558\uc9c0\ub9cc \ub2e4\ub978 \ud2b9\uc9d5\uc744 \uac00\uc9c0\uace0 \uc788\uc2b5\ub2c8\ub2e4. \uc6b0\uc120, Cyclic Cosine Annealing \ub300\uc2e0 piecewise linear cyclical learning rate \uae30\ubc95(\uc704\uc758 \uadf8\ub9bc\uc758 \uc67c\ucabd)\uc744 \uc0ac\uc6a9\ud558\uc600\uace0, cycle\uc758 \uc8fc\uae30\ub3c4 Snapshot Ensemble(SSE)\ubcf4\ub2e4 \ud6e8\uc52c \uc9e7\uac8c \uac00\uc838\uac11\ub2c8\ub2e4. \uadf8\ub798\ub3c4 \uc88b\uc740 \uc131\ub2a5\uc744 \ubcf4\uc77c \uc218 \uc788\uc5c8\uace0, \uc2e4\uc81c\ub85c\ub3c4 Snapshot Ensemble\ubcf4\ub2e4 \ub354 \ube68\ub9ac \uc88b\uc740 \ubaa8\ub378\uc744 \ucc3e\uace0, \ub354 \ub192\uc740 test accuracy\ub97c \ub2ec\uc131\ud558\uac8c \ub429\ub2c8\ub2e4.<\/p>\n\n<blockquote> Stochastic Weight Averaging (SWA) <\/blockquote>\n<p>\uc790 \uc774\uc81c \ubcf8\uaca9\uc801\uc73c\ub85c \uc624\ub298\uc758 \ub17c\ubb38\uc5d0\uc11c \uc81c\uc548\ud55c \uae30\ubc95\uc778 Stochastic Weight Averaging (SWA)\ub97c \uc124\uba85 \ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4. \uc2dc\uac04 \ucd95\uc73c\ub85c \ubaa8\ub378\uc744 Ensemble\ud558\ub294 \uc544\uc774\ub514\uc5b4\ub294 \uadf8\ub300\ub85c \uac00\uc838\uc624\ub294\ub370, \uc120\ud589 \uc5f0\uad6c\uc778 FGE\uc640\uc758 \uac00\uc7a5 \ud070 \ucc28\uc774\uc810\uc740 \ubaa8\ub378\uc758 weight\ub97c \uc2dc\uac04 \ucd95\uc73c\ub85c \uc5ec\ub7ec \uac1c \uc800\uc7a5\ud558\ub294 \ub300\uc2e0 \ubaa8\ub378\uc758 weight\ub97c \uc2dc\uac04 \ucd95\uc73c\ub85c \ub204\uc801(running average)\uc2dc\ud0a8\ub2e4\ub294 \uc810\uc774\uba70, \uc774\ub97c \ud1b5\ud574 \uc5ec\ub7ec \uc774\uc810\uc744 \ub204\ub9b4 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc6b0\uc120, \uc131\ub2a5\ub3c4 \uc88b\uc544\uc9c0\ub294\ub370 \uac00\uc7a5 \ud070 \ucc28\uc774\ub294, FGE\ub294 \ubaa8\ub378\uc758 weight\ub97c k\uac1c \uc800\uc7a5\ud55c \ub4a4, k\ubc88 inference\ub97c \ud558\uc5ec prediction \uac12\ub4e4\uc744 ensemble\ud574\uc57c\ud558\ub294 \ubc18\uba74, SWA\ub294 \ud558\ub098\uc758 \ubaa8\ub378 weight\ub9cc \uc800\uc7a5\ud558\uc600\uae30 \ub54c\ubb38\uc5d0 computational overhead\uac00 \uac70\uc758 \uc5c6\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/swa\/4.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc774 \uadf8\ub9bc\uc774 loss surface \uc0c1\uc5d0\uc11c FGE\uc640 SWA\uc758 \ucc28\uc774\ub97c \uc798 \ubcf4\uc5ec\uc8fc\uace0 \uc788\ub294\ub370\uc694, \uc67c\ucabd \uadf8\ub9bc\uc740 3\uac1c\uc758 FGE sample weight\uc758 test error\uc640 1\uac1c\uc758 SWA weight\uc758 test error\ub97c \ub098\ud0c0\ub0b8 \uac83\uc785\ub2c8\ub2e4. FGE\ub97c \ud1b5\ud574 \ud559\uc2b5\ub41c weight\ub4e4\uc740 \uc8fc\ub85c optimal solution\uc758 \uac00\uc7a5 \uc790\ub9ac \ucabd\uc5d0 \uc704\uce58\ud558\ub294 \uacbd\ud5a5\uc744 \ubcf4\uc774\ub294\ub370, \uc774\ub807\uac8c weight space \uc0c1\uc5d0\uc11c 3\uac1c\uc758 weight\ub97c \ud569\uccd0\ubc84\ub9ac\uba74 test error \uac00 \ub354 \ub0ae\uc740 surface\uc5d0 \uc704\uce58\ud560 \uc218 \uc788\uaca0\uc8e0? \uc2e4\uc81c\ub85c \uc800 3\uac1c\uc758 weight \ub4e4\uc744 \ud3c9\uade0\ub0b4\uba74 SWA\uc758 weight\uc640 \uac70\uc758 \ube44\uc2b7\ud55c \uc9c0\uc810\uc5d0 \uc704\uce58\ud558\uac8c \ub429\ub2c8\ub2e4. \ub610\ud55c SGD\uc640 \ube44\uad50\ub97c \ud588\uc744 \ub54c, SGD\uac00 train loss\ub294 \ub354 \ub0ae\uc558\uc9c0\ub9cc, \uc2e4\uc81c\ub85c \uc911\uc694\ud55c test error\ub294 SWA\uac00 \ub354 \ub0ae\uc740 \uac83\uc744 \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \ub610\ud55c SGD\ub294 \uc5ec\ub7ec \uc5f0\uad6c \uacb0\uacfc\ub97c \ud1b5\ud574 optimal point \uadfc\ucc98\uc758 \ub113\uace0 \ud3c9\ud3c9\ud55c region\uc758 \uacbd\uacc4\uc5d0 \uc218\ub834\ud558\ub294 \ud2b9\uc9d5\uc744 \uac00\uc9c0\uace0 \uc788\ub294\ub370, SWA\ub97c \ud1b5\ud574 weight\ub4e4\uc744 averaging \uc2dc\ud0a4\uba74, SGD\ub85c\ub294 \uac08 \uc218 \uc5c6\uc5c8\ub358 optimal point\uc5d0 \uc9c4\uc785\ud560 \uc218 \uc788\uac8c \ud574\uc8fc\ub294 \ud6a8\uacfc\ub97c \ubcfc \uc218 \uc788\ub2e4\uace0 \ud569\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/swa\/5.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>SWA\uc758 \uc54c\uace0\ub9ac\uc998\uc740 \uc704\uc640 \uac19\uc740 pseudo code\ub85c \ub098\ud0c0\ub0bc \uc218 \uc788\uc73c\uba70 \uad49\uc7a5\ud788 \ub2e8\uc21c\ud569\ub2c8\ub2e4. \uae30\uc874 \ud559\uc2b5 \uacfc\uc815\uacfc \ube44\uad50\ud588\uc744 \ub54c, SWA\uc758 \uc2dc\uac04\uacfc \uba54\ubaa8\ub9ac overhead\ub294 \uac70\uc758 \ubb34\uc2dc\ud560 \ub9cc\ud55c \uc218\uc900\uc785\ub2c8\ub2e4. \ucd94\uac00\ub418\ub294 \uba54\ubaa8\ub9ac \uc18c\ubaa8\ub7c9\uc740 DNN weights\uc758 running average\ub97c \uc800\uc7a5\ud560 \ub54c \ubc1c\uc0dd\ud558\ub294 \ub370, \uc804\uccb4 \ud559\uc2b5 \uacfc\uc815\uc5d0\uc11c \ub300\ubd80\ubd84\uc758 \uba54\ubaa8\ub9ac \uc0ac\uc6a9\ub7c9\uc740 weight \ubcf4\ub2e4\ub294 activation\uc744 \uc800\uc7a5\ud560 \ub54c \ubc1c\uc0dd\ud569\ub2c8\ub2e4. \uc989, DNN weight\uc758 running average\ub97c \uc800\uc7a5\ud558\ub294 \uacfc\uc815\uc740 \uc6c3\uc73c\uba74\uc11c \ub118\uae38 \uc218 \uc788\ub294 \uc218\uc900\uc774\uace0 \ub9ce\uc544\uc57c 10% \uc815\ub3c4 \uba54\ubaa8\ub9ac\ub97c \ub354 \uc0ac\uc6a9\ud55c\ub2e4\uace0 \ud569\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/swa\/6.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub610\ud55c \uae30\uc874 \ubc29\uc2dd\uacfc \ube44\uad50\ud588\uc744 \ub54c, \ub9e4 iteration \ub9c8\ub2e4 aggregated weight average\ub97c \uacc4\uc0b0\ud558\ub294 \uc5f0\uc0b0 1\uac1c\ub9cc \ub354 \ucd94\uac00\ub418\uace0, \uae30\uc874 DNN\uc758 weight\uc640 \uc0c8\ub85c \uacc4\uc0b0\ud55c DNN\uc758 weight\ub97c weight sum \ud574\uc8fc\uba74 \ub418\uae30 \ub54c\ubb38\uc5d0 \ud559\uc2b5 \uc2dc\uac04\ub3c4 \uac70\uc758 \ubb34\uc2dc\ud560 \ub9cc\ud55c \uc218\uc900\uc73c\ub85c \uc99d\uac00\ud569\ub2c8\ub2e4.<\/p>\n\n<p>\uc774 \uc678\uc5d0\ub3c4 local optimum\uc758 width\uac00 SGD\ubcf4\ub2e4 SWA\ub97c \uc0ac\uc6a9\ud560 \ub54c \ub354 \ucee4\uc9c4\ub2e4\ub294 \ubd84\uc11d (3.4), SWA\uacfc Ensemble\uc758 \uad00\uacc4\uc5d0 \ub300\ud55c \ubd84\uc11d (3.5), SWA\uc640 Convex Minimization\uc758 \uad00\uacc4\uc5d0 \ub300\ud55c \ubd84\uc11d (3.6) \ub3c4 \ub17c\ubb38\uc5d0 \uc790\uc138\ud788 \uc11c\uc220\ub418\uc5b4 \uc788\uc73c\ub2c8 \uad00\uc2ec \uc788\uc73c\uc2e0 \ubd84\ub4e4\uc740 \uc77d\uc5b4 \ubcf4\uc2dc\ub294 \uac83\uc744 \ucd94\ucc9c \ub4dc\ub9bd\ub2c8\ub2e4. \ub0b4\uc6a9\uc774 \uc27d\uc9c4 \uc54a\uc2b5\ub2c8\ub2e4..<\/p>\n\n<blockquote> \uc2e4\ud5d8 \uacb0\uacfc \ubc0f Discussion <\/blockquote>\n<p>\uc774\uc81c \uc2e4\ud5d8 \uacb0\uacfc\uc5d0 \ub300\ud574 \uc124\uba85 \ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4. \uc6b0\uc120 CIFAR-10, CIFAR-100, ImageNet \ub370\uc774\ud130\uc14b\uc744 \uc0ac\uc6a9\ud558\uc600\uace0, SGD\uc640 FGE\uacfc \uc131\ub2a5\uc744 \ube44\uad50\ud558\uc600\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>Conventional SGD training\uc740 \uc77c\ubc18\uc801\uc778 learning rate decaying scheduling\uc744 \uc0ac\uc6a9\ud558\uc600\uc73c\uba70, \uad6c\uccb4\uc801\uc73c\ub85c\ub294 \uc804\uccb4 \ud559\uc2b5\uc774 B epoch\uc778 \uacbd\uc6b0\uc5d0 0 ~ 0.5B epoch \uad6c\uac04\uc740 \uace0\uc815\ub41c learning rate \uac12\uc744 \uc0ac\uc6a9\ud558\uace0, 0.5B ~ 0.9B \uad6c\uac04\uc5d0\uc11c\ub294 0.01 * learning rate\ub9cc\ud07c linear \ud558\uac8c \uac10\uc18c\uc2dc\ud0a4\uace0, \ub9c8\uc9c0\ub9c9\uc73c\ub85c 0.9B ~ 1B epoch \uad6c\uac04\uc5d0\uc11c\ub294 0.01 * learning rate \uac12\uc744 \uace0\uc815\uc2dc\ucf1c\uc11c \ud559\uc2b5\uc744 \uc2dc\ucf30\ub2e4\uace0 \ud569\ub2c8\ub2e4.<\/p>\n\n<p>FGE\uc640 SWA\ub294 SGD\uc5d0\uc11c \uc0ac\uc6a9\ud55c \uc804\uccb4 epoch \uac12\uc778 B\ub97c <strong>Budget<\/strong> \uc774\ub77c\uace0 \uc815\uc758\ud55c \ub4a4, 1 Budget\uc73c\ub85c \ud559\uc2b5\uc744 \uc2dc\ucf1c\uc11c \uc5bb\uc740 \ubaa8\ub378\uc758 test accuracy\ub97c \uce21\uc815\ud558\uc600\uc2b5\ub2c8\ub2e4. \ub610\ud55c SWA\uc5d0\uc11c\ub294 Budget\uc744 1.25, 1.5\ub85c \uc62c\ub824\uc11c \uc2e4\ud5d8\uc744 \ud558\uc600\uc73c\uba70, Budget\uc744 \ud0a4\uc6cc\uc8fc\uba74 \uc131\ub2a5\uc774 \uc88b\uc544\uc9c0\ub294 \uacbd\ud5a5\uc744 \ubcf4\uc785\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/swa\/7.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc704\uc758 \ud45c\ub294 CIFAR-10, CIFAR-100\uc758 \uc2e4\ud5d8 \uacb0\uacfc\uc774\uba70, \ubaa8\ub4e0 CNN \ubaa8\ub378\uc5d0 \ub300\ud574 3\ubc88 \ubc18\ubcf5 \uc2e4\ud5d8\uc744 \ud558\uc5ec \ud3c9\uade0, \ud45c\uc900\ud3b8\ucc28\ub97c \uacc4\uc0b0\ud558\uc5ec \ub098\ud0c0\ub0c8\uc2b5\ub2c8\ub2e4. \uc2e4\ud5d8 \uacb0\uacfc SWA\uac00 \ubaa8\ub4e0 \uacbd\uc6b0\uc5d0\uc11c SGD\ubcf4\ub2e4 \uc88b\uc740 \uc131\ub2a5\uc744 \ubcf4\uc600\uace0, 1 Budget\uc5d0\uc11c\ub294 CIFAR-100\uc5d0\uc11c\ub294 FGE\uac00 \ub354 \uc88b\uc740 \uc131\ub2a5\uc744 \ubcf4\uc600\uc9c0\ub9cc, Budget\uc744 \ud0a4\uc6cc\uc8fc\uba74 \ube44\uc2b7\ud55c \uc131\ub2a5\uc744 \uc5bb\uc744 \uc218 \uc788\uc5c8\uace0, CIFAR-10\uc5d0\uc11c\ub294 1 Budget\uc5d0\uc11c\ub3c4 SWA\uac00 \ub354 \uc88b\uc740 \uc131\ub2a5\uc744 \ub2ec\uc131\ud560 \uc218 \uc788\uc5c8\uc2b5\ub2c8\ub2e4. FGE\ub294 \ubaa8\ub378\uc744 \uc5ec\ub7ec \uac1c \uc800\uc7a5\ud558\uc5ec Ensemble\ud558\ub294 \ubc18\uba74, SWA\ub294 \ud558\ub098\uc758 \ubaa8\ub378\ub9cc \uc800\uc7a5\ud558\uc5ec Prediction\uc744 \ud558\ub294 \ubc29\uc2dd\uc774\ub2c8, \ub354 \uc801\uc740 \uacc4\uc0b0\ub7c9\uc73c\ub85c \ub354 \ub192\uc740 \uc815\ud655\ub3c4\ub97c \ub2ec\uc131\ud560 \uc218 \uc788\uc5b4\uc11c \ud070 \uc758\ubbf8\uac00 \uc788\ub294 \uac83 \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/swa\/8.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>ImageNet \ub370\uc774\ud130\uc14b\uc5d0\uc11c\ub3c4 SWA\uac00 SGD\ubcf4\ub2e4 \uc6b0\uc218\ud55c \uc131\ub2a5\uc744 \ubcf4\uc774\ub294 \uac83\uc744 \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/swa\/9.PNG\" alt=\"\" \/> \n<\/figure>\n<p>\uc774\ubc88\uc5d0\ub294 learning rate scheduling\uc5d0 \ub530\ub978 SWA\uc758 \uc131\ub2a5 \ubcc0\ud654\ub97c \uce21\uc815\ud558\ub294 \uc2e4\ud5d8\uc744 \uc218\ud589\ud558\uc600\uc2b5\ub2c8\ub2e4. \ubaa8\ub378\uc740 Preact-ResNet-164, \ub370\uc774\ud130\uc14b\uc740 CIFAR-100\uc744 \uc0ac\uc6a9\ud558\uc600\uace0, conventional SGD\ub85c 125 epoch \ud559\uc2b5\uc2dc\ud0a8 \ubaa8\ub378\uc758 weight\ub85c weight\ub97c initialization \ud55c \ub4a4 \uc2e4\ud5d8\uc744 \uc2dc\uc791\ud558\uc600\ub2e4\uace0 \ud569\ub2c8\ub2e4. \uc704\uc758 \uadf8\ub9bc\uc758 \uc810\uc120\uc740 conventional SGD\ub85c 150 epoch\uc744 \ub3cc\ub824\uc11c \uc5bb\uc740 \uacb0\uacfc\uc785\ub2c8\ub2e4.<\/p>\n\n<p>\uc2e4\ud5d8\uc740 Constant Learning Rate Schedule 4\uac00\uc9c0 \uac12, Cyclical Learning Rate Schedule 4\uac00\uc9c0 \uac12\uc744 \uc0ac\uc6a9\ud558\uc600\uace0 \ub300\uccb4\ub85c Constant Learning Rate Schedule\uc744 \uc0ac\uc6a9\ud558\uba74 \ub354 \ube68\ub9ac \uc218\ub834\ud558\ub294 \uacbd\ud5a5\uc744 \ubcf4\uc600\uace0, Test error\ub3c4 \ub354 \ub0ae\uc740 \uac83\uc744 \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc989, \uad73\uc774 \uc120\ud589 \uc5f0\uad6c\ub4e4\uc5d0\uc11c \uc0ac\uc6a9\ud588\ub358 Cyclical Learning Rate Schedule \ubc29\uc2dd\uc744 \uad73\uc774 \uc0ac\uc6a9\ud558\uc9c0 \uc54a\uc544\ub3c4 \uc88b\uc740 \uc131\ub2a5\uc744 \ub0bc \uc218 \uc788\ub2e4\ub294 \ub73b\uc785\ub2c8\ub2e4. \uc774\ub7ec\uba74 Cycle length\ub77c\ub294 hyper parameter\ub97c \ud558\ub098 \uc904\uc77c \uc218 \uc788\ub2e4\ub294 \uc7a5\uc810\uc774 \uc0dd\uae41\ub2c8\ub2e4.<\/p>\n\n<p>\uc800\uc790\ub4e4\uc740 \uc774\ub7ec\ud55c \ubc1c\uacac\uc5d0\uc11c \ub098\uc544\uac00\uc11c, \uc544\uc608 DNN \ubaa8\ub378\uc744 learning rate scheduling \uc5c6\uc774 \uace0\uc815\ub41c \uac12\uc73c\ub85cscratch\ub85c\ubd80\ud130 \ud559\uc2b5\uc2dc\ud0ac \uc218 \uc788\uc744\uc9c0 \uc2e4\ud5d8\uc744 \uc218\ud589\ud558\uc600\uc2b5\ub2c8\ub2e4. \uc774\ubc88\uc5d4 Wide ResNet-28-10\uc5d0 CIFAR100\uc744 \uc774\uc6a9\ud558\uc5ec \uc2e4\ud5d8\ud558\uc600\uace0, 0.05 learning rate\ub85c 300 epoch\uc744 \ud559\uc2b5\ud558\uc600\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/swa\/10.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>SWA\ub294 SGD\uc758 140 epoch \uc9c0\uc810\ubd80\ud130 300 epoch \uc9c0\uc810\uae4c\uc9c0\uc758 weight\ub97c average\ud558\uc600\uace0, \uc2e4\ud5d8 \uacb0\uacfc constant learning rate scheduling\uc744 \ud558\uc600\uc744 \ub54c\uc758 SGD(\ucd08\ub85d \uc120), decaying learning rate scheduling\uc744 \ud558\uc600\uc744 \ub54c\uc758 SGD(\ud30c\ub780\uc120)\ubcf4\ub2e4 constant learning rate scheduling + SWA\ub97c \uc0ac\uc6a9\ud558\uc600\uc744 \ub54c \ub354 \ube60\ub974\uac8c \uc218\ub834\ud558\uace0 \ub354 \ub0ae\uc740 test error\ub97c \ubcf4\uc774\ub294 \uac83\uc744 \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>Constant learning rate scheduling\uc744 \uc774\uc6a9\ud558\uba74 SGD\ub294 oscillate\ud558\ub294 \ubc18\uba74 SWA\ub294 \uc6d0\ud65c\ud558\uac8c \ud559\uc2b5\uc774 \ub418\ub294 \uac83\uc744 \uad00\ucc30\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc800\uc790\ub4e4\uc740 \ub9d0\ubbf8\uc5d0 \uc2e4\uc6a9\uc801\uc778 \uad00\uc810\uc5d0\uc11c SWA\ub97c \uc0ac\uc6a9\ud560 \ub54c, \uc774 \uc2e4\ud5d8\uacfc \uac19\uc774 \ucd08\ubc18 \ubd80\ubd84\uc740 conventional\ud55c training\uc744 \ud1b5\ud574 \ud559\uc2b5\uc744 \uc2dc\ud0a8 \ub4a4, \uc911\uac04 \uc9c0\uc810\ubd80\ud130 SWA\ub97c \uc0ac\uc6a9\ud558\ub294 \uac83\uc774, \ucc98\uc74c\ubd80\ud130 SWA\ub97c \uc0ac\uc6a9\ud558\ub294 \uac83\ubcf4\ub2e4 \ub354 \ube60\ub974\uace0 \ub354 \uc548\uc815\uc801\uc73c\ub85c \ud559\uc2b5\uc744 \uc2dc\ud0ac \uc218 \uc788\ub2e4\uace0 \uc548\ub0b4\ud558\uace0 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<blockquote> \uacb0\ub860 <\/blockquote>\n<p>\uc624\ub298\uc740 2018\ub144 UAI \ud559\ud68c\uc5d0\uc11c \ubc1c\ud45c\ub41c <a href=\"https:\/\/arxiv.org\/pdf\/1803.05407.pdf\" target=\"_blank\"><b> \u201cAveraging Weights Leads to Wider Optima and Better Generalization\u201d <\/b><\/a> \ub17c\ubb38\uc744 \uac04\ub2e8\ud788 \ub9ac\ubdf0\ud558\uc600\ub294\ub370\uc694, Ensemble \ub290\ub08c\uc774 \ub098\uc9c0\ub9cc \uc5c4\ubc00\ud558\uac8c\ub294 Ensemble\uc740 \uc544\ub2c8\uace0, Single Model\uc744 \uc0ac\uc6a9\ud558\ub294 \uae30\ubc95\uc774\uba70, \ub2e8\uc21c\ud558\uc9c0\ub9cc \ub9e4\uc6b0 \ud6a8\uacfc\uc801\uc774\uc5b4\uc11c \ud604\uc5c5\uc774\ub098 Kaggle \ub4f1\uc5d0\uc11c \uc790\uc8fc \uc0ac\uc6a9\ub420 \uc218 \uc788\uc744 \uac83\uc73c\ub85c \uc0dd\uac01\ub429\ub2c8\ub2e4. \ub610\ud55c \uad6c\ud604\uc774 \uadf8\ub807\uac8c \uc5b4\ub835\uc9c0 \uc54a\uace0, PyTorch\uc5d0\uc11c\ub294 \uc774\uc81c \uacf5\uc2dd\uc801\uc73c\ub85c \uc9c0\uc6d0\ud558\ub294 \ub9cc\ud07c \ub3c5\uc790 \uc5ec\ub7ec\ubd84\ub4e4\ub3c4 \ud55c \ubc88\ucbe4 \uc0b4\ud3b4\ubcf4\uc2dc\ub294 \uac83\uc744 \uad8c\uc7a5 \ub4dc\ub9ac\uace0, \ub2e4\uc74c \uae00\uc5d0\uc11c\ub294 PyTorch\uc758 SWA \uae30\ub2a5\uc744 \uc0b4\ud3b4\ubcf4\ub294 Tutorial \uae00\ub85c \ucc3e\uc544 \ubd59\uaca0\uc2b5\ub2c8\ub2e4. \uae34 \uae00 \uc77d\uc5b4 \uc8fc\uc154\uc11c \uac10\uc0ac\ud569\ub2c8\ub2e4.<\/p>\n\n","<p>\uc548\ub155\ud558\uc138\uc694, \uc624\ub298\uc740 2018\ub144 UAI \ud559\ud68c\uc5d0\uc11c \ubc1c\ud45c\ub41c <a href=\"https:\/\/arxiv.org\/pdf\/1803.05407.pdf\" target=\"_blank\"><b> \u201cAveraging Weights Leads to Wider Optima and Better Generalization\u201d <\/b><\/a> \ub17c\ubb38\uc744 \ub9ac\ubdf0\ud560 \uc608\uc815\uc785\ub2c8\ub2e4. \uc774 \ub17c\ubb38\uc5d0\uc11c \uc81c\uc548\ud55c \ubc29\ubc95\uc778 Stochastic Weight Averaging(\uc774\ud558 SWA)\ub294 \ub525\ub7ec\ub2dd \ubaa8\ub378\uc758 \uc77c\ubc18\ud654 \uc131\ub2a5\uc744 \ub192\uc5ec\uc8fc\ub294 \uac04\ub2e8\ud55c \uae30\ubc95\uc774\uba70, PyTorch 1.6 \ubc84\uc804\uc5d0\uc11c \uacf5\uc2dd\uc801\uc73c\ub85c \uc9c0\uc6d0\ud558\uac8c \ub418\uc5c8\uc2b5\ub2c8\ub2e4. \ubc29\ubc95\uc774 \ub2e8\uc21c\ud55c\ub370 \uaf64 \ud6a8\uacfc\uc801\uc774\uc5b4\uc11c \uac15\ub825 \ucd94\ucc9c\ub4dc\ub9ac\uba70, PyTorch\ub97c \uc774\uc6a9\ud55c Tutorial\uc740 \ub2e4\uc74c \uae00\uc5d0\uc11c \uc790\uc138\ud788 \ub2e4\ub8e8\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n\n"],"pubDate":"Wed, 14 Oct 2020 00:00:00 +0000","link":"https:\/\/hoya012.github.io\/\/blog\/SWA\/","guid":"https:\/\/hoya012.github.io\/\/blog\/SWA\/"},{"title":"ECCV 2020 Virtual Conference \ucc38\uc11d \ud6c4\uae30 \ubc0f \ud504\ub85c\uadf8\ub7a8 \uc18c\uac1c","description":["<p>\uc548\ub155\ud558\uc138\uc694, \uc624\ub298\uc740 \uc9c0\ub09c 8\uc6d4 23\uc77c(\uc77c) ~ 8\uc6d4 28\uc77c(\uae08) 6\uc77c\uac04 \uc9c4\ud589\ub41c European Conference on Computer Vision(\uc774\ud558 ECCV) 2020 \ud559\ud68c\ub97c Virtual\ub85c \ucc38\uc11d\ud558\uba70 \ub290\ub080 \uc810\ub4e4\uc744 \uacf5\uc720 \ub4dc\ub9ac\uace0, \uc8fc\uc694 \ud504\ub85c\uadf8\ub7a8\ub4e4\uc744 \uc18c\uac1c \ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>ECCV\ub294 2\ub144\ub9c8\ub2e4 \uc5f4\ub9ac\uba70 \uc9dd\uc218 \ub144\ub3c4\uc5d0 \uc5f4\ub9ac\ub294 Computer Vision \ud559\ud68c\uc785\ub2c8\ub2e4. \ube44\uc2b7\ud55c \ud559\ud68c\ub85c International Conference on Computer Vision(ICCV)\uac00 \uc788\uc8e0. ICCV\ub3c4 2\ub144\ub9c8\ub2e4 \uc5f4\ub9ac\uba70 \ud640\uc218 \ub144\ub3c4\uc5d0 \uc5f4\ub9bd\ub2c8\ub2e4. \uc800\ub294 \uc6d0\ub798 \uc62c\ud574 \uc601\uad6d Glasgow\uc5d0\uc11c \uc5f4\ub9ac\ub294 ECCV\ub97c \ud604\uc7a5\uc5d0\uc11c \ucc38\uc11d\ud558\ub824\uace0 8\uc6d4\ub2ec\ub9cc \uae30\ub2e4\ub9ac\uace0 \uc788\uc5c8\ub294\ub370 \uc804\uc138\uacc4\uc801\uc73c\ub85c \ud37c\uc9c4 COVID-19\ub85c \uc778\ud574 \uc62c\ud574 \ubaa8\ub4e0 Machine Learning, Computer Vision \uad00\ub828 \ud559\ud68c\ub4e4\uc774 Virtual\ub85c \uc9c4\ud589\ub418\uc5c8\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\uc800\ub294 Virtual\ub85c \ud559\ud68c\uc5d0 \ucc38\uc11d\ud55c \uc801\uc774 \ucc98\uc74c\uc774\ub77c \uc9d1\uc5d0\uc11c \ub4e3\ub294 \ud559\ud68c\uac00 \uacfc\uc5f0 \uc5bc\ub9c8\ub098 \ud6a8\uc728\uc801\uc77c\uc9c0 \uad81\uae08\ud588\ub294\ub370\uc694, \uc9c1\uc811 \uc77c\uc8fc\uc77c\uac04 \uc9d1(\ud639\uc740 \ud68c\uc0ac)\uc5d0\uc11c \ud559\ud68c\ub97c \ucc38\uc11d\ud55c \ud6c4\uae30\ub97c \ub4e4\ub824\ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n\n<blockquote> ECCV 2020 \uc8fc\uc694 \ud1b5\uacc4 <\/blockquote>\n<p>\uc774\ubc88 <a href=\"https:\/\/eccv2020.eu\/\" target=\"_blank\"><b> ECCV 2020<\/b><\/a>\uc740 \uc62c\ud574\ub85c 16\ubc88\uc9f8 \uc5f4\ub838\uace0 \ud559\ud68c\uac00 \uac1c\ucd5c\ub41c \uccab\ub0a0 Opening \ud589\uc0ac\uc5d0\uc11c \uc8fc\uc694 \ud1b5\uacc4\uce58\ub4e4\uc744 \uacf5\uac1c\ud558\uc600\uc2b5\ub2c8\ub2e4. \uc0ac\uc9c4 \uc790\ub8cc\ub294 <a href=\"https:\/\/twitter.com\/CSProfKGD\/status\/1297892662687797255\" target=\"_blank\"><b> Kosta Derpanis \ub77c\ub294 \ubd84\uc758 \ud2b8\uc704\ud130\uc758 \uac8c\uc2dc\ubb3c<\/b><\/a>\uc5d0\uc11c \uc778\uc6a9\ud558\uc600\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/eccv2020\/stat1.jpg\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc6b0\uc120 \ucd1d 5150\ud3b8\uc758 submission\uc774 \uc788\uc5c8\uace0, \uadf8 \uc911 1360\ud3b8\uc774 accept \ub418\uc5c8\uc73c\uba70 accepted rate\ub294 \uc57d 26%\ub97c \ubcf4\uc600\uc2b5\ub2c8\ub2e4. 2014\ub144\uc774 28%, 2016\ub144\uc774 27%, 2018\ub144\uc774 32%\uc600\uace0 \uc62c\ud574\uac00 26%\ub85c \ub300\uccb4\ub85c \ube44\uc2b7\ud55c \ube44\uc728\uc744 \uc720\uc9c0\ud558\uace0 \uc788\uc2b5\ub2c8\ub2e4. \uc804\uccb4 Accepted paper \uc911 Oral paper\ub294 7.5%, Spotlight paper\ub294 11.8%\uc758 \ube44\uc728\uc744 \ubcf4\uc774\uace0 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/eccv2020\/stat2.jpg\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub2e4\uc74c\uc740 \uc804\uccb4 \uc81c\ucd9c\ub41c \ub17c\ubb38\ub4e4\uc758 \uc5f0\uad6c \ubd84\uc57c\ub97c \ubd84\ub958\ud55c \uc790\ub8cc\uc785\ub2c8\ub2e4. Deep Learning\uc758 \uc751\uc6a9, \ubc29\ubc95\ub860, \uc774\ub860\uc744 \ub2e4\ub8ec \ub17c\ubb38\uc774 \uc8fc\ub97c \uc774\ub918\uace0, \uc544\ubb34\ub798\ub3c4 Computer Vision \ud559\ud68c\uc774\ub2e4 \ubcf4\ub2c8 \uc778\uc2dd \ucabd \ub17c\ubb38\ub3c4 \ub9ce\uc740 \ube44\uc728\uc744 \ubcf4\uc774\uace0 \uc788\uc2b5\ub2c8\ub2e4. Unsupervised Learning \ub17c\ubb38\uc774 352\ud3b8\uc774\ub098 \ub418\ub294 \uc810\ub3c4 \uc778\uc0c1\uae4a\ub124\uc694.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/eccv2020\/stat3.jpg\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub2e4\uc74c\uc740 Accepted paper\ub4e4\uc758 \uc800\uc790\ub4e4\uc774 \uc18d\ud55c \uae30\uad00\uc774 \ub300\ud55c \uc790\ub8cc\uc785\ub2c8\ub2e4. \uad6c\uae00\uc774 1\uc704\ub97c \ucc28\uc9c0\ud558\uc600\uace0 \ud398\uc774\uc2a4\ubd81\uc774 5\uc704\ub97c, \ub9c8\uc774\ud06c\ub85c\uc18c\ud504\ud2b8\uac00 7\uc704\ub97c, \ud654\uc6e8\uc774\uac00 11\uc704\ub97c \ucc28\uc9c0\ud558\uc600\uc2b5\ub2c8\ub2e4. \ub300\uccb4\ub85c \ud559\uacc4\uc5d0 \uacc4\uc2e0 \ubd84\ub4e4\uc758 \ube44\uc911\uc774 \ub192\uc558\uc73c\uba70 Computer Vision\uc740 \uc5ed\uc2dc \uc911\uad6d\uc774 \uac15\uc138\uc784\uc744 \ubcf4\uc5ec\uc8fc\uace0 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/eccv2020\/stat4.jpg\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub9c8\uc9c0\ub9c9\uc73c\ub85c 2\ub144\uc804\uacfc \ube44\uad50\ud588\uc744 \ub54c \ud559\ud68c\uc5d0 \uc81c\ucd9c\ub41c \ub17c\ubb38\uc758 \ud3b8\uc218\uac00 \uc5bc\ub9c8\ub098 \uc99d\uac00\ud588\ub294\uc9c0\ub97c \ubcf4\uc5ec\uc8fc\uace0 \uc788\uc2b5\ub2c8\ub2e4. \uc6b0\uc120 ECCV 2018\uc5d0 \ube44\ud574 \uc81c\ucd9c\ub41c \ub17c\ubb38 \ud3b8\uc218\uac00 2.1\ubc30 \ub298\uc5c8\uc73c\uba70, CVPR 2020\uacfc \ube44\uad50\ud558\uc600\uc744 \ub54c 2\ub144\uc804\ubcf4\ub2e4 \uc81c\ucd9c\ub41c \ub17c\ubb38\uc758 \uc218\uac00 \ube44\uc2b7\ud574\uc84c\uc74c\uc744 \ubcf4\uc5ec\uc8fc\uace0 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<blockquote> ECCV \uc8fc\uc694 \ud504\ub85c\uadf8\ub7a8 \uc18c\uac1c \ubc0f Virtual \ucc38\uc11d \ud6c4\uae30 <\/blockquote>\n\n<p>\ub300\ubd80\ubd84\uc758 \ud559\ud68c\ub294 \ud06c\uac8c 6\uac00\uc9c0 \ud504\ub85c\uadf8\ub7a8\uc73c\ub85c \ubd84\ub958\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n<ul>\n  <li>Oral, Spotlight \ub17c\ubb38\ub4e4\uc758 \uad6c\ub450 \ubc1c\ud45c\uac00 \uc9c4\ud589\ub418\ub294 <strong>Main Conference<\/strong><\/li>\n  <li>\ub300\ubd80\ubd84\uc758 \ub17c\ubb38\ub4e4\uc744 \uc815\ud574\uc9c4 \uc2dc\uac04\uc5d0 Poster\uc640 \ud568\uaed8 \ubc1c\ud45c\ud558\uace0 \uc2e4\uc2dc\uac04\uc73c\ub85c \uc9c8\uc758 \uc751\ub2f5\ud558\ub294 <strong>Poster Session<\/strong><\/li>\n  <li>\ud2b9\uc815\ud55c \uc8fc\uc81c\ub97c \uc815\ud574 \ub193\uace0 \uadf8 \uc8fc\uc81c\uc640 \uad00\ub828\ub41c \uc5f0\uad6c\ub4e4\uc744 \uc18c\uac1c\ud558\uace0 \uacbd\uc6b0\uc5d0 \ub530\ub77c Challenge\ub3c4 \uac1c\ucd5c\ud558\ub294 <strong>Workshop<\/strong><\/li>\n  <li>Workshop\uacfc \ube44\uc2b7\ud558\uac8c \ud2b9\uc815\ud55c \uc8fc\uc81c\ub97c \uc815\ud574\ub450\uc9c0\ub9cc \uad50\uc721\uc758 \ubaa9\uc801\uc774 \ud06c\uace0, \uc885\uc885 \uc2e4\uc2b5(Hands-on Training)\ub3c4 \uac19\uc774 \uc9c4\ud589\ud558\ub294 <strong>Tutorial<\/strong><\/li>\n  <li>\ubcf8\uc778\ub4e4\uc758 \uc5f0\uad6c\ub97c Live Demo\ub85c \ubcf4\uc5ec\uc8fc\ub294 <strong>Demo Session<\/strong><\/li>\n  <li>\ud559\ud68c\uc5d0 \uc77c\uc815 \uae08\uc561\uc758 \ub3c8\uc744 \ub0b4\uace0 \uc2a4\ud3f0\uc11c(\ub4f1\uae09\uc774 \ub098\ub258\uc5b4\uc838 \uc788\uc74c)\ub85c \ucc38\uc5ec\ud558\uc5ec \uae30\uc5c5\uc744 \ud64d\ubcf4\ud558\ub294 <strong>Exhibition<\/strong><\/li>\n<\/ul>\n\n<p>\uc774 \ubaa8\ub4e0 \ud504\ub85c\uadf8\ub7a8\uc774 Virtual\ub85c \uc9c4\ud589\uc774 \ub418\uc5c8\uc73c\uba70, \ud559\ud68c\uc5d0 \ub4f1\ub85d\ud55c \uc0ac\ub78c\ub4e4\uc740 \ubcc4\ub3c4\uc758 \ud398\uc774\uc9c0\uc5d0 \uc811\uc18d\ud560 \uc218 \uc788\uac8c \ub429\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/eccv2020\/lobby.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc0ac\uc774\ud2b8\uc5d0 \uc811\uc18d\uc744 \ud558\uba74 \uc774\ub7f0 \ud654\uba74\uc5d0\uc11c \uc2dc\uc791\ud569\ub2c8\ub2e4. Lobby\uc5d0\uc11c \uc6d0\ud558\ub294 \ud504\ub85c\uadf8\ub7a8\uc744 \ud074\ub9ad\ud558\uc5ec \ub4e3\ub294 \ubc29\uc2dd\uc785\ub2c8\ub2e4. \uc5b4\ub5bb\uac8c \ubcf4\uba74 \ub354\uc6b4 \uc5ec\ub984 \ubc1c\ud45c \ub4e4\uc73c\ub7ec \uc774\uacf3 \uc800\uacf3 \uac78\uc5b4 \ub2e4\ub2c8\uc9c0 \uc54a\uace0 \ucef4\ud4e8\ud130 \uc55e\uc5d0\uc11c \ud074\ub9ad \uba87 \ubc88 \ud574\uc8fc\uba74 \ub41c\ub2e4\ub294 \uc810\uc774 \uc7a5\uc810\uc774\ub77c\uace0 \uc0dd\uac01\ud569\ub2c8\ub2e4. \ub2e4\ub9cc \uc544\uc26c\uc6b4 \uc810\ub3c4 \uc788\uc5c8\uc2b5\ub2c8\ub2e4. \uc774\uc81c \uac01 \ud504\ub85c\uadf8\ub7a8\ub9c8\ub2e4 Virtual\ub85c \ucc38\uc11d\ud574\ubcf4\uace0 \ub290\ub080 \uc810\ub4e4\uc744 \uacf5\uc720 \ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"main-conference\">Main Conference<\/h3>\n<figure>\n\t<img src=\"\/assets\/img\/eccv2020\/oral_session.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>Main Conference\ub294 \uc6d4\uc694\uc77c\ubd80\ud130 \uae08\uc694\uc77c\uae4c\uc9c0 5\uc77c\uac04 \uc9c4\ud589\uc774 \ub418\uc5c8\uace0, Oral Paper\uc640 Spotlight Paper\uc5d0 \uc120\uc815\ub41c \uc800\uc790\ub4e4\uc740 \uc815\ud574\uc9c4 \uc2dc\uac04\uc5d0 \uc2e4\uc2dc\uac04\uc73c\ub85c 1\ubc88 \ubc1c\ud45c\ub97c \ud558\ub294 \ubc29\uc2dd\uc73c\ub85c \uc9c4\ud589\uc774 \ub418\uc5c8\uc2b5\ub2c8\ub2e4. \uc624\ud504\ub77c\uc778 \ud559\ud68c\uc5d0\uc11c\ub3c4 Main Conference Room\uc5d0 \uc790\ub9ac\uac00 \uc5c6\uc73c\uba74, \ube48 Conference Room\uc5d0\uc11c \ud654\uba74\uc744 \ub744\uc6cc\uc900 \ucc44\ub85c \uc2e4\uc2dc\uac04 \uc1a1\ucd9c\uc744 \ud574\uc8fc\ub294\ub370, \uc774\uac78 \uc9d1\uc5d0\uc11c \ubcf4\ub294 \ub290\ub08c\uc774\ub77c\uace0 \uc0dd\uac01\ud558\uc2dc\uba74 \uc88b\uc744 \uac83 \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\uc694\uc77c\ub9c8\ub2e4 \uc8fc\uc81c\uac00 \ubc14\ub00c\uc5c8\uace0, \uc911\uac04 \uc911\uac04 Network Break \ud0c0\uc784\uc5d0\ub294 Industry Session\ub3c4 \uc9c4\ud589\uc774 \ub418\uc5c8\uc2b5\ub2c8\ub2e4. \uac01 \uc694\uc77c\ub9c8\ub2e4 \uc5b4\ub5a4 \ubc1c\ud45c\ub4e4\uc774 \uc9c4\ud589\ub418\uc5c8\ub294\uc9c0 \uad81\uae08\ud558\uc2e4 \ubd84\ub4e4\uc744 \uc704\ud574 \ub9c1\ud06c\ub97c \ucca8\ubd80 \ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n<ul>\n  <li><a href=\"https:\/\/eccv2020.eu\/wp-content\/uploads\/2020\/08\/ECCV-Programme-Monday.pdf\" target=\"_blank\"><b> 8\/24 \uc6d4\uc694\uc77c <\/b><\/a><\/li>\n  <li><a href=\"https:\/\/eccv2020.eu\/wp-content\/uploads\/2020\/08\/ECCV-Programme-Tuesday.pdf\" target=\"_blank\"><b> 8\/25 \ud654\uc694\uc77c <\/b><\/a><\/li>\n  <li><a href=\"https:\/\/eccv2020.eu\/wp-content\/uploads\/2020\/08\/ECCV-Programme-Wednesday.pdf\" target=\"_blank\"><b> 8\/26 \uc218\uc694\uc77c <\/b><\/a><\/li>\n  <li><a href=\"https:\/\/eccv2020.eu\/wp-content\/uploads\/2020\/08\/ECCV-Programme-Thursday.pdf\" target=\"_blank\"><b> 8\/27 \ubaa9\uc694\uc77c <\/b><\/a><\/li>\n  <li><a href=\"https:\/\/eccv2020.eu\/wp-content\/uploads\/2020\/08\/ECCV-Programme-Friday-2.pdf\" target=\"_blank\"><b> 8\/28 \uae08\uc694\uc77c <\/b><\/a><\/li>\n<\/ul>\n\n<h3 id=\"poster-session\">Poster Session<\/h3>\n<p>\uc800\ub294 \uc8fc\ub85c \uc624\ud504\ub77c\uc778 \ud559\ud68c\uc5d0 \ucc38\uc11d\ud560 \ub54c, \uc544\ubb34\ub798\ub3c4 \uc694\uc998 \ud559\ud68c\uc758 \uaddc\ubaa8\uac00 \ub108\ubb34 \ucee4\uc9c0\ub2e4 \ubcf4\ub2c8 \ubaa8\ub4e0 \ub17c\ubb38\uc744 \ub2e4 \uc0b4\ud3b4\ubcf4\uae30\uc5d4 \ubb34\ub9ac\uac00 \ucee4\uc11c, \uc8fc\uc694 \ub17c\ubb38\ub4e4\uc744 \ubbf8\ub9ac \ucd94\ub824\uc11c \uac04\ub2e8\ud788 \uc77d\uc5b4\ubcf4\uace0 \ucc38\uc11d\uc744 \ud569\ub2c8\ub2e4. \uadf8\ub9ac\uace0 Poster Session\uc744 \ube60\ub974\uac8c \ub3cc\uc544\ub2e4\ub2c8\uba74\uc11c \uc7ac\ubbf8\uc788\uc5b4 \ubcf4\uc774\ub294 Poster\ub97c \ucc3e\uc73c\uba74 \uc77c\ub2e8 \uc0ac\uc9c4\uc744 \ucc0d\uace0, \uc11c\uc788\ub294 \uc800\uc790\uc5d0\uac8c \uc124\uba85\uc744 \ud574\ub2ec\ub77c\uace0 \ubd80\ud0c1\ub4dc\ub9ac\uace0, \uad81\uae08\ud55c \uc810\uc744 \uc9c8\ubb38\ud558\uba74\uc11c \ub3cc\uc544\ub2e4\ub2c8\ub294 \uc7ac\ubbf8\uac00 \uc788\uc5c8\ub294\ub370\uc694. \uac1c\uc778\uc801\uc73c\ub85c Virtual Conference\ub85c \ubc14\ub00c\uba74\uc11c \uac00\uc7a5 \uc544\uc26c\uc6e0\ub358 \ud504\ub85c\uadf8\ub7a8\uc774 \ubc14\ub85c Poster Session\uc774\uc5c8\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/eccv2020\/papers.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc6b0\uc120 Lobby\uc758 Papers and Presentations\ub97c \ub4e4\uc5b4\uac00\uba74 \ubaa8\ub4e0 \ub17c\ubb38\ub4e4\uc758 \uc81c\ubaa9\uacfc \ub17c\ubb38(.pdf), 10\ubd84 \ubd84\ub7c9\uc758 \uc804\uccb4 \ubc1c\ud45c Video, 1\ubd84 \ubd84\ub7c9\uc758 Short Video\ub97c \ud655\uc778\ud560 \uc218 \uc788\uace0, \uc8fc\uc694 \uc5f0\uad6c \uc8fc\uc81c\ub4e4\ub07c\ub9ac \ubaa8\uc5ec \uc788\ub294 \uc810\uc740 \uc88b\uc558\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/eccv2020\/poster_session.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub2e4\ub9cc, \uc81c\uac00 \uc6d0\ud588\ub358 Poster Session\uc740 \uc624\ud504\ub77c\uc778\uc5d0\uc11c \ub113\uc740 \uacf5\uac04\uc5d0 \ud3ec\uc2a4\ud130\uac00 \ubc30\uce58 \ub418\uc5b4\uc788\ub294 \uac83\ucc98\ub7fc, \uc628\ub77c\uc778\uc73c\ub85c\ub3c4 \ube60\ub974\uac8c \ud3ec\uc2a4\ud130 \ud558\ub098 \ud558\ub098\uc529 \ub118\uae30\uba70 \ubcfc \uc218 \uc788\ub294 \uac83\uc744 \uae30\ub300\ud588\ub294\ub370 \uadf8\ub807\uc9c4 \uc54a\uc558\uc2b5\ub2c8\ub2e4. \uc815\ud574\uc9c4 \uc2dc\uac04\uc5d0 Poster Session\uc5d0 \ub4e4\uc5b4\uac00\uba74 \uc704\uc640 \uac19\uc774 \uac01 \ub17c\ubb38 \ub9c8\ub2e4 \ubcc4\ub3c4\uc758 \ud0ed\uc774 \uc874\uc7ac\ud558\uc600\uace0, Launch \ubc84\ud2bc\uc744 \ub204\ub974\uba74 zoom\uc73c\ub85c \uc5f0\uacb0\ub418\ub294 \ubc29\uc2dd\uc774\uc5c8\uc2b5\ub2c8\ub2e4. \ub4e4\uc5b4\uac00\uba74 \uc800\uc790\uc640 \uc18c\ud1b5\uc744 \ud560 \uc218 \uc788\uae34 \ud558\uc9c0\ub9cc zoom \ud2b9\uc131 \uc0c1 \uc811\uc18d\ud558\ub294\ub370 \ub51c\ub808\uc774\uac00 \uc788\ub2e4 \ubcf4\ub2c8 \ube60\ub974\uac8c \ud6d1\ub294 \uac83\uc740 \ubd88\uac00\ub2a5\ud588\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"workshop\">Workshop<\/h3>\n<p>Workshop\uacfc Tutorial\uc740 \ubcf4\ud1b5 \uac19\uc740 \ub0a0 \uac19\uc740 \uc2dc\uac04\uc5d0 \ub3d9\uc2dc \ub2e4\ubc1c\uc801\uc73c\ub85c \uc9c4\ud589\uc774 \ub418\ub2e4 \ubcf4\ub2c8 \ub4e3\uace0 \uc2f6\uc740 \uc138\uc158\ub4e4\uc744 \ubbf8\ub9ac \ucd94\ub824\uc11c \uc2a4\ucf00\uc974 \ud45c\ub97c \ub9cc\ub4e4\uc5b4\uc11c \ub2e4\ub154\uc2b5\ub2c8\ub2e4. Workshop\uc740 \ud558\ub098\uc758 \uc8fc\uc81c\uac00 \ud558\ub8e8\uc5d0 2\ubc88\uc529 zoom\uc744 \ud1b5\ud574 \uc9c4\ud589\uc774 \ub418\uc5c8\uace0, \ud559\ud68c\uc758 \uccab\ub0a0\uc778 \uc77c\uc694\uc77c\uacfc, \ub9c8\uc9c0\ub9c9 \ub0a0\uc778 \uae08\uc694\uc77c\uc5d0 \uc9c4\ud589\ub418\uc5c8\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/eccv2020\/sunday_workshop.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc800\ub294 \uc774\ubc88 ECCV 2020\uc5d0\uc11c <a href=\"https:\/\/vipriors.github.io\/\" target=\"_blank\"><b> \u201cVisual Inductive Priors for Data-Efficient Deep Learning\u201d <\/b><\/a> \uc758 Action Recognition \uacfc Semantic Segmentation Challenge\uc5d0 \ucc38\uac00\ub97c \ud558\uc600\uace0, \uadf8 \uc911 <a href=\"http:\/\/mvp.yonsei.ac.kr\/\" target=\"_blank\"><b> \uc5f0\uc138\ub300\ud559\uad50 MVP Lab <\/b><\/a> \ubc15\uc0ac\uacfc\uc815\ubd84\ub4e4, \uc800\ud76c Cognex Deep Learning Lab \uc870\ub3d9\ud5cc \uc5f0\uad6c\uc6d0\uacfc \ud568\uaed8 \ucc38\uc5ec\ud55c Action Recognition\uc5d0\uc11c 4\uc704\uc758 \uc131\uc801\uc744 \uac70\ub450\uc5c8\uc2b5\ub2c8\ub2e4. \ub610\ud55c Challenge\uc5d0\uc11c \uc2dc\ub3c4\ud588\ub358 \ubc29\ubc95\ub4e4\uc744 \ub17c\ubb38\uc73c\ub85c \uc81c\ucd9c\ud558\uc5ec Oral Paper\uc5d0 \uc120\uc815\uc774 \ub418\uc5c8\uc2b5\ub2c8\ub2e4. \uc88b\uc740 \ud300\uc6d0 \ubd84\ub4e4\uc744 \ub9cc\ub09c \ub355\uc5d0 \uc88b\uc740 \uc131\uacfc\ub97c \uc5bb\uc744 \uc218 \uc788\uc5c8\ub358 \uac83 \uac19\uc2b5\ub2c8\ub2e4. \ub2e4\uc2dc \ud55c\ubc88 \uac10\uc0ac\ub4dc\ub9bd\ub2c8\ub2e4! \ud639\uc2dc\ub098 \uad81\uae08\ud574\ud558\uc2e4 \ubd84\ub4e4\uc744 \uc704\ud574 \uc800\ud76c \uc5f0\uad6c \uc131\uacfc\ub97c \ub2f4\uc740 \ub17c\ubb38 \ub9c1\ud06c\ub97c \ub0a8\uae41\ub2c8\ub2e4.<\/p>\n<ul>\n  <li>Paper: <a href=\"https:\/\/arxiv.org\/pdf\/2008.05721.pdf\" target=\"_blank\"><b> Learning Temporally Invariant and Localizable Features via Data Augmentation for Video Recognition <\/b><\/a><\/li>\n<\/ul>\n\n<p>\uc800\ub294 \uc704\uc758 \u201cVisual Inductive Priors for Data-Efficient Deep Learning\u201c Workshop \uc678\uc5d0\ub3c4 <a href=\"https:\/\/sites.google.com\/view\/ipcv2020\/\" target=\"_blank\"><b> \u201cImbalance Problems in Computer Vision (IPCV)\u201d <\/b><\/a>, <a href=\"http:\/\/www.robustvision.net\/index.php\" target=\"_blank\"><b> \u201cRobust Vision Challenge 2020\u201d <\/b><\/a> \ub3c4 \uc7ac\ubbf8\uc788\uac8c \ub4e4\uc5c8\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\uc804\uccb4 Workshop\uc758 \ub9ac\uc2a4\ud2b8\uc640 \ud648\ud398\uc774\uc9c0\ub4e4\uc740 <a href=\"https:\/\/eccv2020.eu\/workshops\/\" target=\"_blank\"><b> ECCV 2020 \uacf5\uc2dd \ud648\ud398\uc774\uc9c0<\/b><\/a> \uc5d0\uc11c \ud655\uc778\ud558\uc2e4 \uc218 \uc788\uace0, \uc77c\ubd80 Workshop\uc758 \uacbd\uc6b0 \ud559\ud68c\ub97c \ub4f1\ub85d\ud558\uc9c0 \uc54a\uc740 \uc0ac\ub78c\ub4e4\ub3c4 \ubc1c\ud45c \uc790\ub8cc\uc640 \ubc1c\ud45c Video\ub97c \ubcfc \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"tutorial\">Tutorial<\/h3>\n<p>Tutorial\ub3c4 Workshop\uacfc \ub9c8\ucc2c\uac00\uc9c0\ub85c \ud558\ub098\uc758 \uc8fc\uc81c\uac00 \ud558\ub8e8\uc5d0 2\ubc88\uc529 zoom\uc744 \ud1b5\ud574 \uc9c4\ud589\uc774 \ub418\uc5c8\uace0, \uc2e4\uc6a9\uc801\uc778 \ub0b4\uc6a9\uc774 \ub9ce\uc544\uc11c \uac1c\uc778\uc801\uc73c\ub860 \ub9ce\uc740 \ub3c4\uc6c0\uc774 \ub418\uc5c8\uc2b5\ub2c8\ub2e4. \uc774\ubc88 \ud559\ud68c\uc5d0\uc11c \uac00\uc7a5 \ub9cc\uc871\uc2a4\ub7ec\uc6e0\ub358 \ud504\ub85c\uadf8\ub7a8\uc744 \uaf3d\uc73c\ub77c\uba74 \uc800\ub294 \ubc14\ub85c Tutorial\uc744 \uaf3d\uc744 \uc218 \uc788\uc744 \uac83 \uac19\ub124\uc694.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/eccv2020\/sunday_tutorial.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc81c\uac00 \ub4e4\uc5c8\ub358 Tutorial \uc911\uc5d0 \uc720\uc775\ud558\uba74\uc11c\ub3c4 \ub204\uad6c\ub098 \ubc1c\ud45c \uc790\ub8cc\uc640 \ubc1c\ud45c Video\ub97c \ud655\uc778\ud560 \uc218 \uc788\ub294 \ud504\ub85c\uadf8\ub7a8\ub4e4\uc744 \uc18c\uac1c \ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n<ul>\n  <li><a href=\"https:\/\/nvlabs.github.io\/eccv2020-mixed-precision-tutorial\/\" target=\"_blank\"><b> \u201cAccelerating Computer Vision with Mixed Precision\u201d<\/b><\/a> : TensorFlow\uc640 PyTorch\uc758 \ud559\uc2b5\uc744 \uac00\uc18d\uc2dc\ud0a4\uae30 \uc704\ud55c Mixed Precision Training \uae30\ubc95\uc744 \ub2e4\ub8ec Tutorial. \uc800\ub294 \uac1c\uc778\uc801\uc73c\ub85c Code Optimization Tricks \ubc1c\ud45c \u201cPyTorch Performance Tuning Guide\u201d \uac00 \uac00\uc7a5 \ud070 \ub3c4\uc6c0\uc774 \ub418\uc5c8\uc2b5\ub2c8\ub2e4.<\/li>\n  <li><a href=\"https:\/\/hangzhang.org\/ECCV2020\/\" target=\"_blank\"><b> \u201cFrom HPO to NAS: Automated Deep Learning\u201d<\/b><\/a> : AutoML\uc744 \uc774\uc6a9\ud55c Hyper Parameter Optimization\ubd80\ud130 Neural Architecture Search \uae4c\uc9c0 \ucd5c\uadfc \uae09\uc131\uc7a5\ud55c \uc5f0\uad6c\ub4e4\uc744 \uc544\uc8fc \uc790\uc138\ud788 \ub2e4\ub8e8\uace0 \uc788\uc2b5\ub2c8\ub2e4.<\/li>\n  <li><a href=\"https:\/\/hbilen.github.io\/wsl-eccv20.github.io\/\" target=\"_blank\"><b> \u201cWeakly-Supervised Learning in Computer Vision\u201d<\/b><\/a> : Computer Vision\uc5d0\uc11c \ub2e4\ub904\uc9c0\ub294 Weakly Supervised Learning, \uc0ac\ub78c\uc774 annotation pipeline\uc5d0 \uac1c\uc785\ud558\uc5ec \ud6a8\uc728\uc131\uc744 \ub192\uc774\ub294 Human-in-the-Loop\uc5d0 \ub300\ud55c \ub0b4\uc6a9, Weakly Supervised Learning\uc744 \uc62c\ubc14\ub974\uac8c \ud3c9\uac00\ud558\uae30 \uc704\ud55c \ubc29\ubc95 \ub4f1\uc744 \uad49\uc7a5\ud788 \uc790\uc138\ud788 \ub2e4\ub8e8\uace0 \uc788\uc2b5\ub2c8\ub2e4.<\/li>\n<\/ul>\n\n<h3 id=\"demo-session\">Demo Session<\/h3>\n<p>\ub2e4\uc74c\uc73c\ub85c \ub370\ubaa8\ub294 \uc6d4\uc694\uc77c\ubd80\ud130 \ubaa9\uc694\uc77c\uae4c\uc9c0 \uc9c4\ud589\ub418\uc5c8\uace0, \ucd1d 41\uac00\uc9c0 \uc8fc\uc81c\uc758 \ub370\ubaa8\uac00 \uc874\uc7ac\ud569\ub2c8\ub2e4. \ud558\ub098\uc758 \uc8fc\uc81c \ub2f9 \ud558\ub8e8\uc5d0 2\ubc88, \uac01\uac01 2\uc2dc\uac04\uc529 \uc9c4\ud589\uc774 \ub418\uc5c8\uace0 lobby\uc758 \ud558\ub2e8\uc5d0 Networking Lounge\ub97c \ub20c\ub7ec\uc11c \ub4e4\uc5b4\uac00\uba74 \ubaa8\ub4e0 \ub370\ubaa8\ub4e4\uc758 \uc124\uba85\uc774 \ub2f4\uae34 \uc6cc\ub4dc \ud30c\uc77c\uacfc 5\ubd84 \ub0b4\uc678\uc758 Demo Video\ub97c \ubcfc \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/eccv2020\/demo.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub2e4\ub9cc Poster\uc640 \ub9c8\ucc2c\uac00\uc9c0\ub85c Demo\ub3c4 \uc2e4\uc2dc\uac04\uc73c\ub85c \ub3d9\uc791\ud558\ub294 \uac83\uc744 \ubcf4\uc5ec\uc8fc\ub294 \uac83\uc774 \uc758\ubbf8\uac00 \uc788\ub2e4\uace0 \uc0dd\uac01\ud558\ub294\ub370 \ub2e8\uc21c\ud788 \ubb38\uc11c\uc640 Video\ub85c\ub9cc \uc81c\uacf5\uc774 \ub418\ub2c8 \uc544\ubb34\ub798\ub3c4 \ubab0\uc785\ub3c4\uac00 \ub5a8\uc5b4\uc84c\uace0, \ubaa8\ub4e0 \ucee8\ud150\uce20\uac00 Video \ud615\ud0dc\ub85c \uc81c\uacf5\ub418\ub2e4 \ubcf4\ub2c8 Demo\ub9cc\uc758 \ub9e4\ub825\uc774 \uc0ac\ub77c\uc9c4 \ub290\ub08c\uc744 \ubc1b\uc558\uc2b5\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"exhibition\">Exhibition<\/h3>\n<p>\ub9c8\uc9c0\ub9c9\uc740 \uae30\uc5c5\ub4e4\uc774 \ub3c8\uc744 \uc9c0\ubd88\ud558\uace0, \uc9c0\ubd88\ud55c \uae08\uc561\uc5d0 \ub530\ub77c \ub2e4\uc774\uc544\ubaac\ub4dc, \ud50c\ub798\ud2f0\ub118, \uace8\ub4dc, \uc2e4\ubc84, \uc2a4\ud0c0\ud2b8\uc5c5 \ub4f1\uae09\uc73c\ub85c \ub098\ub258\uc5b4\uc11c \ubcf8\uc778\ub4e4\uc758 \ubd80\uc2a4\ub97c \uc6b4\uc601\ud558\ub294 Exhibition\uc785\ub2c8\ub2e4. \uac01 \ub4f1\uae09\ubcc4 \uac00\uaca9\uacfc \ud61c\ud0dd\uc740 <a href=\"https:\/\/eccv2020.eu\/wp-content\/uploads\/2020\/05\/ECCV20-Partnership-Package-Overview-dig.jpg\" target=\"_blank\"><b> ECCV 2020 Partnership Package Overview \uc790\ub8cc <\/b><\/a>\uc5d0\uc11c \ud655\uc778\ud558\uc2e4 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/eccv2020\/exhibition.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>Lobby\uc5d0\uc11c Exhibition\uc5d0 \ub4e4\uc5b4\uac00\uba74 \ub2e4\uc74c\uacfc \uac19\uc774 \ub4f1\uae09\uc5d0 \ub530\ub77c \ubd80\uc2a4\uac00 \ub098\ub258\uc5b4\uc838 \uc788\uc73c\uba70, \uac01 \ubd80\uc2a4\ub97c \ud074\ub9ad\ud558\uba74 \ubcf8\uc778\ub4e4\uc758 \uc5f0\uad6c\uc640 \uc81c\ud488\ub4e4\uc744 Video \ud615\ud0dc\ub85c \ubcf4\uc5ec\uc8fc\uace0, \ucc44\uc6a9 \uad00\ub828 \ud0ed\ub3c4 \ubcc4\ub3c4\ub85c \uc6b4\uc601\ud558\uace0 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/eccv2020\/facebook_booth.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc704\uc758 \uadf8\ub9bc\uc740 \ub2e4\uc774\uc544\ubaac\ud2b8 \ud30c\ud2b8\ub108 Facebook\uc758 \ubd80\uc2a4\uc774\uba70 \uc800\ub294 <a href=\"https:\/\/cdn-akamai.6connex.eu\/\/53\/75\/\/PyTorch_Video_Resources_15980345972578817.pdf\" target=\"_blank\"><b> PyTorch\uc758 \uc8fc\uc694 \uae30\ub2a5\ub4e4\uc744 \ub2e4\ub8ec \uc790\ub8cc<\/b><\/a> \uac00 \ud070 \ub3c4\uc6c0\uc774 \ub418\uc5c8\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/eccv2020\/apple_booth.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\ub2e4\uc74c\uc740 \uace8\ub4dc \ud30c\ud2b8\ub108 Apple\uc758 \ubd80\uc2a4\uc774\uba70 \uad49\uc7a5\ud788 \uc2ec\ud50c\ud558\uac8c \ubd80\uc2a4\ub97c \uafb8\uba84\uc73c\uba70, \uc8fc\ub85c \uc5f0\uad6c \ub0b4\uc6a9 \uc18c\uac1c\uc640 \ucc44\uc6a9 \uad00\ub828 \ub0b4\uc6a9\uc774 \uc8fc\ub97c \uc774\ub8e8\uace0 \uc788\uc5c8\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\uc624\ud504\ub77c\uc778\uc5d0\uc11c\ub294 \uc2a4\ud3f0\uc11c\uc758 \ub4f1\uae09\uc5d0 \ub530\ub77c \ubd80\uc2a4\uc758 \ud06c\uae30, \uba74\uc801 \ub4f1\uc774 \ub2ec\ub790\ub358 \uac83 \uac19\uc740\ub370 \uc628\ub77c\uc778\uc5d0\uc11c\ub294 \ub4f1\uae09\uc5d0 \ubb34\uad00\ud558\uac8c \ub3d9\uc77c\ud55c \ud615\ud0dc\ub85c \uc6b4\uc601\uc774 \ub418\ub294 \uac83 \uac19\uc558\uc2b5\ub2c8\ub2e4. \uadf8\ub9ac\uace0 \ubb34\uc5c7\ubcf4\ub2e4, \uc624\ud504\ub77c\uc778\uc5d0\uc11c \ubd80\uc2a4\ub97c \ub3cc\uc544\ub2e4\ub2c8\uba74 \uac01\uc885 \uc2a4\ud2f0\ucee4\uc640 \uae30\ub150\ud488, \uac04\uc2dd\uc744 \uc5bb\uc744 \uc218 \uc788\uc5c8\ub294\ub370 \uc628\ub77c\uc778\uc740 \uadf8\ub807\uc9c0 \ubabb\ud55c \uc810\uc774 \uc544\uc26c\uc6e0\uc2b5\ub2c8\ub2e4. \uaec4\uaec4\uaec4<\/p>\n\n<blockquote> \uacb0\ub860 <\/blockquote>\n<p>\uc774\ubc88 \ud3ec\uc2a4\ud305\uc5d0\uc11c\ub294 COVID-19\ub85c \uc778\ud574 Virtual Conference\ub85c \uc9c4\ud589\ub41c ECCV 2020\uc758 \uc8fc\uc694 \ud504\ub85c\uadf8\ub7a8\ub4e4\uc744 \uc18c\uac1c \ub4dc\ub9ac\uace0, \uac01 \ud504\ub85c\uadf8\ub7a8\ub9c8\ub2e4 \ub290\ub080 \uc810\ub4e4\uc744 \uacf5\uc720 \ub4dc\ub838\uc2b5\ub2c8\ub2e4. \uac00\uaca9\uc774 \ub0ae\uc544\uc84c\ub2e4\ub294 \uc810\uacfc \uc9d1\uc5d0\uc11c \ub4e4\uc744 \uc218 \uc788\ub2e4\ub294 \uc810\uc774 \uc7a5\uc810\uc774 \ub420 \uc218\ub3c4 \uc788\uc9c0\ub9cc, \uc800\uc5d0\uac8c\ub294 \ub2e8\uc810\uc774 \ub354 \ud06c\uac8c \ub290\uaef4\uc84c\ub358 \uac83 \uac19\uc2b5\ub2c8\ub2e4. \uc544\ubb34\ub798\ub3c4 Poster Session\uacfc Demo\ub294 \ud604\uc7a5\uc5d0\uc11c \uc0dd\uc0dd\ud558\uac8c \uccb4\ud5d8\uc744 \ud558\uace0, \uc11c\ub85c \ub9c8\uc8fc\ubcf4\uba70 \uc5f4\ub764 \ud1a0\ub860\uc744 \ud574\uc57c \uc7ac\ubc0c\ub294\ub370 \ubaa8\ub2c8\ud130 \ud654\uba74 \ub108\uba38\ub85c \uc18c\ud1b5\uc744 \ud558\ub2e4 \ubcf4\ub2c8 \ubab0\uc785\ub3c4\uac00 \ub5a8\uc5b4\uc84c\uc2b5\ub2c8\ub2e4. \ud558\uc9c0\ub9cc Workshop, Tutorials\uc740 \uc88b\uc740 \ub0b4\uc6a9\uc774 \ub9ce\uc774 \ub2e4\ub904\uc84c\uace0, \uc790\ub8cc\ub3c4 \ub300\ubd80\ubd84 \uacf5\uac1c\uac00 \ub418\uc5b4\uc788\uc5b4\uc11c \ud559\ud68c\uc5d0 \ucc38\uc11d\ud558\uc9c0 \uc54a\uc544\ub3c4 \uc591\uc9c8\uc758 \uc790\ub8cc\ub4e4\uc744 \uc811\ud560 \uc218 \uc788\ub2e4\ub294 \uc810\uc740 \ucc38 \uc88b\uc740 \uac83 \uac19\uc2b5\ub2c8\ub2e4. \uc804\uc138\uacc4\uc801\uc73c\ub85c \uc548\uc815\uc744 \ucc3e\uc544\uc11c \ub2e4\uc2dc \uc624\ud504\ub77c\uc778 \ud559\ud68c\uac00 \uac1c\ucd5c\ub420 \uadf8 \ub0a0\uc744 \uae30\ub2e4\ub9ac\uba70 \uc624\ub298\uc758 \uae00 \ub9c8\uce58\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n\n<blockquote> Reference <\/blockquote>\n<ul>\n  <li><a href=\"https:\/\/twitter.com\/CSProfKGD\/status\/1297892662687797255\" target=\"_blank\"><b> ECCV 2020 \ud1b5\uacc4\uce58 \uc790\ub8cc <\/b><\/a><\/li>\n  <li><a href=\"https:\/\/github.com\/lixin4ever\/Conference-Acceptance-Rate\" target=\"_blank\"><b> \uc8fc\uc694 AI \ud559\ud68c acceptance rate \uc790\ub8cc <\/b><\/a><\/li>\n<\/ul>\n\n","<p>\uc548\ub155\ud558\uc138\uc694, \uc624\ub298\uc740 \uc9c0\ub09c 8\uc6d4 23\uc77c(\uc77c) ~ 8\uc6d4 28\uc77c(\uae08) 6\uc77c\uac04 \uc9c4\ud589\ub41c European Conference on Computer Vision(\uc774\ud558 ECCV) 2020 \ud559\ud68c\ub97c Virtual\ub85c \ucc38\uc11d\ud558\uba70 \ub290\ub080 \uc810\ub4e4\uc744 \uacf5\uc720 \ub4dc\ub9ac\uace0, \uc8fc\uc694 \ud504\ub85c\uadf8\ub7a8\ub4e4\uc744 \uc18c\uac1c \ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n\n"],"pubDate":"Thu, 03 Sep 2020 00:00:00 +0000","link":"https:\/\/hoya012.github.io\/\/blog\/ECCV-2020-review\/","guid":"https:\/\/hoya012.github.io\/\/blog\/ECCV-2020-review\/"},{"title":"Image Classification with Automatic Mixed-Precision Training PyTorch Tutorial","description":["<p>\uc548\ub155\ud558\uc138\uc694, \uc9c0\ub09c <a href=\"https:\/\/hoya012.github.io\/blog\/Mixed-Precision-Training\/\" target=\"_blank\"><b> \u201cMixed-Precision Training of Deep Neural Networks\u201d <\/b><\/a> \uae00\uc5d0 \uc774\uc5b4\uc11c \uc624\ub298\uc740 PyTorch 1.6\uc5d0\uc11c \uacf5\uc2dd \uc9c0\uc6d0\ud558\uae30 \uc2dc\uc791\ud55c Automatic Mixed Precision Training \uae30\ub2a5\uc744 \uc9c1\uc811 \uc2e4\ud5d8\ud574\ubcfc \uc218 \uc788\ub294 Tutorial \ucf54\ub4dc\uc640 \uc124\uba85\uc744 \uae00\ub85c \uc791\uc131\ud558\uc600\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\uc2e4\ud5d8\uc5d0 \uc0ac\uc6a9\ud55c \ucf54\ub4dc\ub294 <a href=\"https:\/\/github.com\/hoya012\/automatic-mixed-precision-tutorials-pytorch\" target=\"_blank\"><b> \uc81c GitHub Repository<\/b><\/a> \uc5d0 \uc62c\ub824 \ub450\uc5c8\uc2b5\ub2c8\ub2e4.<\/p>\n\n<blockquote> PyTorch 1.6 \u2013 Automatic Mixed Precision <\/blockquote>\n<p>\uc9c0\ub09c 2020\ub144 7\uc6d4 \ub9d0, PyTorch\uc758 \uc0c8\ub85c\uc6b4 \ubc84\uc804\uc778 1.6\uc774 \ub9b4\ub9ac\uc988 \ub418\uc5c8\uc2b5\ub2c8\ub2e4. <a href=\"https:\/\/pytorch.org\/blog\/pytorch-1.6-released\/\" target=\"_blank\"><b> \ub9b4\ub9ac\uc988 \ub178\ud2b8<\/b><\/a> \uc5d0\ub294 \uc804\ubc18\uc801\uc778 \uc131\ub2a5 \ud5a5\uc0c1\uacfc Memory Profiling \uae30\ub2a5, Distributed Training &amp; RPC(Remote Procedure Call), Frontend APU \uc5c5\ub370\uc774\ud2b8, Torchvision, Torchaudio\uc758 \uc2e0 \ubc84\uc804 \uacf5\uac1c \ub4f1 \ub2e4\uc591\ud55c \ub0b4\uc6a9\uc774 \ub2f4\uaca8\uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\uadf8 \uc911 \uc624\ub298 \uc18c\uac1c\ub4dc\ub9b4 Automatic Mixed Precision(\uc774\ud558, AMP) \uae30\ub2a5\uc774 \uc800\ub294 \uac00\uc7a5 \uc778\uc0c1\uae4a\uc5c8\uace0 \uacb0\uacfc\uc801\uc73c\ub85c \uc774\ub807\uac8c \ube14\ub85c\uadf8 \uae00\uc744 \uc791\uc131\ud558\uac8c \ub418\uc5c8\uc2b5\ub2c8\ub2e4. Mixed Precision Training\uc5d0 \ub300\ud55c \uc774\ub860\uc801\uc778 \ub0b4\uc6a9\uc740 \uc9c0\ub09c \ud3ec\uc2a4\ud305\uc5d0\uc11c \uc124\uba85\uc744 \ub4dc\ub838\uc73c\ub2c8 \uc790\uc138\ud55c \uc124\uba85\uc740 \uc0dd\ub7b5\ub4dc\ub9ac\uace0 \uc5b4\ub5bb\uac8c \uc0ac\uc6a9\ud560 \uc218 \uc788\ub294\uc9c0\uc5d0 \ucd08\uc810\uc744 \ub9de\ucdb0\uc11c \uc124\uba85\uc744 \ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\uae30\uc874\uc5d0\ub3c4 NVIDIA\uc5d0\uc11c 2018\ub144 \uac1c\ubc1c\ud55c <a href=\"https:\/\/developer.nvidia.com\/blog\/apex-pytorch-easy-mixed-precision-training\/\" target=\"_blank\"><b> Apex<\/b><\/a> \ub97c \uc774\uc6a9\ud558\uba74 PyTorch\uc5d0\uc11c Mixed Precision Training\uc744 \ud560 \uc218 \uc788\uc5c8\uc2b5\ub2c8\ub2e4. \ubcc4\ub3c4\uc758 \ub77c\uc774\ube0c\ub7ec\ub9ac \ud615\ud0dc\ub85c \uc874\uc7ac\ud558\uc600\ub294\ub370 \uc774\ubc88 PyTorch 1.6\uc5d0\uc11c NVIDIA\uc640 Facebook \uac1c\ubc1c\uc790\ub4e4\uc774 \ud798\uc744 \ud569\ud574\uc11c \uacf5\uc2dd\uc801\uc73c\ub85c \uc9c0\uc6d0\ud558\uac8c \ub418\uc5c8\ub2e4\uace0 \ud569\ub2c8\ub2e4. (\uac13\ube44\ub514\uc544, \uac13\ud398\ubd81 \uc120\uc0dd\ub2d8\ub4e4 \uac10\uc0ac\ud569\ub2c8\ub2e4!!)<\/p>\n\n<p>\ud558\ub098\uc758 \ud558\ub298 \uc544\ub798 \ub450 \uac1c\uc758 \ud0dc\uc591\uc774 \ub5a0\uc788\uc744 \uc21c \uc5c6\uaca0\uc8e0? PyTorch\uc5d0 AMP \uae30\ub2a5\uc774 \ud569\uccd0\uc9c0\uba74\uc11c \uae30\uc874 Apex\uc758 AMP \uae30\ub2a5\uc740 \uc9c0\uc6d0\uc774 \uc911\ub2e8\ub420 \uc608\uc815\uc774\ub77c\uace0 \ud569\ub2c8\ub2e4. (With AMP being added to PyTorch core, we have started the process of deprecating apex.amp.) \ud558\uc9c0\ub9cc Apex\uc758 AMP \uae30\ub2a5\uc744 \uc0ac\uc6a9\ud558\uc2dc\ub358 \ubd84\ub4e4\ub3c4 \ucf54\ub4dc \uba87 \uc904\ub9cc \uc218\uc815\ud558\uc2dc\uba74 \ubc14\ub85c \uc0ac\uc6a9\ud558\uc2e4 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>Torch.cuda.amp \ud615\ud0dc\ub85c \uc811\uadfc\ud574\uc11c \uc0ac\uc6a9\ud560 \uc218 \uc788\uc73c\uba70, AMP \uae30\ub2a5\uc758 \uc0ac\uc6a9 \ubc29\ubc95\uc5d0 \ub300\ud55c \uacf5\uc2dd \ubb38\uc11c\ub294 \ub2e4\uc74c\uacfc \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n<ul>\n  <li><a href=\"https:\/\/pytorch.org\/docs\/stable\/amp.html\" target=\"_blank\"><b> https:\/\/pytorch.org\/docs\/stable\/amp.html <\/b><\/a><\/li>\n  <li><a href=\"https:\/\/pytorch.org\/docs\/stable\/notes\/amp_examples.html\" target=\"_blank\"><b> https:\/\/pytorch.org\/docs\/stable\/notes\/amp_examples.html <\/b><\/a><\/li>\n<\/ul>\n\n<blockquote> Image Classification with AMP Tutorials <\/blockquote>\n<p>\uacf5\uc2dd \ubb38\uc11c\uc5d0\uc11c \uc608\uc81c\ub85c \uc62c\ub824\ub454 \ucf54\ub4dc\ub294 \ub2e4\uc74c\uacfc \uac19\uc2b5\ub2c8\ub2e4.<\/p>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"kn\">import<\/span> <span class=\"nn\">torch<\/span> \n<span class=\"c1\"># Creates once at the beginning of training \n<\/span><span class=\"n\">scaler<\/span> <span class=\"o\">=<\/span> <span class=\"n\">torch<\/span><span class=\"p\">.<\/span><span class=\"n\">cuda<\/span><span class=\"p\">.<\/span><span class=\"n\">amp<\/span><span class=\"p\">.<\/span><span class=\"n\">GradScaler<\/span><span class=\"p\">()<\/span> \n \n<span class=\"k\">for<\/span> <span class=\"n\">data<\/span><span class=\"p\">,<\/span> <span class=\"n\">label<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">data_iter<\/span><span class=\"p\">:<\/span> \n   <span class=\"n\">optimizer<\/span><span class=\"p\">.<\/span><span class=\"n\">zero_grad<\/span><span class=\"p\">()<\/span> \n   <span class=\"c1\"># Casts operations to mixed precision \n<\/span>   <span class=\"k\">with<\/span> <span class=\"n\">torch<\/span><span class=\"p\">.<\/span><span class=\"n\">cuda<\/span><span class=\"p\">.<\/span><span class=\"n\">amp<\/span><span class=\"p\">.<\/span><span class=\"n\">autocast<\/span><span class=\"p\">():<\/span> \n      <span class=\"n\">loss<\/span> <span class=\"o\">=<\/span> <span class=\"n\">model<\/span><span class=\"p\">(<\/span><span class=\"n\">data<\/span><span class=\"p\">)<\/span> \n \n   <span class=\"c1\"># Scales the loss, and calls backward() \n<\/span>   <span class=\"c1\"># to create scaled gradients \n<\/span>   <span class=\"n\">scaler<\/span><span class=\"p\">.<\/span><span class=\"n\">scale<\/span><span class=\"p\">(<\/span><span class=\"n\">loss<\/span><span class=\"p\">).<\/span><span class=\"n\">backward<\/span><span class=\"p\">()<\/span> \n \n   <span class=\"c1\"># Unscales gradients and calls \n<\/span>   <span class=\"c1\"># or skips optimizer.step() \n<\/span>   <span class=\"n\">scaler<\/span><span class=\"p\">.<\/span><span class=\"n\">step<\/span><span class=\"p\">(<\/span><span class=\"n\">optimizer<\/span><span class=\"p\">)<\/span> \n \n   <span class=\"c1\"># Updates the scale for next iteration \n<\/span>   <span class=\"n\">scaler<\/span><span class=\"p\">.<\/span><span class=\"n\">update<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>\uae30\uc874\uc5d0 \uc0ac\uc6a9\ud558\ub358 \ucf54\ub4dc\uc5d0\uc11c GradScaler \ub97c \uc120\uc5b8\ud574\uc8fc\uace0, data\ub97c model\uc5d0 \ub123\uc5b4\uc8fc\ub294 \ubd80\ubd84\uc744 \uc218\uc815\ud558\uace0, loss\uc640 optimizer\ub97c step \uc2dc\ud0a4\ub294 \ubd80\ubd84\ub9cc \uc218\uc815\ud574\uc8fc\uba74 \ubc14\ub85c AMP\ub97c \uc801\uc6a9\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uad49\uc7a5\ud788 \uac04\ub2e8\ud558\uc8e0?<\/p>\n\n<p>\ud558\uc9c0\ub9cc \uc9c1\uc811 \ucc98\uc74c\ubd80\ud130 \ub05d\uae4c\uc9c0 \ub3cc\ub824\ubcfc \uc218 \uc788\ub294 Tutorial Code\uac00 \uc5c6\uc5b4\uc11c \uc9c1\uc811 Image Classification \ub370\uc774\ud130\uc14b\uc73c\ub85c \uc2e4\ud5d8\uc744 \ud574\ubcfc \uc218 \uc788\ub294 Codebase\ub97c \uc81c\uc791\ud558\uc600\uc2b5\ub2c8\ub2e4.<\/p>\n\n<ul>\n  <li>\ucf54\ub4dc \uc8fc\uc18c: <a href=\"https:\/\/github.com\/hoya012\/automatic-mixed-precision-tutorials-pytorch\" target=\"_blank\"><b> https:\/\/github.com\/hoya012\/automatic-mixed-precision-tutorials-pytorch <\/b><\/a><\/li>\n<\/ul>\n\n<p>\ub3c4\uc6c0\uc774 \ub418\uc168\ub2e4\uba74! \uad6c\ub3c5\uacfc \uc88b\uc544\uc694, \uc54c\ub9bc \uc124\uc815\uae4c\uc9c0!.. \ub294 \uc544\ub2c8\uace0 \uc2a4\ud0c0 \ud558\ub098\uc529 \ub20c\ub7ec \uc8fc\uc2dc\uba74 \uac10\uc0ac\ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4 \u314e\u314e<\/p>\n\n<h3 id=\"0-experimental-setup\">0. Experimental Setup<\/h3>\n<p>\uc6b0\uc120 \ucf54\ub4dc\ub97c \ub2e4\uc6b4\ubc1b\uc73c\uc2e0 \ub4a4 \uc2e4\ud5d8\uc5d0 \ud544\uc694\ud55c \ub77c\uc774\ube0c\ub7ec\ub9ac\ub4e4\uc744 \uc124\uce58\ud574\uc90d\ub2c8\ub2e4.<\/p>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"n\">pip<\/span> <span class=\"n\">install<\/span> <span class=\"o\">-<\/span><span class=\"n\">r<\/span> <span class=\"n\">requirements<\/span><span class=\"p\">.<\/span><span class=\"n\">txt<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>\uadf8 \ub4a4 \uc2e4\ud5d8\uc5d0 \uc0ac\uc6a9\ud560 \ub370\uc774\ud130\uc14b\uc744 \ub2e4\uc6b4\ubc1b\uc544\uc57c \ud558\ub294\ub370\uc694, \uc800\ub294 \ub9e8\ub0a0 \uc4f0\ub294 ImageNet, CIFAR \ub4f1\uc758 \ub370\uc774\ud130\uc14b \ub9d0\uace0 \uc0c8\ub85c\uc6b4 \ub370\uc774\ud130\uc14b\uc744 \uc0ac\uc6a9\ud574\ubcf4\uace0 \uc2f6\uc5b4\uc11c \uc774\uac83 \uc800\uac83 \ucc3e\uc544\ubcf4\ub2e4\uac00 \uc791\ub144 Kaggle\uc5d0\uc11c \uc9c4\ud589\ub418\uc5c8\ub358 <strong>Intel Image Classification<\/strong> \ub370\uc774\ud130\uc14b\uc774 \ub9c8\uc74c\uc5d0 \ub4e4\uc5b4\uc11c \uc790\uc8fc \uc0ac\uc6a9\ud558\uace0 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/mixed_precision\/10.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc774 \ub370\uc774\ud130\uc14b\uc740 \ube4c\ub529, \uc232, \ube59\ud558, \uc0b0, \ubc14\ub2e4, \uac70\ub9ac \ucd1d 6\uac00\uc9c0\uc758 class\ub85c \uad6c\uc131\ub418\uc5b4 \uc788\uace0, 150x150 \ud06c\uae30\uc758 image 25000\uc7a5\uc774 \uc81c\uacf5\ub429\ub2c8\ub2e4. \ube44\uad50\uc801 \uad6c\ubd84\uc774 \uc798 \ub418\ub294 class\uae34 \ud55c\ub370 \uc9c1\uc811 \ub370\uc774\ud130\ub97c \uae4c\ubcf4\uba74 class\uac00 \uc560\ub9e4\ud558\uac70\ub098 \uc798\ubabb labeling \ub41c image\ub3c4 \uc874\uc7ac\ud574\uc11c \ub098\ub984 \uc7ac\ubbf8\uc788\uc2b5\ub2c8\ub2e4. \uc624\ub298 \uc2e4\ud5d8\uc5d0\uc11c\ub294 \uc774 Intel Classification \ub370\uc774\ud130\uc14b\uc744 \uc0ac\uc6a9\ud560 \uc608\uc815\uc785\ub2c8\ub2e4.<\/p>\n\n<h3 id=\"1-baseline-training\">1. Baseline Training<\/h3>\n<p>\uc624\ub298 \uc2e4\ud5d8\uc5d0\uc11c\ub294 \uae30\uc874 \ubc29\uc2dd(FP32)\uacfc AMP\ub97c \uc801\uc6a9\ud558\uc600\uc744 \ub54c\ub97c \ube44\uad50\ud560 \uc608\uc815\uc774\uba70, \uae30\ubcf8\uc801\uc778 \uc2e4\ud5d8 \uc14b\ud305\uc740 \ub2e4\uc74c\uacfc \uac19\uc774 \uc0ac\uc6a9\ud558\uc600\uc2b5\ub2c8\ub2e4.<\/p>\n\n<ul>\n  <li>ImageNet Pretrained ResNet-18 from torchvision.models<\/li>\n  <li>Batch Size 256 \/ Epochs 120 \/ Initial Learning Rate 0.0001<\/li>\n  <li>Training Augmentation: Resize((256, 256)), RandomHorizontalFlip()<\/li>\n  <li>Adam + Cosine Learning rate scheduling with warmup<\/li>\n<\/ul>\n\n<p>\uad49\uc7a5\ud788 \uae30\ubcf8\uc801\uc778 \uae30\ubc95\ub4e4\ub9cc \uc801\uc6a9\uc744 \ud558\uc600\uc73c\uba70 \uc2e4\ud5d8\uc5d0 \uc0ac\uc6a9\ud55c \ud558\ub4dc\uc6e8\uc5b4(GPU)\ub294 Tensor Core\uac00 \uc5c6\ub294 Pascal \uc138\ub300\uc758 GTX 1080 Ti 1\uac1c\uc640, Tensor Core\uac00 \uc788\ub294 Turing \uc138\ub300\uc758 RTX 2080 Ti 1\uac1c\ub97c \uc0ac\uc6a9\ud558\uc600\uc2b5\ub2c8\ub2e4. \uc774 \ub450 \uac1c\uc758 GPU\uac00 \uc544\ubb34\ub798\ub3c4 \ub9ce\uc774 \uc0ac\uc6a9\uc774 \ub418\uae30\ub3c4 \ud558\uace0, \uc800 \uac19\uc740 \uc11c\ubbfc\ub4e4\uc774 \uc0ac\uc6a9\ud560 \uc218 \uc788\ub294 \ud558\uc774\uc5d4\ub4dc GPU\uae30\ub3c4 \ud569\ub2c8\ub2e4. (GPU \ub9ce\uc740 \uc11c\ubc84 \uac16\uace0 \uc2f6\ub124\uc694..)<\/p>\n\n<figure>\n\t<img src=\"\/assets\/img\/mixed_precision\/11.PNG\" alt=\"\" \/> \n<\/figure>\n\n<p>\uc81c\uac00 \uc5c5\ub85c\ub4dc\ud55c \ucf54\ub4dc\ub97c \ub2e4\uc6b4 \ubc1b\uc73c\uc2dc\uace0 \ub370\uc774\ud130\uc14b\uc744 <strong>data<\/strong> \ud3f4\ub354\uc5d0 \ub123\uc5b4 \uc8fc\uc2dc\uba74 \uc900\ube44\ub294 \ub05d\uc785\ub2c8\ub2e4. \n\ud559\uc2b5\uc744 \ub3cc\ub9ac\uae30 \uc704\ud574\uc120 \ub2e4\uc74c\uacfc \uac19\uc740 Command Line \uba85\ub839\uc5b4\ub97c \uc785\ub825\ud574\uc8fc\uc2dc\uba74 \ub429\ub2c8\ub2e4.<\/p>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"n\">Python<\/span> <span class=\"n\">main<\/span><span class=\"p\">.<\/span><span class=\"n\">py<\/span> <span class=\"o\">--<\/span><span class=\"n\">checkpoint_name<\/span> <span class=\"n\">baseline<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<h3 id=\"2-automatic-mixed-precision-training\">2. Automatic Mixed Precision Training<\/h3>\n<p>\ub2e4\uc74c\uc740 PyTorch 1.6\uc758 AMP \uae30\ub2a5\uc744 \ucd94\uac00\ud558\uc5ec \uc2e4\ud5d8\uc744 \ub3cc\ub9ac\ub294 \ubc29\ubc95\uc744 \uc124\uba85 \ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\uc81c \ucf54\ub4dc\uc758 <strong>learning\/trainer.py<\/strong> \uc5d0\uc11c training loop\uac00 \ub3cc\uc544\uac00\ub294\ub370 \uc774 \ubd80\ubd84\uc5d0\uc11c torch.cuda.amp \ub97c \ubd99\uc5ec\uc11c AMP \uae30\ub2a5\uc744 \uc0ac\uc6a9\ud558\uc600\uc2b5\ub2c8\ub2e4.<\/p>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"s\">\"\"\" define loss scaler for automatic mixed precision \"\"\"<\/span>\n<span class=\"n\">scaler<\/span> <span class=\"o\">=<\/span> <span class=\"n\">torch<\/span><span class=\"p\">.<\/span><span class=\"n\">cuda<\/span><span class=\"p\">.<\/span><span class=\"n\">amp<\/span><span class=\"p\">.<\/span><span class=\"n\">GradScaler<\/span><span class=\"p\">()<\/span>\n\n<span class=\"k\">for<\/span> <span class=\"n\">batch_idx<\/span><span class=\"p\">,<\/span> <span class=\"p\">(<\/span><span class=\"n\">inputs<\/span><span class=\"p\">,<\/span> <span class=\"n\">labels<\/span><span class=\"p\">)<\/span> <span class=\"ow\">in<\/span> <span class=\"nb\">enumerate<\/span><span class=\"p\">(<\/span><span class=\"n\">data_loader<\/span><span class=\"p\">):<\/span>\n  <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">optimizer<\/span><span class=\"p\">.<\/span><span class=\"n\">zero_grad<\/span><span class=\"p\">()<\/span>\n\n  <span class=\"k\">with<\/span> <span class=\"n\">torch<\/span><span class=\"p\">.<\/span><span class=\"n\">cuda<\/span><span class=\"p\">.<\/span><span class=\"n\">amp<\/span><span class=\"p\">.<\/span><span class=\"n\">autocast<\/span><span class=\"p\">():<\/span>\n    <span class=\"n\">outputs<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">model<\/span><span class=\"p\">(<\/span><span class=\"n\">inputs<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">loss<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">criterion<\/span><span class=\"p\">(<\/span><span class=\"n\">outputs<\/span><span class=\"p\">,<\/span> <span class=\"n\">labels<\/span><span class=\"p\">)<\/span>\n\n  <span class=\"c1\"># Scales the loss, and calls backward() \n<\/span>  <span class=\"c1\"># to create scaled gradients \n<\/span>  <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">scaler<\/span><span class=\"p\">.<\/span><span class=\"n\">scale<\/span><span class=\"p\">(<\/span><span class=\"n\">loss<\/span><span class=\"p\">).<\/span><span class=\"n\">backward<\/span><span class=\"p\">()<\/span>\n\n  <span class=\"c1\"># Unscales gradients and calls \n<\/span>  <span class=\"c1\"># or skips optimizer.step() \n<\/span>  <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">scaler<\/span><span class=\"p\">.<\/span><span class=\"n\">step<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">optimizer<\/span><span class=\"p\">)<\/span>\n\n  <span class=\"c1\"># Updates the scale for next iteration \n<\/span>  <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">scaler<\/span><span class=\"p\">.<\/span><span class=\"n\">update<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>\uc2e4\uc81c \ucf54\ub4dc\uc5d0\uc11c\ub294 args.amp \ub97c \ud1b5\ud574 amp\ub97c \uc0ac\uc6a9\ud560\uc9c0 \ub9d0\uc9c0\ub97c \uacb0\uc815\ud558\ub3c4\ub85d \uad6c\ud604\uc774 \ub418\uc5b4\uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\ub9c8\ucc2c\uac00\uc9c0\ub85c AMP \uae30\ub2a5\uc744 \uc0ac\uc6a9\ud558\uc5ec \ud559\uc2b5\uc744 \ub3cc\ub9ac\uae30 \uc704\ud55c Command Line \uba85\ub839\uc5b4\ub294 \ub2e4\uc74c\uacfc \uac19\uc2b5\ub2c8\ub2e4. \uac04\ub2e8\ud558\uc8e0?<\/p>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"n\">python<\/span> <span class=\"n\">main<\/span><span class=\"p\">.<\/span><span class=\"n\">py<\/span> <span class=\"o\">--<\/span><span class=\"n\">checkpoint_name<\/span> <span class=\"n\">baseline_amp<\/span> <span class=\"o\">--<\/span><span class=\"n\">amp<\/span><span class=\"p\">;<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<h3 id=\"3-performance-table\">3. Performance Table<\/h3>\n<p>\ub2e4\uc74c\uc740 \uc81c GPU (1080 Ti, 2080 Ti)\uc5d0\uc11c Baseline(FP32)\uacfc AMP\ub97c \uc801\uc6a9\ud558\uc600\uc744 \ub54c\uc758 \uc131\ub2a5\uc744 \uc0b4\ud3b4\ubcfc \uc608\uc815\uc785\ub2c8\ub2e4. Baseline\uacfc AMP \ubaa8\ub450 \uc2e4\ud5d8 \uc14b\ud305(\ubaa8\ub378, \ud559\uc2b5 \ud30c\ub77c\ubbf8\ud130 \ub4f1)\uc744 \ub3d9\uc77c\ud558\uac8c \ud558\uc5ec \uc9c4\ud589\ud558\uc600\uc2b5\ub2c8\ub2e4.\n\uc131\ub2a5\uc5d0\ub294 Test Accuracy, GPU Memory, \uc804\uccb4 \ud559\uc2b5 \uc2dc\uac04\uc744 Metric\uc73c\ub85c \uc0ac\uc6a9\ud558\uc600\uace0, GPU Memory \uc0ac\uc6a9\ub7c9\uc740 <strong>nvidia-smi<\/strong>\uc640 <strong>gpustat<\/strong>\uc744 \ud1b5\ud574 \uce21\uc815\ud558\uc600\uc2b5\ub2c8\ub2e4.<\/p>\n\n<table>\n  <thead>\n    <tr>\n      <th style=\"text-align: center\">Algorithm<\/th>\n      <th style=\"text-align: center\">Test Accuracy<\/th>\n      <th style=\"text-align: center\">GPU Memory<\/th>\n      <th style=\"text-align: center\">Total Training Time<\/th>\n    <\/tr>\n  <\/thead>\n  <tbody>\n    <tr>\n      <td style=\"text-align: center\">B - 1080 Ti<\/td>\n      <td style=\"text-align: center\">94.13<\/td>\n      <td style=\"text-align: center\">10737MB<\/td>\n      <td style=\"text-align: center\">64.9m<\/td>\n    <\/tr>\n    <tr>\n      <td style=\"text-align: center\">B - 2080 Ti<\/td>\n      <td style=\"text-align: center\">94.17<\/td>\n      <td style=\"text-align: center\">10855MB<\/td>\n      <td style=\"text-align: center\">54.3m<\/td>\n    <\/tr>\n    <tr>\n      <td style=\"text-align: center\">AMP - 1080 Ti<\/td>\n      <td style=\"text-align: center\">94.07<\/td>\n      <td style=\"text-align: center\">6615MB<\/td>\n      <td style=\"text-align: center\">64.7m<\/td>\n    <\/tr>\n    <tr>\n      <td style=\"text-align: center\">AMP - 2080 Ti<\/td>\n      <td style=\"text-align: center\">94.23<\/td>\n      <td style=\"text-align: center\">7799MB<\/td>\n      <td style=\"text-align: center\">37.3m<\/td>\n    <\/tr>\n  <\/tbody>\n<\/table>\n\n<p>\uc6b0\uc120 B\ub294 Baseline\uc744 \uc758\ubbf8\ud558\uace0 AMP\ub294 Automatic Mixed Precision\uc744 \uc758\ubbf8\ud569\ub2c8\ub2e4. \uac19\uc740 \uc14b\ud305\uc774\uc9c0\ub9cc GPU\uac00 \ub2ec\ub77c\uc9c0\uba74 \uc810\uc720\ud558\ub294 GPU Memory\ub3c4 \ub2ec\ub77c\uc9c0\ub294 \uc810\uc774 \uc870\uae08 \ud2b9\uc774\ud55c \uacb0\uacfc\uc600\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\ubaa8\ub4e0 \uc14b\ud305\uc5d0\uc11c \uac70\uc758 \ube44\uc2b7\ud55c Test Accuracy\ub97c \ubcf4\uc5ec\uc8fc\uc5c8\uace0 AMP\ub97c \uc0ac\uc6a9\ud558\uba74 GPU Memory\ub97c FP32\ubcf4\ub2e4 \uc801\uac8c \uc810\uc720\ud558\ub294 \uac83\uc744 \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n\n<p>\ub610\ud55c Tensor Core\uac00 \uc5c6\ub294 1080 Ti\uc5d0\uc11c\ub294 \ud559\uc2b5 \uc2dc\uac04\uc774 \uac70\uc758 \uc904\uc5b4\ub4e4\uc9c0 \uc54a\uc740 \ubc18\uba74, Tensor Core\uac00 \uc788\ub294 2080 Ti\uc5d0\uc11c\ub294 \ud559\uc2b5 \uc2dc\uac04\uc774 \ubb34\ub824 17\ubd84\uc774\ub098 \uc904\uc5b4\ub4e4\uc5c8\uc2b5\ub2c8\ub2e4. \ud559\uc2b5 \uc18d\ub3c4\uac00 \uc57d 1.46\ubc30 \ube68\ub77c\uc9c4 \uc148\uc774\uc8e0. GPU Memory\ub3c4 \uc801\uac8c \uc7a1\uc544\uba39\uc73c\uba74\uc11c\uc694!<\/p>\n\n<p>AMP\ub294 \uc190\ud574\ubcf4\ub294 \uac83\uc774 \uac70\uc758 \uc5c6\uc73c\uba74\uc11c GPU Memory\ub97c \uc801\uac8c \uc7a1\uc544\uba39\uc5b4\uc11c Batch Size\ub97c \ud0a4\uc6b0\uac70\ub098 \ub354 \ud070 Model\uc744 \ud559\uc2b5\uc2dc\ud0ac \uc218 \uc788\ub294\uac8c \uac00\uc7a5 \ud070 \uc7a5\uc810\uc774\uba70, \ucd5c\uc2e0 GPU\uc5d0\uc11c\ub294 \ud559\uc2b5 \uc2dc\uac04\ub3c4 \uc904\uc5b4\ub4dc\ub294 \ud6a8\uacfc\ub97c \uc5bb\uc744 \uc218 \uc788\uc5b4\uc11c \uac70\uc758 \ud544\uc218\ub85c \uc0ac\uc6a9\ud574\uc57c \ud558\ub294 \uae30\ub2a5\uc774\ub77c\uace0 \uc0dd\uac01\ud569\ub2c8\ub2e4.<\/p>\n\n<p>NVIDIA\uc5d0\uc11c \uacf5\uc2dd\uc801\uc73c\ub85c \uc81c\uacf5\ud558\ub294 <a href=\"https:\/\/github.com\/NVIDIA\/DeepLearningExamples\" target=\"_blank\"><b> DeepLearningExamples <\/b><\/a> \uc5d0\uc11c \ub2e4\ub8ec Image Classification, Object Detection, Segmentation, Natural Language Processing, Recommender Systems, Speech to Text, Text to Speech \uc5d0\uc11c\ub294 \ub300\ubd80\ubd84 \uc131\ub2a5\uc774 \uc88b\uc544\uc9d1\ub2c8\ub2e4. \ubb3c\ub860 \ubaa8\ub4e0 \uacbd\uc6b0\uc5d0 \uc131\ub2a5\uc774 \uc88b\uc544\uc9c0\uc9c4 \uc54a\uc2b5\ub2c8\ub2e4. \uacf5\uc2dd \ubb38\uc11c\uc5d0\uc11c \ub2e4\ub8e8\uace0 \uc788\uc9c0 \uc54a\uc740 Video Classification\uc5d0 AMP\ub97c \uc801\uc6a9\ud574\ubd24\ub294\ub370 \uc800\ub294 \uc624\ud788\ub824 GPU Memory\uac00 \ub298\uc5b4\ub098\uc11c \uc0ac\uc6a9\ud558\uc9c0 \ubabb\ud588\uc2b5\ub2c8\ub2e4. \uc6d0\uc778\uc740 \uc544\uc9c1 \ubc1d\ud600\ub0b4\uc9c0 \ubabb\ud55c \uc0c1\ud669\uc785\ub2c8\ub2e4..<\/p>\n\n<p>\uc800\ub294 1\uac1c\uc758 GPU\ub85c 1\uac1c\uc758 \ubaa8\ub378\uc5d0 1\uac1c\uc758 Loss, 1\uac1c\uc758 Optimizer\ub97c \uc0ac\uc6a9\ud558\ub294 \uac00\uc7a5 \ub2e8\uc21c\ud55c \uacfc\uc815\ub9cc \ubcf4\uc5ec\ub4dc\ub838\ub294\ub370\uc694, Gradient\ub97c \ub2e4\uc591\ud558\uac8c \ub2e4\ub8e8\ub294 \uacbd\uc6b0\uc5d0 \ub300\ud55c \uc608\uc2dc\ub3c4 \uacf5\uc2dd \ubb38\uc11c\uc5d0\uc11c \ud655\uc778\ud558\uc2e4 \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/p>\n<ul>\n  <li><a href=\"https:\/\/pytorch.org\/docs\/stable\/notes\/amp_examples.html\" target=\"_blank\"><b> PyTorch Automatic Mixed Precision Examples <\/b><\/a><\/li>\n<\/ul>\n\n<blockquote> \uacb0\ub860 <\/blockquote>\n<p>\uc624\ub298\uc740 \uc9c0\ub09c \uae00\uc5d0 \uc774\uc5b4\uc11c, PyTorch 1.6\uc5d0 \uacf5\uc2dd\uc801\uc73c\ub85c \uc9c0\uc6d0\ub418\uae30 \uc2dc\uc791\ud55c Automatic Mixed Precision(AMP) \uae30\ub2a5\uc744 \uc9c1\uc811 \uc0ac\uc6a9\ud574\ubcf4\uae30 \uc704\ud574 Image Classification Codebase\ub97c \ub9cc\ub4e4\uace0 \uc2e4\ud5d8\uc744 \ud558\uc5ec \uacb0\uacfc\ub97c \uacf5\uc720 \ub4dc\ub838\uc2b5\ub2c8\ub2e4. \uc81c \uc2e4\ud5d8 \ud658\uacbd\uc5d0\uc11c\ub294 1080 Ti\uc5d0\uc11c\ub294 GPU Memory\ub9cc \uc904\uc5b4\ub4e4\uace0 \ud559\uc2b5 \uc2dc\uac04\uc740 \uc904\uc5b4\ub4e4\uc9c0 \uc54a\uc740 \ubc18\uba74, 2080 Ti\uc5d0\uc11c\ub294 GPU Memory, \ud559\uc2b5 \uc2dc\uac04\uc774 \ubaa8\ub450 \uc904\uc5b4\ub4dc\ub294 \uacb0\uacfc\ub97c \ubcf4\uc600\ub294\ub370\uc694, \ub354 \ub2e4\uc591\ud55c \ubaa8\ub378\uc5d0 \ub300\ud574 \uc2e4\ud5d8\uc744 \ud574\ubcf4\uba74 \uacbd\ud5a5\uc744 \uc790\uc138\ud788 \uc54c \uc218 \uc788\uc744 \uac83 \uac19\uae34 \ud569\ub2c8\ub2e4. \ub2e8 5~6\uc904 \uc815\ub3c4\uc758 \ucf54\ub4dc\ub9cc \ucd94\uac00\ud558\uba74 \ubc14\ub85c \uc0ac\uc6a9\ud560 \uc218 \uc788\ub294\ub370, \uc815\ud655\ub3c4 \uc190\uc2e4\ub3c4 \uac70\uc758 \uc5c6\uc73c\uba74\uc11c \uc5bb\uc744 \uc218 \uc788\ub294 \uc810\uc774 \ub9ce\uc740 \ub9cc\ud07c \ud604\uc7ac \uc9c4\ud589 \uc911\uc774\uc2e0 \uc5f0\uad6c\uc5d0 \ud55c \ubc88 \uc801\uc6a9\ud574\ubcf4\uc2dc\ub294 \uac83\uc744 \uad8c\uc7a5 \ub4dc\ub9bd\ub2c8\ub2e4. \uc800\ub294 \uc55e\uc73c\ub85c \uc790\uc8fc \uc0ac\uc6a9\ud560 \uac83 \uac19\ub124\uc694. \uc77d\uc5b4 \uc8fc\uc154\uc11c \uac10\uc0ac\ud569\ub2c8\ub2e4!<\/p>\n\n","<p>\uc548\ub155\ud558\uc138\uc694, \uc9c0\ub09c <a href=\"https:\/\/hoya012.github.io\/blog\/Mixed-Precision-Training\/\" target=\"_blank\"><b> \u201cMixed-Precision Training of Deep Neural Networks\u201d <\/b><\/a> \uae00\uc5d0 \uc774\uc5b4\uc11c \uc624\ub298\uc740 PyTorch 1.6\uc5d0\uc11c \uacf5\uc2dd \uc9c0\uc6d0\ud558\uae30 \uc2dc\uc791\ud55c Automatic Mixed Precision Training \uae30\ub2a5\uc744 \uc9c1\uc811 \uc2e4\ud5d8\ud574\ubcfc \uc218 \uc788\ub294 Tutorial \ucf54\ub4dc\uc640 \uc124\uba85\uc744 \uae00\ub85c \uc791\uc131\ud558\uc600\uc2b5\ub2c8\ub2e4.<\/p>\n\n"],"pubDate":"Tue, 25 Aug 2020 00:00:00 +0000","link":"https:\/\/hoya012.github.io\/\/blog\/Image-Classification-with-Mixed-Precision-Training-PyTorch-Tutorial\/","guid":"https:\/\/hoya012.github.io\/\/blog\/Image-Classification-with-Mixed-Precision-Training-PyTorch-Tutorial\/"}]}}