发新帖

图片分割模型训练问题(不稳定)

[复制链接]
60 0

快来加入 TensorFlowers 大家庭!

您需要 登录 才可以下载或查看,没有帐号?加入社区

x
在最近训练图片分割模型的时候,会遇到一个很奇怪的问题: loss和mean_iou 一起下降。。。随机发生(一般来讲loss函数会下降,然后mean_iou会上升),估计是跟参数初始化有关,但是我也不知道具体的所以然,跑来这里问问。
  1. Epoch 1/50

  2.   1/100 [..............................] - ETA: 29:49 - loss: 3.9435 - mean_iou: 0.0000e+00
  3.   2/100 [..............................] - ETA: 15:02 - loss: 2.1499 - mean_iou: 0.1280   
  4.   3/100 [..............................] - ETA: 10:06 - loss: 1.4744 - mean_iou: 0.1610
  5.   4/100 [>.............................] - ETA: 7:37 - loss: 1.1493 - mean_iou: 0.1646
  6.   5/100 [>.............................] - ETA: 6:08 - loss: 0.9578 - mean_iou: 0.1650
  7.   6/100 [>.............................] - ETA: 5:09 - loss: 0.8293 - mean_iou: 0.1654
  8.   7/100 [=>............................] - ETA: 4:26 - loss: 0.7303 - mean_iou: 0.1660
  9.   8/100 [=>............................] - ETA: 3:54 - loss: 0.6543 - mean_iou: 0.1655
  10.   9/100 [=>............................] - ETA: 3:29 - loss: 0.5980 - mean_iou: 0.1644
  11. 10/100 [==>...........................] - ETA: 3:09 - loss: 0.5532 - mean_iou: 0.1635
  12. 11/100 [==>...........................] - ETA: 2:53 - loss: 0.5168 - mean_iou: 0.1629
  13. 12/100 [==>...........................] - ETA: 2:39 - loss: 0.4864 - mean_iou: 0.1625
  14. 13/100 [==>...........................] - ETA: 2:27 - loss: 0.4599 - mean_iou: 0.1622
  15. 14/100 [===>..........................] - ETA: 2:17 - loss: 0.4342 - mean_iou: 0.1620
  16. 15/100 [===>..........................] - ETA: 2:08 - loss: 0.4140 - mean_iou: 0.1614
  17. 16/100 [===>..........................] - ETA: 2:00 - loss: 0.3973 - mean_iou: 0.1607
  18. 17/100 [====>.........................] - ETA: 1:54 - loss: 0.3805 - mean_iou: 0.1603
  19. 18/100 [====>.........................] - ETA: 1:47 - loss: 0.3655 - mean_iou: 0.1597
  20. 19/100 [====>.........................] - ETA: 1:42 - loss: 0.3537 - mean_iou: 0.1591
  21. 20/100 [=====>........................] - ETA: 1:37 - loss: 0.3420 - mean_iou: 0.1587
  22. 21/100 [=====>........................] - ETA: 1:32 - loss: 0.3313 - mean_iou: 0.1582
  23. 22/100 [=====>........................] - ETA: 1:28 - loss: 0.3222 - mean_iou: 0.1577
  24. 23/100 [=====>........................] - ETA: 1:24 - loss: 0.3131 - mean_iou: 0.1572
  25. 24/100 [======>.......................] - ETA: 1:21 - loss: 0.3060 - mean_iou: 0.1567
  26. 25/100 [======>.......................] - ETA: 1:18 - loss: 0.2990 - mean_iou: 0.1564
  27. 26/100 [======>.......................] - ETA: 1:14 - loss: 0.2919 - mean_iou: 0.1560
  28. 27/100 [=======>......................] - ETA: 1:12 - loss: 0.2858 - mean_iou: 0.1556
  29. 28/100 [=======>......................] - ETA: 1:09 - loss: 0.2797 - mean_iou: 0.1553
  30. 29/100 [=======>......................] - ETA: 1:06 - loss: 0.2738 - mean_iou: 0.1549
  31. 30/100 [========>.....................] - ETA: 1:04 - loss: 0.2694 - mean_iou: 0.1545
  32. 31/100 [========>.....................] - ETA: 1:02 - loss: 0.2645 - mean_iou: 0.1542
  33. 32/100 [========>.....................] - ETA: 1:00 - loss: 0.2597 - mean_iou: 0.1538
  34. 33/100 [========>.....................] - ETA: 58s - loss: 0.2558 - mean_iou: 0.1534
  35. 34/100 [=========>....................] - ETA: 56s - loss: 0.2514 - mean_iou: 0.1531
  36. 35/100 [=========>....................] - ETA: 54s - loss: 0.2474 - mean_iou: 0.1527
  37. 36/100 [=========>....................] - ETA: 52s - loss: 0.2437 - mean_iou: 0.1523
  38. 37/100 [==========>...................] - ETA: 51s - loss: 0.2399 - mean_iou: 0.1518
  39. 38/100 [==========>...................] - ETA: 49s - loss: 0.2367 - mean_iou: 0.1514
  40. 39/100 [==========>...................] - ETA: 47s - loss: 0.2334 - mean_iou: 0.1509
  41. 40/100 [===========>..................] - ETA: 46s - loss: 0.2307 - mean_iou: 0.1505
  42. 41/100 [===========>..................] - ETA: 45s - loss: 0.2279 - mean_iou: 0.1500
  43. 42/100 [===========>..................] - ETA: 43s - loss: 0.2253 - mean_iou: 0.1496
  44. 43/100 [===========>..................] - ETA: 42s - loss: 0.2224 - mean_iou: 0.1492
  45. 44/100 [============>.................] - ETA: 41s - loss: 0.2201 - mean_iou: 0.1488
  46. 45/100 [============>.................] - ETA: 39s - loss: 0.2175 - mean_iou: 0.1484
  47. 46/100 [============>.................] - ETA: 38s - loss: 0.2150 - mean_iou: 0.1479
  48. 47/100 [=============>................] - ETA: 37s - loss: 0.2128 - mean_iou: 0.1475
  49. 48/100 [=============>................] - ETA: 36s - loss: 0.2105 - mean_iou: 0.1471
  50. 49/100 [=============>................] - ETA: 35s - loss: 0.2082 - mean_iou: 0.1467
  51. 50/100 [==============>...............] - ETA: 34s - loss: 0.2062 - mean_iou: 0.1463
  52. 51/100 [==============>...............] - ETA: 33s - loss: 0.2043 - mean_iou: 0.1459
  53. 52/100 [==============>...............] - ETA: 32s - loss: 0.2023 - mean_iou: 0.1455
  54. 53/100 [==============>...............] - ETA: 31s - loss: 0.2005 - mean_iou: 0.1451
  55. 54/100 [===============>..............] - ETA: 30s - loss: 0.1986 - mean_iou: 0.1448
  56. 55/100 [===============>..............] - ETA: 29s - loss: 0.1971 - mean_iou: 0.1444
  57. 56/100 [===============>..............] - ETA: 28s - loss: 0.1955 - mean_iou: 0.1440
  58. 57/100 [================>.............] - ETA: 27s - loss: 0.1937 - mean_iou: 0.1437
  59. 58/100 [================>.............] - ETA: 26s - loss: 0.1922 - mean_iou: 0.1433
  60. 59/100 [================>.............] - ETA: 25s - loss: 0.1907 - mean_iou: 0.1429
  61. 60/100 [=================>............] - ETA: 25s - loss: 0.1890 - mean_iou: 0.1426
  62. 61/100 [=================>............] - ETA: 24s - loss: 0.1875 - mean_iou: 0.1423
  63. 62/100 [=================>............] - ETA: 23s - loss: 0.1860 - mean_iou: 0.1419
  64. 63/100 [=================>............] - ETA: 22s - loss: 0.1847 - mean_iou: 0.1416
  65. 64/100 [==================>...........] - ETA: 21s - loss: 0.1835 - mean_iou: 0.1412
  66. 65/100 [==================>...........] - ETA: 21s - loss: 0.1826 - mean_iou: 0.1409
  67. 66/100 [==================>...........] - ETA: 20s - loss: 0.1812 - mean_iou: 0.1406
  68. 67/100 [===================>..........] - ETA: 19s - loss: 0.1804 - mean_iou: 0.1403
  69. 68/100 [===================>..........] - ETA: 18s - loss: 0.1793 - mean_iou: 0.1399
  70. 69/100 [===================>..........] - ETA: 18s - loss: 0.1780 - mean_iou: 0.1396
  71. 70/100 [====================>.........] - ETA: 17s - loss: 0.1768 - mean_iou: 0.1393
  72. 71/100 [====================>.........] - ETA: 16s - loss: 0.1759 - mean_iou: 0.1390
  73. 72/100 [====================>.........] - ETA: 16s - loss: 0.1748 - mean_iou: 0.1387
  74. 73/100 [====================>.........] - ETA: 15s - loss: 0.1734 - mean_iou: 0.1384
  75. 74/100 [=====================>........] - ETA: 14s - loss: 0.1726 - mean_iou: 0.1381
  76. 75/100 [=====================>........] - ETA: 14s - loss: 0.1717 - mean_iou: 0.1379
  77. 76/100 [=====================>........] - ETA: 13s - loss: 0.1712 - mean_iou: 0.1376
  78. 77/100 [======================>.......] - ETA: 12s - loss: 0.1701 - mean_iou: 0.1373
  79. 78/100 [======================>.......] - ETA: 12s - loss: 0.1690 - mean_iou: 0.1370
  80. 79/100 [======================>.......] - ETA: 11s - loss: 0.1680 - mean_iou: 0.1367
  81. 80/100 [=======================>......] - ETA: 11s - loss: 0.1672 - mean_iou: 0.1365
  82. 81/100 [=======================>......] - ETA: 10s - loss: 0.1663 - mean_iou: 0.1362
  83. 82/100 [=======================>......] - ETA: 9s - loss: 0.1658 - mean_iou: 0.1359
  84. 83/100 [=======================>......] - ETA: 9s - loss: 0.1649 - mean_iou: 0.1357
  85. 84/100 [========================>.....] - ETA: 8s - loss: 0.1641 - mean_iou: 0.1354
  86. 85/100 [========================>.....] - ETA: 8s - loss: 0.1633 - mean_iou: 0.1352
  87. 86/100 [========================>.....] - ETA: 7s - loss: 0.1627 - mean_iou: 0.1349
  88. 87/100 [=========================>....] - ETA: 6s - loss: 0.1618 - mean_iou: 0.1347
  89. 88/100 [=========================>....] - ETA: 6s - loss: 0.1612 - mean_iou: 0.1344
  90. 89/100 [=========================>....] - ETA: 5s - loss: 0.1605 - mean_iou: 0.1342
  91. 90/100 [==========================>...] - ETA: 5s - loss: 0.1596 - mean_iou: 0.1340
  92. 91/100 [==========================>...] - ETA: 4s - loss: 0.1586 - mean_iou: 0.1337
  93. 92/100 [==========================>...] - ETA: 4s - loss: 0.1577 - mean_iou: 0.1335
  94. 93/100 [==========================>...] - ETA: 3s - loss: 0.1570 - mean_iou: 0.1333
  95. 94/100 [===========================>..] - ETA: 3s - loss: 0.1562 - mean_iou: 0.1330
  96. 95/100 [===========================>..] - ETA: 2s - loss: 0.1561 - mean_iou: 0.1328
  97. 96/100 [===========================>..] - ETA: 2s - loss: 0.1554 - mean_iou: 0.1326
  98. 97/100 [============================>.] - ETA: 1s - loss: 0.1548 - mean_iou: 0.1324
  99. 98/100 [============================>.] - ETA: 1s - loss: 0.1541 - mean_iou: 0.1321
  100. 99/100 [============================>.] - ETA: 0s - loss: 0.1536 - mean_iou: 0.1319
  101. 100/100 [==============================] - 51s 511ms/step - loss: 0.1529 - mean_iou: 0.1317
  102. Epoch 2/50

  103.   1/100 [..............................] - ETA: 32s - loss: 0.1072 - mean_iou: 0.1100
  104.   2/100 [..............................] - ETA: 32s - loss: 0.1023 - mean_iou: 0.1100
  105.   3/100 [..............................] - ETA: 32s - loss: 0.0973 - mean_iou: 0.1099
  106.   4/100 [>.............................] - ETA: 31s - loss: 0.0917 - mean_iou: 0.1098
  107.   5/100 [>.............................] - ETA: 31s - loss: 0.0989 - mean_iou: 0.1096
复制代码



具体代码在这里 https://github.com/shenshutao/image_segmentation,写错了帮忙告诉我,还没写完,还在改 。。。


我知道答案 回答被采纳将会获得10 金币 + 30 金币 已有0人回答
本楼点评(0) 收起
您需要登录后才可以回帖 登录 | 加入社区

本版积分规则

主题

帖子

16

积分
快速回复 返回顶部 返回列表