kuroの覚え書き

96の個人的覚え書き

Fujitsu PRIMERGY TX1310M3にCentOSをインストールする

昨年度末にディスコン間近と思われるTX1310M3を購入したっきり、全然時間がなくて放置してあったのだが、ちょっと触ってみる。

まず、これまでのRX200や300とはBIOSUEFIになっている点でかなり勝手が違う。同じUEFIのRX1330M3ともなにやら違っていてこのTX1310M3は特に曲者らしい・・・

色々やってみたがembedded MegaRAID (software RAID)がCentOSでもUbuntuでもうまく使えない。

暇を見つけてやってみたことを並べていくことにする。

買ったままの状態では何もOSが入ってないのでF2を押さなくても勝手にUEFIの画面に入る。
f:id:k-kuro:20200521191421j:plain

AdvancedからLSI Software RAID Configuration Utilityに進む。
f:id:k-kuro:20200521191838j:plain

ここからのRAID設定はややこしいので、
富士通 PRIMERGY TX1310 M3 ソフトウェアRAID設定手順 | PGDブログ
こちら参照で。

ちなみにAdvanced>SATA ConfigurationでAHCIが選択されているとEmbedded RAIDがスキップされ、BIOS上にもメニューが表示されない。
RAIDに変更してリセットするとこのメニューが現れる。

【PRIMERGY】uEFI時のEmbedded SATA RAIDの設定 | OTTOSERVER "TECH" Blog
こちらの説明によると旧BIOSモードに切り替えてからのRAID設定ができないと言っているがそんなことはなく
AdvancedでCSM Configurationをenableにして再起動してやると、起動シークエンスの途中でRAID設定に入ることができるはずだ。

LSIの昔ながらのインターフェースで設定できるので、なれた人ならこちらのほうがやりやすかろう。

さて、RAIDが設定できたら早速インストール。
CentOS8.1なら標準でドライバを持っているらしいという情報を得たので、試しに入れてみようと8.1を探してみると、どうやら8以降はインストーラが片面2層DVD出ないと入り切らないサイズになっているらしい。そんなディスクもない、ということでUSBメモリにネットワークインストーラを入れてやってUSBからのインストールを試してみる。
f:id:k-kuro:20200521193443j:plain
いけそうでいけない。anacondaが起動してインストーラが立ち上がった時点でこのようなエラーが出て止まってしまう。

ネットワークインストールの問題かもしれないので、とりあえず何も考えずにCentOS7のインストーラディスクを装着して起動してみる。
ちなみに今度は色々枯れたちょっと古めのバージョン7.5のディスクでやってみる。

まあ、当然というかインストーラRAIDディスクにアクセスできず、行き詰まる。

次、MEGASR2ドライバを読ませる。
【PRIMERGY】PRIMERGY M3世代へのCentOSインストールについて【CentOS】 | OTTOSERVER "TECH" Blog
ドライバは7.5のものをSDカードに仕込んでおく。

http://azby.fmworld.net/app/customer/driversearch/ia/drviadownload?driverNumber=F1027982

お約束の
インストール項目選択で「e」を押す。
quietを消して

modprobe.blacklist=ahci inst.dd

を追加しctrl + x
でドライバを読ませる。

あれ?ちゃんとインストール出来たよ。
前やったときはどうしてもドライバが動かなかったのに。ドライバのバージョンが上がったか?

RNAmotifのコンパイル(OSX)

config.hは以下のようになっている。

#  Edit the configuration variables below to match your system.  The default
#  values given here should work on GNU/Linux systems.

CC= gcc
CFLAGS= -O2 -Wall

#  Notes: clang can replace gcc above;
#         for icc (version 13.0.1) you need to use -O1 in cflags.

#  The yacc compiler is typically "yacc" (for the *real* yacc), "byacc"
#  (for the Berkeley equivalent) or "bison -y" (to get yacc compatible output 
#  from bison).  Note that /bin/yacc is sometimes a symlink to bison; on
#  such systems you need the -y flag.

#  Note that you need to get version 1.875 or later of bison.  Since this 
#  was released in 2003, that should not be much of a problem....
#  Go to http://www.gnu.org/software/bison/bison.html for more information.

YACC = bison -y

#  The GNU version of lex is called flex.  So the following variable is 
#  either "lex" or "flex", depending upon your system.

LEX = flex

# Uncomment & use these 3 lines to use Genbank in addition to FASTN:

# GBHOME=/home/macke/gensearch
# GBLIB=$(GBHOME)/libgb.a
# CFLAGS=-O2 -DUSE_GENBANK

you may need to locate the C-compiler, lex or yacc, and set compiler
flags.
と説明があったのでbisonをインストールしておく。

$ brew install bison

・・・

bison is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have bison first in your PATH run:
  echo 'export PATH="/usr/local/opt/bison/bin:$PATH"' >> ~/.bash_profile

For compilers to find bison you may need to set:
  export LDFLAGS="-L/usr/local/opt/bison/lib"

言われたとおりPATHを通しておく。

lexもいるようなので

$ brew install flex

・・・

flex is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have flex first in your PATH run:
  echo 'export PATH="/usr/local/opt/flex/bin:$PATH"' >> ~/.bash_profile

For compilers to find flex you may need to set:
  export LDFLAGS="-L/usr/local/opt/flex/lib"
  export CPPFLAGS="-I/usr/local/opt/flex/include"

これまたPATHを通しておく。


さてmakeしてみる。

$ make
cd src; make
gcc -O2 -Wall   -c -o rnamot.o rnamot.c
flex rmlex.l
bison -y -d -v -t rmgrm.y
rmgrm.y: warning: 1 shift/reduce conflict [-Wconflicts-sr]
gcc -O2 -Wall   -c -o compile.o compile.c
gcc -O2 -Wall   -c -o dbutil.o dbutil.c
gcc -O2 -Wall   -c -o dump.o dump.c
gcc -O2 -Wall   -c -o efn.o efn.c
gcc -O2 -Wall   -c -o efn2.o efn2.c
gcc -O2 -Wall   -c -o errormsg.o errormsg.c
gcc -O2 -Wall   -c -o find_motif.o find_motif.c
gcc -O2 -Wall   -c -o getargs.o getargs.c
gcc -O2 -Wall   -c -o log.o log.c
gcc -O2 -Wall   -c -o mm_regexp.o mm_regexp.c
./mk_dumpnode.h.sh y.tab.h > dumpnode.h
gcc -O2 -Wall   -c -o node.o node.c
gcc -O2 -Wall   -c -o preprocessor.o preprocessor.c
gcc -O2 -Wall   -c -o regexp.o regexp.c
regexp.c:344:15: warning: implicit conversion from 'int' to 'char' changes value from 255
      to -1 [-Wconstant-conversion]
                                                *ep++ = 255;
                                                      ~ ^~~
1 warning generated.
gcc -O2 -Wall   -c -o score.o score.c
gcc -O2 -Wall   -c -o split.o split.c
gcc -O2 -Wall   -c -o y.tab.o y.tab.c
y.tab.c:1601:16: warning: implicit declaration of function 'yylex' is invalid in C99
      [-Wimplicit-function-declaration]
      yychar = yylex ();
               ^
y.tab.c:2631:7: warning: implicit declaration of function 'yyerror' is invalid in C99
      [-Wimplicit-function-declaration]
      yyerror (YY_("syntax error"));
      ^
y.tab.c:2742:3: warning: implicit declaration of function 'yyerror' is invalid in C99
      [-Wimplicit-function-declaration]
  yyerror (YY_("memory exhausted"));
  ^
In file included from rmgrm.y:545:
lex.yy.c:1691:17: warning: unused function 'yyunput' [-Wunused-function]
    static void yyunput (int c, char * yy_bp )
                ^
4 warnings generated.
gcc	-O2 -Wall -o rnamotif rnamot.o compile.o dbutil.o dump.o efn.o efn2.o errormsg.o find_motif.o getargs.o log.o mm_regexp.o node.o preprocessor.o regexp.o score.o split.o y.tab.o	  -lm
gcc -O2 -Wall   -c -o rmprune.o rmprune.c
gcc	-O2 -Wall -o rmprune rmprune.o split.o log.o
gcc -O2 -Wall   -c -o rm2ct.o rm2ct.c
gcc	-O2 -Wall -o rm2ct rm2ct.o split.o log.o
gcc -O2 -Wall   -c -o rmfmt.o rmfmt.c
gcc	-O2 -Wall -o rmfmt rmfmt.o split.o log.o
gcc -O2 -Wall   -c -o efn_drv.o efn_drv.c
gcc	-O2 -Wall -o efn_drv efn_drv.o efn.o split.o log.o -lm
gcc -O2 -Wall   -c -o efn2_drv.o efn2_drv.c
gcc	-O2 -Wall -o efn2_drv efn2_drv.o efn2.o split.o log.o -lm 

warningはでたものの、makeはできたらしく、srcディレクトリに色々できている。

testを走らせてみる。

$ make test
cd test; make
=====================================================
Running nanlin test:

nanlin.descr: complete descr length: min/max = 54/58
  PASSED
=====================================================
Running pk1 test:

pk1.descr: complete descr length: min/max = 22/46
  PASSED
=====================================================
Running pk_j1+2 test:

pk_j1+2.descr: complete descr length: min/max = 66/118
  PASSED
=====================================================
Running qu+tr test:

qu+tr.descr: complete descr length: min/max = 45/101
  PASSED
=====================================================
Running score.1 test:

score.1.descr: complete descr length: min/max = 18/24
  PASSED
=====================================================
Running score.2 test:

score.2.descr: complete descr length: min/max = 25/40
  PASSED
=====================================================
Running trna test:

trna.descr: complete descr length: min/max = 63/95
  PASSED
=====================================================
Running mp.ends test:

mp.ends.descr: complete descr length: min/max = 20/24
  PASSED
=====================================================
Running efn test:

・・・

< SEU28SRRNA -21.400 0  152 28 tcaac cgg  .  ggcg g  actg tcct cagt g  cgcc .   ccg cccgc
< TTERRDA    -21.700 0   59 27 cgagt cgc  .  gcgc .  ccgg ggcg ccgg .  gcgc g   gcg cacgg
< MEPMETH    -22.100 0 1234 32 acaat ggga .  tcc  ga cccc gaaa gggg ga  gga aa tccc ctaaa
< MVORR16SY  -22.700 0 1201 32 acaat ggga .  cgc  ga cccc gaaa gggg ga  gcg aa tccc ctaaa
< THCLRRNA   -23.200 0  261 22 ctgaa cc   .  ctc  .  cggg gaaa cccg .   gag .    gg gatgt
< HRTSSRRNA  -24.400 0  667 31 tcttt agtg gt ccgc .  ccc  ttcg  ggg a  gcgg gt cact ggcct
   FAILED (possibly; see if diffs above look OK)

FAILEDで終わった。なにかまずいのか。

echo 'export EFNDATA="/Users/kuro/local/bin/rnamotif-master/efndata"' >> ~/.bash_profile 

これでヨシ。

xgboostのインストール

前回xgboostのインストールがすんなりいかなくて色々回り道をしたが、ちゃんとエラーの内容を紐解けば正しいやり方がわかったろうに、ということでやり直し。

まず、一旦戻ってpip3のところから。

$ pip3 list
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Package            Version    
------------------ -----------
abifpy             0.9        
alembic            0.9.5

・・・

WARNING: You are using pip version 20.0.2; however, version 20.1 is available.
You should consider upgrading via the '/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 -m pip install --upgrade pip' command.

ここでまずひとつわかることはpipの使い方が

$ pip3 list

というやり方から

$ python3 -m pip list

というやり方に今後は変えますよ、ということ。

あとpipのバージョンが古そうなので

$ python3 -m pip install --upgrade pip

で早速バージョンアップ。

tensorflow==1.14
keras
は正常にインストールできて

$ python3 -m pip install xgboost
Defaulting to user installation because normal site-packages is not writeable
Collecting xgboost
  Downloading xgboost-1.0.2.tar.gz (821 kB)
     |████████████████████████████████| 821 kB 3.8 MB/s 
    ERROR: Command errored out with exit status 1:
     command: /Library/Frameworks/Python.framework/Versions/3.6/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/sf/3lk5x37135586w0jgqyjq_yc0000gn/T/pip-install-c294olba/xgboost/setup.py'"'"'; __file__='"'"'/private/var/folders/sf/3lk5x37135586w0jgqyjq_yc0000gn/T/pip-install-c294olba/xgboost/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/sf/3lk5x37135586w0jgqyjq_yc0000gn/T/pip-pip-egg-info-f1lsy3uh
         cwd: /private/var/folders/sf/3lk5x37135586w0jgqyjq_yc0000gn/T/pip-install-c294olba/xgboost/
    Complete output (27 lines):
    ++ pwd
    + oldpath=/private/var/folders/sf/3lk5x37135586w0jgqyjq_yc0000gn/T/pip-install-c294olba/xgboost
    + cd ./xgboost/
    + mkdir -p build
    + cd build
    + cmake ..
    ./xgboost/build-python.sh: line 21: cmake: command not found
    + echo -----------------------------
    -----------------------------
    + echo 'Building multi-thread xgboost failed'
    Building multi-thread xgboost failed
    + echo 'Start to build single-thread xgboost'
    Start to build single-thread xgboost
    + cmake .. -DUSE_OPENMP=0
    ./xgboost/build-python.sh: line 27: cmake: command not found
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/sf/3lk5x37135586w0jgqyjq_yc0000gn/T/pip-install-c294olba/xgboost/setup.py", line 42, in <module>
        LIB_PATH = libpath['find_lib_path']()
      File "/private/var/folders/sf/3lk5x37135586w0jgqyjq_yc0000gn/T/pip-install-c294olba/xgboost/xgboost/libpath.py", line 50, in find_lib_path
        'List of candidates:\n' + ('\n'.join(dll_path)))
    XGBoostLibraryNotFound: Cannot find XGBoost Library in the candidate path, did you install compilers and run build.sh in root path?
    List of candidates:
    /private/var/folders/sf/3lk5x37135586w0jgqyjq_yc0000gn/T/pip-install-c294olba/xgboost/xgboost/libxgboost.dylib
    /private/var/folders/sf/3lk5x37135586w0jgqyjq_yc0000gn/T/pip-install-c294olba/xgboost/xgboost/../../lib/libxgboost.dylib
    /private/var/folders/sf/3lk5x37135586w0jgqyjq_yc0000gn/T/pip-install-c294olba/xgboost/xgboost/./lib/libxgboost.dylib
    /Library/Frameworks/Python.framework/Versions/3.6/xgboost/libxgboost.dylib
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

やはりこうなる。

./xgboost/build-python.sh: line 21: cmake: command not found

ここでわかることはcmakeがないよ、ってこと。
なのでcmakeをインストールしてやろうとすると

$ brew install cmake
Error: /usr/local/Cellar is not writable. You should change the
ownership and permissions of /usr/local/Cellar back to your
user account:
  sudo chown -R $(whoami) /usr/local/Cellar
Error: The following directories are not writable by your user:
/usr/local/Cellar
/usr/local/Homebrew
/usr/local/bin
/usr/local/etc
/usr/local/include
/usr/local/lib
/usr/local/opt
/usr/local/share

You should change the ownership of these directories to your user.
  sudo chown -R $(whoami) /usr/local/Cellar /usr/local/Homebrew /usr/local/bin /usr/local/etc /usr/local/include /usr/local/lib /usr/local/opt /usr/local/share

brewがコケる。

$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: The following directories are not writable by your user:
/usr/local/Cellar
/usr/local/Homebrew
/usr/local/bin
/usr/local/etc
/usr/local/include
/usr/local/lib
/usr/local/opt
/usr/local/share

You should change the ownership of these directories to your user.
  sudo chown -R $(whoami) /usr/local/Cellar /usr/local/Homebrew /usr/local/bin /usr/local/etc /usr/local/include /usr/local/lib /usr/local/opt /usr/local/share

Warning: The following directories do not exist:
/usr/local/sbin

You should create these directories and change their ownership to your account.
  sudo mkdir -p /usr/local/sbin
  sudo chown -R $(whoami) /usr/local/sbin

Warning: "config" scripts exist outside your system or Homebrew directories.
`./configure` scripts often look for *-config scripts to determine if
software packages are installed, and what additional flags to use when
compiling and linking.

Having additional scripts in your path can confuse software installed via
Homebrew if the config script overrides a system or Homebrew provided
script of the same name. We found the following "config" scripts:
  /Library/Frameworks/Python.framework/Versions/2.7/bin/python2-config
  /Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7-config
  /Library/Frameworks/Python.framework/Versions/2.7/bin/python-config
  /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6m-config
  /Library/Frameworks/Python.framework/Versions/3.6/bin/python3-config
  /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6-config

Warning: Putting non-prefixed coreutils in your path can cause gmp builds to fail.

Warning: Unbrewed dylibs were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected dylibs:
  /usr/local/lib/libtcl8.6.dylib
  /usr/local/lib/libtk8.6.dylib

Warning: Unbrewed header files were found in /usr/local/include.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

・・・

Warning: Unbrewed .pc files were found in /usr/local/lib/pkgconfig.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected .pc files:
  /usr/local/lib/pkgconfig/tcl.pc
  /usr/local/lib/pkgconfig/tk.pc

Warning: Unbrewed static libraries were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected static libraries:
  /usr/local/lib/libtclstub8.6.a
  /usr/local/lib/libtkstub8.6.a

Warning: You have unlinked kegs in your Cellar
Leaving kegs unlinked can lead to build-trouble and cause brews that depend on
those kegs to fail to run properly once built. Run `brew link` on these:
  texi2html

Warning: /usr/bin occurs before /usr/local/bin
This means that system-provided programs will be used instead of those
provided by Homebrew. The following tools exist at both paths:
  smtpd2.7.py
  idle2.7
  idle
  ctags
  git-receive-pack
  texi2pdf
  2to3-2.7
  etags
  easy_install
  pydoc
  python
  pythonw
  python2.7-config
  pydoc2.7
  git-upload-archive
  pythonw2.7
  emacs
  emacsclient
  texindex
  python-config
  info
  easy_install-2.7
  texi2dvi
  python2.7
  git-cvsserver
  infokey
  git-shell
  git-upload-pack
  smtpd.py
  git
  makeinfo
  install-info

Consider setting your PATH so that /usr/local/bin
occurs before /usr/bin. Here is a one-liner:
  echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bash_profile

Warning: Your Xcode (8.2.1) is too outdated.
Please update to Xcode 10.1 (or delete it).
Xcode can be updated from the App Store.


Warning: Your Xcode (8.2.1) is outdated.
Please update to Xcode 10.1 (or delete it).
Xcode can be updated from the App Store.


Warning: Broken symlinks were found. Remove them with `brew prune`:
  /usr/local/bin/ctags
  /usr/local/bin/ebrowse
  /usr/local/bin/emacs
  /usr/local/bin/emacsclient
  /usr/local/bin/etags
Warning: Use hdf5 instead of deprecated homebrew/science/hdf5
Warning: Use htslib instead of deprecated homebrew/science/htslib
Warning: Use bowtie2 instead of deprecated homebrew/science/bowtie2

とまあ盛大にWarningが出る。
この中の、最初のWarningの対処を言われたとおりにやる。

$ sudo chown -R $(whoami) /usr/local/Cellar /usr/local/Homebrew /usr/local/bin /usr/local/etc /usr/local/include /usr/local/lib /usr/local/opt /usr/local/share
Password:

するとbrew installがちゃんと使えるようになった。

$ brew install cmake
Updating Homebrew...
==> Downloading https://homebrew.bintray.com/bottles-portable-ruby/portable-ruby-2.6.3.mavericks.bottle.tar.gz
######################################################################## 100.0%
==> Pouring portable-ruby-2.6.3.mavericks.bottle.tar.gz
==> Homebrew has enabled anonymous aggregate formulae and cask analytics.
Read the analytics documentation (and how to opt-out) here:
  https://docs.brew.sh/Analytics
No analytics have been recorded yet (or will be during this `brew` run).

==> Auto-updated Homebrew!
Updated 4 taps (homebrew/core, homebrew/cask, brewsci/bio and brewsci/science).
==> New Formulae

・・・

==> Downloading https://homebrew.bintray.com/bottles/cmake-3.17.2.mojave.bottle.
==> Downloading from https://akamai.bintray.com/ed/edc5ec271841a8b8558f8d60ef510
######################################################################## 100.0%
==> Pouring cmake-3.17.2.mojave.bottle.tar.gz
==> Caveats
Emacs Lisp files have been installed to:
  /usr/local/share/emacs/site-lisp/cmake
==> Summary
🍺  /usr/local/Cellar/cmake/3.17.2: 6,156 files, 58.1MB

ここまでやってやると

$ python3 -m pip install xgboost
Defaulting to user installation because normal site-packages is not writeable
Collecting xgboost
  Using cached xgboost-1.0.2.tar.gz (821 kB)
Requirement already satisfied: numpy in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from xgboost) (1.16.1)
Requirement already satisfied: scipy in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from xgboost) (0.19.1)
Building wheels for collected packages: xgboost
  Building wheel for xgboost (setup.py) ... done
  Created wheel for xgboost: filename=xgboost-1.0.2-cp36-cp36m-macosx_10_14_intel.whl size=3381969 sha256=f8ef36fabe464c9b019fe99671876efa3350dd3c4f275cb5a9b04f717eb0e6d9
  Stored in directory: /Users/kkuro/Library/Caches/pip/wheels/06/0a/03/1dd5317e4ad7882450a41265354839831f7094739ee401043c
Successfully built xgboost
Installing collected packages: xgboost
Successfully installed xgboost-1.0.2

このようにエラーなく、スッキリインストールが完了した。
gccのバージョンアップは不要だったわけだ。


最近のサーバ構成

サーバ   CPU               メモリ            理論性能
RX1330M3 E3-1230v6(4C/8T, 3.50GHz)   DDR4-2400 (PC4-19200E) UDIMM 64GB  224.0GFLOPS
RX300S7  E5-2667(6C/12T, 2.90GHz)x2  DDR3-1600 (PC3L-12800R) LV-RDIMM 32GB 278.4GFLOPS
RX300S7  E5-2643(4C/8T, 3.30GHz)x2  DDR3-1600 (PC3L-12800E) LV-UDIMM 24GB    211.2GFLOPS
RX200S7  E5-2630(6C/12T, 2.30GHz)x2  DDR3-1333 (PC3-10600R) RDIMM 44GB  220.8GFLOPS
RX300S7  E5-2620(6C/12T, 2.00GHz)x2  DDR3-1600 (PC3L-12800R) LV-RDIMM 24GB   192.0GFLOPS

NEC Express5800R110e-1e Pentium G2020 (2C/2T, 2.9GHz)  DDR3-1333 (PC3-10600E) UDIMM 4GB   #ファイルサーバ

E5600(Westmere-EP)は退役し、台数は減ったものの、十分な計算能力は確保していると思う。
各機メモリがもうちょっと欲しいところ。
1CPUあたり64GBくらいないとCufflinksがコケる。

この他TX1310M3も入手してあるが未整備。

ちなみに自宅用には
NEC Express5800T110f-e E3-1220v3(4C/4T, 3.30GHz) DDR3-1600 (PC3L-12800E) LV-UDIMM 8GB

THIRDWAVE R121-340 (CPU/Mem未実装)
がある。

PythonでGUIアプリを作る

プログラミングの練習も兼ねてPythonでゲームを作ってみる。
まずはtkinterの使い方練習。

import tkinter    #tkinterモジュールをインポート

root = tkinter.Tk()    # ウィンドウの部品を作る
root.geometry("400x300")      #ウィンドウのサイズを決める
root.title("PythonでGUIを扱う")   #ウィンドウのタイトルを指定
label = tkinter.Label(text="GUIのウィンドウを開く", font=("Times New Roman", 20))      #ラベルの部品を作る
label.place(x=80,y=60)    #ラベルを配置
button = tkinter.Button(text="button", command=button_click)      #ボタンを作る
button.place(x=80,y=100)        #ボタンを配置
text = tkinter.Entry(width=20)      #テキストの入力欄を作る
text.place(x=80,y=140)      #入力欄を配置

root.mainloop()   #ウィンドウを表示

f:id:k-kuro:20200503200139p:plain
tkinter

背景の挿入を試す。

import tkinter    #tkinterモジュールをインポート
from PIL import ImageTk

root = tkinter.Tk()    # ウィンドウの部品を作る
root.title("Canvasに画像を描画する")   #ウィンドウのタイトルを指定
canvas = tkinter.Canvas(width=480, height=300)      #ウィンドウのサイズを設定
canvas.pack()       #canvasを作成
img_bg = ImageTk.PhotoImage(file="id_1.png")        #イメージを読み込み
canvas.create_image(240, 150, image=img_bg)     #イメージの中心を設定

root.mainloop()   #ウィンドウを表示

最初、書籍通りにやってみると

Traceback (most recent call last):
  File "canvas_test.py", line 8, in <module>
    img_bg = tkinter.PhotoImage(file="id_1.png")
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tkinter/__init__.py", line 3545, in __init__
    Image.__init__(self, 'photo', name, cnf, master, **kw)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tkinter/__init__.py", line 3501, in __init__
    self.tk.call(('image', 'create', imgtype, name,) + options)
_tkinter.TclError: couldn't recognize data in image file "id_1.png"

こういうエラーが出た。
どうやらpython3の標準ライブラリのtkinterモジュールではpngが扱えないことが原因らしい。

img_bg = tkinter.PhotoImage(file="id_1.png")        #イメージを読み込み

こうなっていたからだめだった模様。
Pillowモジュールをインストールして

from PIL import ImageTk

とした上で

img_bg = ImageTk.PhotoImage(file="id_1.png") 

これでちゃんと表示されるようになった。

f:id:k-kuro:20200506003720p:plain
canvas


マウスの位置検知とヒットチェック(オブジェクトが重なっているかどうか判定)

import tkinter
from math import sqrt        #平方根の計算のため

def hitcheck_circle():
    dist = sqrt((x2-x1)*(x2-x1)+(y2-y1)*(y2-y1))
    if dist <= r1 + r2:    #中心間の距離が半径の和より短いか
        return True
    return False

def mouse_move(e):
    global x1, y1  #マウスの指す位置
    x1 = e.x
    y1 = e.y
    col = "green"
    if hitcheck_circle()== True:
        col = "yellow"
    canvas.delete("CIRC1")
    canvas.create_oval(x1-r1,y1-r1,x1+r1,y1+r1, fill=col, tag="CIRC1")

root = tkinter.Tk()
root.title("円によるヒットチェック")
canvas = tkinter.Canvas(width=400, height=400, bg="white")
canvas.pack()
canvas.bind("<Button>", mouse_move)     #マウスクリックしたところに円を置く

x1 = 50
y1 = 50
r1 = 40
canvas.create_oval(x1-r1,y1-r1,x1+r1,y1+r1, fill="green", tag="CIRC1")

x2 = 200
y2 = 200
r2 = 60
canvas.create_oval(x2-r2,y2-r2,x2+r2,y2+r2, fill="red")

root.mainloop()

f:id:k-kuro:20200506185939p:plain
hitcheck
だいぶゲームの基本ができてきたかも。


文字を入力してボタンを押した時に計算させる仕組み。これでこれまでWebアプリで作ってきたプログラムをそのままデスクトップアプリに作り変えることもできそうだ。

import tkinter    #tkinterモジュールをインポート
import math

def button_click():
    try:
        d = float(entry.get())
        a = math.radians(d)
        s = math.sin(a)
        c = math.cos(a)
        t = math.tan(a)
        label_s["text"] = "sin "+str(s)
        label_c["text"] = "cos "+str(c)
        label_t["text"] = "tan "+str(t)
        label_e["text"] = ""
    except:
        label_s["text"] = "sin "
        label_c["text"] = "cos "
        label_t["text"] = "tan "
        label_e["text"] = "角度を入力してください"

root = tkinter.Tk()    # ウィンドウの部品を作る
root.geometry("400x300")      #ウィンドウのサイズを決める
root.title("三角関数計算機")   #ウィンドウのタイトルを指定
button = tkinter.Button(text="計算する", command=button_click)      #ボタンを作る
button.place(x=120,y=20)        #ボタンを配置
entry = tkinter.Entry(width=10)      #テキストの入力欄を作る
entry.place(x=20,y=20)      #入力欄を配置
label_s = tkinter.Label(text="sin ", font=("Times New Roman", 20))
label_s.place(x=20, y=80)
label_c = tkinter.Label(text="cos ", font=("Times New Roman", 20))
label_c.place(x=20, y=120)
label_t = tkinter.Label(text="tan ", font=("Times New Roman", 20))
label_t.place(x=20, y=160)
label_e = tkinter.Label(text="角度を入力してください", font=("Times New Roman", 10))
label_e.place(x=20, y=60)
root.mainloop()   #ウィンドウを表示

f:id:k-kuro:20200506194214p:plain
起動時
f:id:k-kuro:20200506194239p:plain
計算させると

教師なし機械学習

deep learningの本を読めば読むほどに自分のやりたいことが教師あり深層学習ではできないんじゃないかと思い始めた。結局正解、不正解が明確に判定できる例が多数ないと、その特徴を抽出する学習が機能しないわけだが、世の中そんなに白黒はっきりした例が揃っていることなんてなかなかない。よくわからない集団の中から特徴を見分けて分類できるか、ということのほうが多いはずだ。

ということですっかり深層学習一辺倒になりつつある時代に逆行して、教師なし機械学習に戻ってくるという感じ。

早速書籍を購入し、ハンズオンを試していこうと思う。

さて心機一転Macに環境構築していく。

Pythonはすでに3.6がインストールされているので、tensorflowとkerasをpipでインストールする。
次にxgboostをインストールしようとしてちょっと詰まる。
書籍ではgitでダウンロードしてきたファイルからインストールしているがwin環境用のインストーラなのでインストールできない。

$ cd handson-unsupervised-learning/
$ cd xgboost/
$ pip3 install xgboost-0.81-cp36-cp36m-win_amd64.whl 
ERROR: xgboost-0.81-cp36-cp36m-win_amd64.whl is not a supported wheel on this platform.

よく見ると書籍の欄外に注釈として
pip install xgbootでインストールできると書いてあったので

$ pip3 install xgboost
Collecting xgboost
  Downloading xgboost-1.0.2.tar.gz (821 kB)
     |████████████████████████████████| 821 kB 5.0 MB/s 
    ERROR: Command errored out with exit status 1:
     command: /usr/local/opt/python/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/bh/ggq7fvb9581379cgprt5p9r40000gn/T/pip-install-4x26emtu/xgboost/setup.py'"'"'; __file__='"'"'/private/var/folders/bh/ggq7fvb9581379cgprt5p9r40000gn/T/pip-install-4x26emtu/xgboost/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/bh/ggq7fvb9581379cgprt5p9r40000gn/T/pip-pip-egg-info-s7yrttks
         cwd: /private/var/folders/bh/ggq7fvb9581379cgprt5p9r40000gn/T/pip-install-4x26emtu/xgboost/
    Complete output (27 lines):
    ++ pwd
    + oldpath=/private/var/folders/bh/ggq7fvb9581379cgprt5p9r40000gn/T/pip-install-4x26emtu/xgboost
    + cd ./xgboost/
    + mkdir -p build
    + cd build
    + cmake ..
    ./xgboost/build-python.sh: line 21: cmake: command not found
    + echo -----------------------------
    -----------------------------
    + echo 'Building multi-thread xgboost failed'
    Building multi-thread xgboost failed
    + echo 'Start to build single-thread xgboost'
    Start to build single-thread xgboost
    + cmake .. -DUSE_OPENMP=0
    ./xgboost/build-python.sh: line 27: cmake: command not found
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/bh/ggq7fvb9581379cgprt5p9r40000gn/T/pip-install-4x26emtu/xgboost/setup.py", line 42, in <module>
        LIB_PATH = libpath['find_lib_path']()
      File "/private/var/folders/bh/ggq7fvb9581379cgprt5p9r40000gn/T/pip-install-4x26emtu/xgboost/xgboost/libpath.py", line 50, in find_lib_path
        'List of candidates:\n' + ('\n'.join(dll_path)))
    XGBoostLibraryNotFound: Cannot find XGBoost Library in the candidate path, did you install compilers and run build.sh in root path?
    List of candidates:
    /private/var/folders/bh/ggq7fvb9581379cgprt5p9r40000gn/T/pip-install-4x26emtu/xgboost/xgboost/libxgboost.dylib
    /private/var/folders/bh/ggq7fvb9581379cgprt5p9r40000gn/T/pip-install-4x26emtu/xgboost/xgboost/../../lib/libxgboost.dylib
    /private/var/folders/bh/ggq7fvb9581379cgprt5p9r40000gn/T/pip-install-4x26emtu/xgboost/xgboost/./lib/libxgboost.dylib
    /usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/xgboost/libxgboost.dylib
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

これはコンパイラがまずいのか?

$ gcc -dumpversion
4.2.1

gccの最新バージョンは8らしいのだが、4と5で互換性がないとかで、とりあえず6入れてみようかなと思い、

$ brew install gcc6
Updating Homebrew...
==> Downloading https://homebrew.bintray.com/bottles-portable-ruby/portable-ruby-2.6.3.mavericks.bottle.tar.gz
######################################################################## 100.0%
==> Pouring portable-ruby-2.6.3.mavericks.bottle.tar.gz
==> Auto-updated Homebrew!
Updated 3 taps (brewsci/science, homebrew/cask and homebrew/core).

.........

Warning: You are using macOS 10.12.
We (and Apple) do not provide support for this old version.
You will encounter build failures with some formulae.
Please create pull requests instead of asking for help on Homebrew's GitHub,
Discourse, Twitter or IRC. You are responsible for resolving any issues you
experience while you are running this old version.

..........

ん?6は古いから保証しない?やはり素直に8を入れるべきなのか。
いや、OSXのバージョンが古いのか?

長い時間がかかってようやくgcc6が入った。
しかし、まだエラーは消えず。
cmakeが入ってないせいかもしれない。
というわけで

$ brew install make
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 2 taps (homebrew/cask and homebrew/core).
==> Updated Formulae
re2
==> Updated Casks
keka                                     mactracker

Warning: You are using macOS 10.12.
We (and Apple) do not provide support for this old version.
You will encounter build failures with some formulae.
Please create pull requests instead of asking for help on Homebrew's GitHub,
Discourse, Twitter or IRC. You are responsible for resolving any issues you
experience while you are running this old version.

..........

これまた長い時間がかかる。
しかしまだエラーが出てインストールできない。

$ pip3 install xgboost
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)': /simple/xgboost/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)': /simple/xgboost/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)': /simple/xgboost/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)': /simple/xgboost/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)': /simple/xgboost/
Could not fetch URL https://pypi.org/simple/xgboost/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/xgboost/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)) - skipping
ERROR: Could not find a version that satisfies the requirement xgboost (from versions: none)
ERROR: No matching distribution found for xgboost

次に試したのは
Installation Guide — xgboost 1.1.0-SNAPSHOT documentation
ここを参照して(というか最初にここ見ろよという話)

$ brew install libomp
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
balena-cli          emscripten          gatsby-cli          neomutt
cfn-lint            faudio              git-annex           proj
conserver           fluent-bit          git-quick-stats
dxpy                fwup                gopass

Warning: You are using macOS 10.12.
We (and Apple) do not provide support for this old version.
You will encounter build failures with some formulae.
Please create pull requests instead of asking for help on Homebrew's GitHub,
Discourse, Twitter or IRC. You are responsible for resolving any issues you
experience while you are running this old version.

==> Downloading https://github.com/llvm/llvm-project/releases/download/llvmorg-1
==> Downloading from https://github-production-release-asset-2e65be.s3.amazonaws
######################################################################## 100.0%
==> cmake . -DLIBOMP_INSTALL_ALIASES=OFF
==> make install
==> cmake . -DLIBOMP_ENABLE_SHARED=OFF -DLIBOMP_INSTALL_ALIASES=OFF
==> make install
🍺  /usr/local/Cellar/libomp/10.0.0: 9 files, 1.4MB, built in 45 seconds

やっとインストールできそう。

$ cd ../python-package/
Mac-mini-2014:python-package kkuro$ python3 setup.py install
/usr/local/lib/python3.6/site-packages/setuptools/dist.py:472: UserWarning: The version specified ('1.1.0-SNAPSHOT') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
  "details." % version
running install
running build

..........

今度はエラーなく完了したようだけど果たしてこれでいいのか?

Clusteringしたデータを折れ線グラフとして一気に作成

%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
import matplotlib as mpl
color_codes = {1:'#E60012', 2:'#F39800', 3:'#FFF100', 4:'#8FC31F', 5:'#009944', 6:'#009E96', 7:'#00A0E9', 8:'#0068B7', 9:'#1D2088', 10:'#920783', 11:'#E4007F', 0:'#E5004F'}
fig_x=10
fig_y=10
l_wide=0.15
numclusters = [0,1,2]
for num_cluster in numclusters:
    gcolor=color_codes[num_cluster]
    df = pd.read_csv('iris_k.csv', index_col=0)
    df = df[df['kmeans'].isin([num_cluster])]
    df = df.loc[:,['0','1','2','3']]
    df = df.T
    print(num_cluster)
    df.plot(color=gcolor, legend=False, figsize=(fig_x, fig_y), linewidth=l_wide )
    plt.savefig('g'+str(num_cluster)+'.png')

csvから該当箇所をちまちま切り出していては面倒臭すぎるのでpythonで一気にやってもらう。

f:id:k-kuro:20200221093256p:plain
こんな感じにk-meansで3クラスターに分けたirisデータの一部を使用すると
f:id:k-kuro:20200221094022p:plainf:id:k-kuro:20200221094024p:plainf:id:k-kuro:20200221094028p:plain
こんな感じにそれぞれのクラスターの折れ線が描ける。

Fujitsu Primergy TX1310M3にUbuntuをインストールしたい

ファイルサーバ兼深層学習プラットフォームとして使用するつもりでTX1310M3を入手してみた。

これまで富士通のサーバにはCentOS7をインストールして使ってきたのだが、深層学習プラットフォームとしてはRedhat系のCentOSよりDebian系のUbuntuの方が構築しやすいことがわかっていたのでUbuntuをインストールしたいのだが、富士通Ubuntuはあんまり相性が良くないのかもしれない。
というのもTX1310M3が備えているembedded software RAIDRedHat系でしかドライバが供給されていないので使えない。かといってUbuntuはCentOS7よりRAIDの対応がイマイチでOS組み込みのsoftware RAIDをOSのインストーラで簡単に設定できるようにはなっていないようだ。さてどうしたものか。
ヤフオクで2000円で入手したNECのExpress 5800 T110f-Eの方はhardware RAIDカードを持っているのでRAID上にUbuntuのインストールも簡単に行えた。
うーん、TX1310M3をファイルサーバにするならUbuntuではなくCentOS7にするべきだし、深層学習するなら起動ディスクはRAIDを組まずにUbuntuを入れるべきだ、ということでやはりどちらも専用で用意するしかないね。ファイルサーバはこれまで使ってきたExpress5800R110eの方に任せることにしてTX1310M3は深層学習専用とすることにしよう。
T110f-EはRAIDUbuntuが両立出来ているのでいいのだが、pci eスロットの制約があって結局RAIDビデオカードが両立できないという難しさ。やはり既製のサーバではなくカスタムメイドで専用機を建てるようにすべきなんだろうな。

deeplearning テキストの2値化

:~$ python3
Python 3.6.9 (default, Nov  7 2019, 10:44:02) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from keras.datasets import imdb
/usr/lib/python3/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
>>> (train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)
Downloading data from https://s3.amazonaws.com/text-datasets/imdb.npz
17465344/17464789 [==============================] - 14s 1us/step
>>> train_data[0]
[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 5244, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 5952, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 7486, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 5535, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 5345, 19, 178, 32]
>>> train_labels[0]
1
>>> max([max(sequence) for sequence in train_data])
9999
>>> word_index = imdb.get_word_index()
Downloading data from https://s3.amazonaws.com/text-datasets/imdb_word_index.json
1646592/1641221 [==============================] - 3s 2us/step
>>> reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])
>>> decode_review = ' '.join([reverse_word_index.get(i - 3, '?') for i in train_data[0]])
>>> decoded_review
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'decoded_review' is not defined
>>> decode_review
"? this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert ? is an amazing actor and now the same being director ? father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for ? and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also ? to the two little boy's that played the ? of norman and paul they were just brilliant children are often left out of the ? list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all"
>>> import numpy as np
>>> def vectorize_sequences(sequences, dimension=10000):
...     results = np.zeros((len(sequences),dimension))
...     for i, sequence in enumerate(sequences):
...             results[i, sequence] =1.
...     return results
... 
>>> x_train = vectorize_sequences(train_data)
>>> x_test = vectorize_sequences(test_data)
>>> x_train[0]
array([0., 1., 1., ..., 0., 0., 0.])
>>> y_train = np.asarray(train_labels).astype('float32')
>>> y_test = np.asarray(test_labels).astype('float32')
>>> from keras import models
>>> from keras import layers
>>> model = models.Sequential()
>>> model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))
>>> model.add(layers.Dense(16, activation='relu'))
>>> model.add(layers.Dense(1, activation='sigmoid'))
>>> model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
>>> x_val = x_train[:10000]
>>> partial_x_train = x_train[10000:]
>>> y_val = y_train[:10000]
>>> partial_y_train = y_train[10000:]
>>> model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])
>>> history = model.fit(partial_x_train, partial_y_train, epochs=20, batch_size=512, validation_data=(x_val, y_val))
2020-02-09 18:04:15.573627: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-02-09 18:04:15.591352: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-02-09 18:04:15.592341: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: CUDA_ERROR_SYSTEM_DRIVER_MISMATCH: system has unsupported display driver / cuda driver combination
2020-02-09 18:04:15.592388: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: kkuro-E5800-T110f-E
2020-02-09 18:04:15.592399: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: kkuro-E5800-T110f-E
2020-02-09 18:04:15.592530: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 440.59.0
2020-02-09 18:04:15.592562: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 440.48.2
2020-02-09 18:04:15.592573: E tensorflow/stream_executor/cuda/cuda_diagnostics.cc:313] kernel version 440.48.2 does not match DSO version 440.59.0 -- cannot find working devices in this configuration
2020-02-09 18:04:15.729381: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3092885000 Hz
2020-02-09 18:04:15.738386: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xa13df70 executing computations on platform Host. Devices:
2020-02-09 18:04:15.738459: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2020-02-09 18:04:15.883609: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

Train on 15000 samples, validate on 10000 samples
Epoch 1/20
  512/15000 [>.............................] - ETA: 10s - loss: 0.6936 - acc: 0. 1536/15000 [==>...........................] - ETA: 3s - loss: 0.6820 - acc: 0.5 3072/15000 [=====>........................] - ETA: 1s - loss: 0.6504 - acc: 0.6 4608/15000 [========>.....................] - ETA: 1s - loss: 0.6226 - acc: 0.6 6144/15000 [===========>..................] - ETA: 0s - loss: 0.5979 - acc: 0.7 7168/15000 [=============>................] - ETA: 0s - loss: 0.5834 - acc: 0.7 8704/15000 [================>.............] - ETA: 0s - loss: 0.5644 - acc: 0.710240/15000 [===================>..........] - ETA: 0s - loss: 0.5449 - acc: 0.711776/15000 [======================>.......] - ETA: 0s - loss: 0.5276 - acc: 0.712800/15000 [========================>.....] - ETA: 0s - loss: 0.5176 - acc: 0.714336/15000 [===========================>..] - ETA: 0s - loss: 0.5047 - acc: 0.715000/15000 [==============================] - 1s 96us/step - loss: 0.4989 - acc: 0.7990 - val_loss: 0.3895 - val_acc: 0.8521
Epoch 2/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.3523 - acc: 0.8 2048/15000 [===>..........................] - ETA: 0s - loss: 0.3424 - acc: 0.8 3584/15000 [======>.......................] - ETA: 0s - loss: 0.3343 - acc: 0.8 5120/15000 [=========>....................] - ETA: 0s - loss: 0.3295 - acc: 0.8 6656/15000 [============>.................] - ETA: 0s - loss: 0.3250 - acc: 0.8 8192/15000 [===============>..............] - ETA: 0s - loss: 0.3193 - acc: 0.8 9728/15000 [==================>...........] - ETA: 0s - loss: 0.3166 - acc: 0.811264/15000 [=====================>........] - ETA: 0s - loss: 0.3121 - acc: 0.812800/15000 [========================>.....] - ETA: 0s - loss: 0.3074 - acc: 0.814336/15000 [===========================>..] - ETA: 0s - loss: 0.3026 - acc: 0.815000/15000 [==============================] - 1s 60us/step - loss: 0.3011 - acc: 0.8980 - val_loss: 0.3204 - val_acc: 0.8742
Epoch 3/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.2231 - acc: 0.9 2048/15000 [===>..........................] - ETA: 0s - loss: 0.2363 - acc: 0.9 3584/15000 [======>.......................] - ETA: 0s - loss: 0.2362 - acc: 0.9 5120/15000 [=========>....................] - ETA: 0s - loss: 0.2305 - acc: 0.9 6656/15000 [============>.................] - ETA: 0s - loss: 0.2272 - acc: 0.9 8192/15000 [===============>..............] - ETA: 0s - loss: 0.2249 - acc: 0.9 9728/15000 [==================>...........] - ETA: 0s - loss: 0.2263 - acc: 0.911264/15000 [=====================>........] - ETA: 0s - loss: 0.2234 - acc: 0.912800/15000 [========================>.....] - ETA: 0s - loss: 0.2210 - acc: 0.914336/15000 [===========================>..] - ETA: 0s - loss: 0.2224 - acc: 0.915000/15000 [==============================] - 1s 61us/step - loss: 0.2227 - acc: 0.9269 - val_loss: 0.2949 - val_acc: 0.8825
Epoch 4/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.2064 - acc: 0.9 2048/15000 [===>..........................] - ETA: 0s - loss: 0.1824 - acc: 0.9 3584/15000 [======>.......................] - ETA: 0s - loss: 0.1813 - acc: 0.9 5120/15000 [=========>....................] - ETA: 0s - loss: 0.1775 - acc: 0.9 6656/15000 [============>.................] - ETA: 0s - loss: 0.1725 - acc: 0.9 8192/15000 [===============>..............] - ETA: 0s - loss: 0.1758 - acc: 0.9 9728/15000 [==================>...........] - ETA: 0s - loss: 0.1761 - acc: 0.911264/15000 [=====================>........] - ETA: 0s - loss: 0.1746 - acc: 0.912800/15000 [========================>.....] - ETA: 0s - loss: 0.1742 - acc: 0.914336/15000 [===========================>..] - ETA: 0s - loss: 0.1743 - acc: 0.915000/15000 [==============================] - 1s 62us/step - loss: 0.1769 - acc: 0.9413 - val_loss: 0.2784 - val_acc: 0.8900
Epoch 5/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.1694 - acc: 0.9 2048/15000 [===>..........................] - ETA: 0s - loss: 0.1491 - acc: 0.9 3584/15000 [======>.......................] - ETA: 0s - loss: 0.1454 - acc: 0.9 5120/15000 [=========>....................] - ETA: 0s - loss: 0.1404 - acc: 0.9 6656/15000 [============>.................] - ETA: 0s - loss: 0.1407 - acc: 0.9 8192/15000 [===============>..............] - ETA: 0s - loss: 0.1404 - acc: 0.9 9728/15000 [==================>...........] - ETA: 0s - loss: 0.1421 - acc: 0.911264/15000 [=====================>........] - ETA: 0s - loss: 0.1400 - acc: 0.912800/15000 [========================>.....] - ETA: 0s - loss: 0.1401 - acc: 0.914336/15000 [===========================>..] - ETA: 0s - loss: 0.1427 - acc: 0.915000/15000 [==============================] - 1s 60us/step - loss: 0.1429 - acc: 0.9542 - val_loss: 0.2769 - val_acc: 0.8894
Epoch 6/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.1140 - acc: 0.9 2048/15000 [===>..........................] - ETA: 0s - loss: 0.1128 - acc: 0.9 3584/15000 [======>.......................] - ETA: 0s - loss: 0.1098 - acc: 0.9 5120/15000 [=========>....................] - ETA: 0s - loss: 0.1139 - acc: 0.9 6656/15000 [============>.................] - ETA: 0s - loss: 0.1152 - acc: 0.9 8192/15000 [===============>..............] - ETA: 0s - loss: 0.1160 - acc: 0.9 9728/15000 [==================>...........] - ETA: 0s - loss: 0.1150 - acc: 0.911264/15000 [=====================>........] - ETA: 0s - loss: 0.1154 - acc: 0.912800/15000 [========================>.....] - ETA: 0s - loss: 0.1167 - acc: 0.914336/15000 [===========================>..] - ETA: 0s - loss: 0.1198 - acc: 0.915000/15000 [==============================] - 1s 62us/step - loss: 0.1204 - acc: 0.9624 - val_loss: 0.2901 - val_acc: 0.8873
Epoch 7/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.1014 - acc: 0.9 2048/15000 [===>..........................] - ETA: 0s - loss: 0.0914 - acc: 0.9 3584/15000 [======>.......................] - ETA: 0s - loss: 0.0904 - acc: 0.9 5120/15000 [=========>....................] - ETA: 0s - loss: 0.0928 - acc: 0.9 6656/15000 [============>.................] - ETA: 0s - loss: 0.0924 - acc: 0.9 8192/15000 [===============>..............] - ETA: 0s - loss: 0.0922 - acc: 0.9 9728/15000 [==================>...........] - ETA: 0s - loss: 0.0981 - acc: 0.911264/15000 [=====================>........] - ETA: 0s - loss: 0.0978 - acc: 0.912800/15000 [========================>.....] - ETA: 0s - loss: 0.0979 - acc: 0.914336/15000 [===========================>..] - ETA: 0s - loss: 0.0986 - acc: 0.915000/15000 [==============================] - 1s 61us/step - loss: 0.0976 - acc: 0.9714 - val_loss: 0.3057 - val_acc: 0.8846
Epoch 8/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.0633 - acc: 0.9 2048/15000 [===>..........................] - ETA: 0s - loss: 0.0676 - acc: 0.9 3584/15000 [======>.......................] - ETA: 0s - loss: 0.0700 - acc: 0.9 5120/15000 [=========>....................] - ETA: 0s - loss: 0.0747 - acc: 0.9 6656/15000 [============>.................] - ETA: 0s - loss: 0.0742 - acc: 0.9 8192/15000 [===============>..............] - ETA: 0s - loss: 0.0789 - acc: 0.9 9728/15000 [==================>...........] - ETA: 0s - loss: 0.0785 - acc: 0.911264/15000 [=====================>........] - ETA: 0s - loss: 0.0786 - acc: 0.912800/15000 [========================>.....] - ETA: 0s - loss: 0.0787 - acc: 0.914336/15000 [===========================>..] - ETA: 0s - loss: 0.0802 - acc: 0.915000/15000 [==============================] - 1s 61us/step - loss: 0.0811 - acc: 0.9777 - val_loss: 0.3367 - val_acc: 0.8823
Epoch 9/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.0877 - acc: 0.9 2048/15000 [===>..........................] - ETA: 0s - loss: 0.0710 - acc: 0.9 3584/15000 [======>.......................] - ETA: 0s - loss: 0.0641 - acc: 0.9 5120/15000 [=========>....................] - ETA: 0s - loss: 0.0635 - acc: 0.9 6656/15000 [============>.................] - ETA: 0s - loss: 0.0613 - acc: 0.9 8192/15000 [===============>..............] - ETA: 0s - loss: 0.0629 - acc: 0.9 9728/15000 [==================>...........] - ETA: 0s - loss: 0.0653 - acc: 0.911264/15000 [=====================>........] - ETA: 0s - loss: 0.0672 - acc: 0.912800/15000 [========================>.....] - ETA: 0s - loss: 0.0674 - acc: 0.914336/15000 [===========================>..] - ETA: 0s - loss: 0.0682 - acc: 0.915000/15000 [==============================] - 1s 60us/step - loss: 0.0681 - acc: 0.9809 - val_loss: 0.3593 - val_acc: 0.8772
Epoch 10/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.0630 - acc: 0.9 2048/15000 [===>..........................] - ETA: 0s - loss: 0.0468 - acc: 0.9 3584/15000 [======>.......................] - ETA: 0s - loss: 0.0513 - acc: 0.9 5120/15000 [=========>....................] - ETA: 0s - loss: 0.0556 - acc: 0.9 6656/15000 [============>.................] - ETA: 0s - loss: 0.0548 - acc: 0.9 8192/15000 [===============>..............] - ETA: 0s - loss: 0.0555 - acc: 0.9 9728/15000 [==================>...........] - ETA: 0s - loss: 0.0554 - acc: 0.911264/15000 [=====================>........] - ETA: 0s - loss: 0.0555 - acc: 0.912800/15000 [========================>.....] - ETA: 0s - loss: 0.0575 - acc: 0.914336/15000 [===========================>..] - ETA: 0s - loss: 0.0584 - acc: 0.915000/15000 [==============================] - 1s 61us/step - loss: 0.0581 - acc: 0.9850 - val_loss: 0.3755 - val_acc: 0.8761
Epoch 11/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.0474 - acc: 0.9 2048/15000 [===>..........................] - ETA: 0s - loss: 0.0390 - acc: 0.9 3584/15000 [======>.......................] - ETA: 0s - loss: 0.0370 - acc: 0.9 5120/15000 [=========>....................] - ETA: 0s - loss: 0.0394 - acc: 0.9 6656/15000 [============>.................] - ETA: 0s - loss: 0.0406 - acc: 0.9 8192/15000 [===============>..............] - ETA: 0s - loss: 0.0397 - acc: 0.9 9728/15000 [==================>...........] - ETA: 0s - loss: 0.0408 - acc: 0.911264/15000 [=====================>........] - ETA: 0s - loss: 0.0445 - acc: 0.912800/15000 [========================>.....] - ETA: 0s - loss: 0.0454 - acc: 0.914336/15000 [===========================>..] - ETA: 0s - loss: 0.0448 - acc: 0.915000/15000 [==============================] - 1s 60us/step - loss: 0.0448 - acc: 0.9894 - val_loss: 0.3992 - val_acc: 0.8777
Epoch 12/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.0336 - acc: 0.9 2048/15000 [===>..........................] - ETA: 0s - loss: 0.0314 - acc: 0.9 3584/15000 [======>.......................] - ETA: 0s - loss: 0.0328 - acc: 0.9 5120/15000 [=========>....................] - ETA: 0s - loss: 0.0330 - acc: 0.9 6656/15000 [============>.................] - ETA: 0s - loss: 0.0325 - acc: 0.9 8192/15000 [===============>..............] - ETA: 0s - loss: 0.0338 - acc: 0.9 9728/15000 [==================>...........] - ETA: 0s - loss: 0.0367 - acc: 0.911264/15000 [=====================>........] - ETA: 0s - loss: 0.0371 - acc: 0.912800/15000 [========================>.....] - ETA: 0s - loss: 0.0371 - acc: 0.914336/15000 [===========================>..] - ETA: 0s - loss: 0.0373 - acc: 0.915000/15000 [==============================] - 1s 61us/step - loss: 0.0378 - acc: 0.9915 - val_loss: 0.4584 - val_acc: 0.8677
Epoch 13/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.0358 - acc: 0.9 2048/15000 [===>..........................] - ETA: 0s - loss: 0.0280 - acc: 0.9 3584/15000 [======>.......................] - ETA: 0s - loss: 0.0266 - acc: 0.9 5120/15000 [=========>....................] - ETA: 0s - loss: 0.0265 - acc: 0.9 6656/15000 [============>.................] - ETA: 0s - loss: 0.0260 - acc: 0.9 8192/15000 [===============>..............] - ETA: 0s - loss: 0.0257 - acc: 0.9 9728/15000 [==================>...........] - ETA: 0s - loss: 0.0261 - acc: 0.911264/15000 [=====================>........] - ETA: 0s - loss: 0.0298 - acc: 0.912800/15000 [========================>.....] - ETA: 0s - loss: 0.0295 - acc: 0.914336/15000 [===========================>..] - ETA: 0s - loss: 0.0296 - acc: 0.915000/15000 [==============================] - 1s 61us/step - loss: 0.0296 - acc: 0.9942 - val_loss: 0.4602 - val_acc: 0.8744
Epoch 14/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.0186 - acc: 0.9 2048/15000 [===>..........................] - ETA: 0s - loss: 0.0189 - acc: 0.9 3584/15000 [======>.......................] - ETA: 0s - loss: 0.0188 - acc: 0.9 5120/15000 [=========>....................] - ETA: 0s - loss: 0.0185 - acc: 0.9 6656/15000 [============>.................] - ETA: 0s - loss: 0.0188 - acc: 0.9 8192/15000 [===============>..............] - ETA: 0s - loss: 0.0193 - acc: 0.9 9728/15000 [==================>...........] - ETA: 0s - loss: 0.0198 - acc: 0.911264/15000 [=====================>........] - ETA: 0s - loss: 0.0214 - acc: 0.912800/15000 [========================>.....] - ETA: 0s - loss: 0.0248 - acc: 0.914336/15000 [===========================>..] - ETA: 0s - loss: 0.0248 - acc: 0.915000/15000 [==============================] - 1s 61us/step - loss: 0.0249 - acc: 0.9954 - val_loss: 0.4959 - val_acc: 0.8729
Epoch 15/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.0154 - acc: 0.9 2048/15000 [===>..........................] - ETA: 0s - loss: 0.0139 - acc: 0.9 3584/15000 [======>.......................] - ETA: 0s - loss: 0.0139 - acc: 0.9 5120/15000 [=========>....................] - ETA: 0s - loss: 0.0139 - acc: 0.9 6656/15000 [============>.................] - ETA: 0s - loss: 0.0153 - acc: 0.9 8192/15000 [===============>..............] - ETA: 0s - loss: 0.0153 - acc: 0.9 9728/15000 [==================>...........] - ETA: 0s - loss: 0.0152 - acc: 0.911264/15000 [=====================>........] - ETA: 0s - loss: 0.0157 - acc: 0.912800/15000 [========================>.....] - ETA: 0s - loss: 0.0169 - acc: 0.914336/15000 [===========================>..] - ETA: 0s - loss: 0.0188 - acc: 0.915000/15000 [==============================] - 1s 61us/step - loss: 0.0191 - acc: 0.9970 - val_loss: 0.5268 - val_acc: 0.8717
Epoch 16/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.0097 - acc: 1.0 2048/15000 [===>..........................] - ETA: 0s - loss: 0.0092 - acc: 1.0 3584/15000 [======>.......................] - ETA: 0s - loss: 0.0109 - acc: 0.9 5120/15000 [=========>....................] - ETA: 0s - loss: 0.0108 - acc: 0.9 6656/15000 [============>.................] - ETA: 0s - loss: 0.0109 - acc: 0.9 8192/15000 [===============>..............] - ETA: 0s - loss: 0.0109 - acc: 0.9 9728/15000 [==================>...........] - ETA: 0s - loss: 0.0112 - acc: 0.911264/15000 [=====================>........] - ETA: 0s - loss: 0.0123 - acc: 0.912800/15000 [========================>.....] - ETA: 0s - loss: 0.0134 - acc: 0.914336/15000 [===========================>..] - ETA: 0s - loss: 0.0146 - acc: 0.915000/15000 [==============================] - 1s 61us/step - loss: 0.0150 - acc: 0.9983 - val_loss: 0.5586 - val_acc: 0.8688
Epoch 17/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.0094 - acc: 1.0 2048/15000 [===>..........................] - ETA: 0s - loss: 0.0091 - acc: 0.9 3584/15000 [======>.......................] - ETA: 0s - loss: 0.0097 - acc: 0.9 5120/15000 [=========>....................] - ETA: 0s - loss: 0.0091 - acc: 0.9 6656/15000 [============>.................] - ETA: 0s - loss: 0.0090 - acc: 0.9 8192/15000 [===============>..............] - ETA: 0s - loss: 0.0088 - acc: 0.9 9728/15000 [==================>...........] - ETA: 0s - loss: 0.0089 - acc: 0.911264/15000 [=====================>........] - ETA: 0s - loss: 0.0091 - acc: 0.912800/15000 [========================>.....] - ETA: 0s - loss: 0.0111 - acc: 0.914336/15000 [===========================>..] - ETA: 0s - loss: 0.0121 - acc: 0.915000/15000 [==============================] - 1s 61us/step - loss: 0.0121 - acc: 0.9985 - val_loss: 0.6028 - val_acc: 0.8654
Epoch 18/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.0057 - acc: 1.0 2048/15000 [===>..........................] - ETA: 0s - loss: 0.0078 - acc: 0.9 3584/15000 [======>.......................] - ETA: 0s - loss: 0.0069 - acc: 0.9 5120/15000 [=========>....................] - ETA: 0s - loss: 0.0070 - acc: 0.9 6656/15000 [============>.................] - ETA: 0s - loss: 0.0071 - acc: 0.9 8192/15000 [===============>..............] - ETA: 0s - loss: 0.0071 - acc: 0.9 9728/15000 [==================>...........] - ETA: 0s - loss: 0.0073 - acc: 0.911264/15000 [=====================>........] - ETA: 0s - loss: 0.0072 - acc: 0.912800/15000 [========================>.....] - ETA: 0s - loss: 0.0074 - acc: 0.914336/15000 [===========================>..] - ETA: 0s - loss: 0.0084 - acc: 0.915000/15000 [==============================] - 1s 61us/step - loss: 0.0092 - acc: 0.9989 - val_loss: 0.6289 - val_acc: 0.8659
Epoch 19/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.0056 - acc: 1.0 2048/15000 [===>..........................] - ETA: 0s - loss: 0.0064 - acc: 0.9 3584/15000 [======>.......................] - ETA: 0s - loss: 0.0060 - acc: 0.9 5120/15000 [=========>....................] - ETA: 0s - loss: 0.0057 - acc: 0.9 6656/15000 [============>.................] - ETA: 0s - loss: 0.0054 - acc: 0.9 8192/15000 [===============>..............] - ETA: 0s - loss: 0.0055 - acc: 0.9 9728/15000 [==================>...........] - ETA: 0s - loss: 0.0054 - acc: 0.911264/15000 [=====================>........] - ETA: 0s - loss: 0.0054 - acc: 0.912800/15000 [========================>.....] - ETA: 0s - loss: 0.0054 - acc: 0.914336/15000 [===========================>..] - ETA: 0s - loss: 0.0054 - acc: 0.915000/15000 [==============================] - 1s 61us/step - loss: 0.0055 - acc: 0.9999 - val_loss: 0.7878 - val_acc: 0.8475
Epoch 20/20
  512/15000 [>.............................] - ETA: 0s - loss: 0.0142 - acc: 0.9 2048/15000 [===>..........................] - ETA: 0s - loss: 0.0088 - acc: 0.9 3584/15000 [======>.......................] - ETA: 0s - loss: 0.0067 - acc: 0.9 5120/15000 [=========>....................] - ETA: 0s - loss: 0.0058 - acc: 0.9 6656/15000 [============>.................] - ETA: 0s - loss: 0.0056 - acc: 0.9 8192/15000 [===============>..............] - ETA: 0s - loss: 0.0053 - acc: 0.9 9728/15000 [==================>...........] - ETA: 0s - loss: 0.0051 - acc: 0.911264/15000 [=====================>........] - ETA: 0s - loss: 0.0050 - acc: 0.912800/15000 [========================>.....] - ETA: 0s - loss: 0.0058 - acc: 0.914336/15000 [===========================>..] - ETA: 0s - loss: 0.0094 - acc: 0.915000/15000 [==============================] - 1s 61us/step - loss: 0.0091 - acc: 0.9979 - val_loss: 0.7043 - val_acc: 0.8644
>>> history_dict= history.history
>>> history_dict.keys()
dict_keys(['val_loss', 'val_acc', 'loss', 'acc'])
>>> import matplotlib.pyplot as plt
>>> loss_values=history_dict['loss']
>>> val_loss_values=history_dict['val_loss']
>>> epochs = range(1,len(loss_values) + 1)
>>> plt.plot(epochs, loss_values, 'bo', label='Training loss')
[<matplotlib.lines.Line2D object at 0x7f94904d43c8>]
>>> plt.plot(epochs, val_loss_values, 'b', label='Validation loss')
[<matplotlib.lines.Line2D object at 0x7f95d8c3ccc0>]
>>> plt.title('Training and Validation loss')
Text(0.5,1,'Training and Validation loss')
>>> plt.xlabel('Epochs')
Text(0.5,0,'Epochs')
>>> plt.ylabel('Loss')
Text(0,0.5,'Loss')
>>> plt.legend()
<matplotlib.legend.Legend object at 0x7f94904d4dd8>
>>> plt.show()

f:id:k-kuro:20200209222440p:plain

>>> acc = history_dict['acc']
>>> val_acc = history_dict['val_acc']
>>> plt.plot(epochs, acc, 'bo', label='Training_acc')
[<matplotlib.lines.Line2D object at 0x7f948c4115c0>]
>>> plt.plot(epochs, val_acc, 'b', label='Validation_acc')
[<matplotlib.lines.Line2D object at 0x7f949264b3c8>]
>>> plt.title('Training and Validation accuracy')
Text(0.5,1,'Training and Validation accuracy')
>>> plt.xlabel('Epochs')
Text(0.5,0,'Epochs')
>>> plt.ylabel('Accuracy')
Text(0,0.5,'Accuracy')
>>> plt.legend()
<matplotlib.legend.Legend object at 0x7f948c411fd0>
>>> plt.show()

f:id:k-kuro:20200209222506p:plain
確かにEpoch4を過ぎると損失率(loss)は上昇し、成功率(Accuracy)も低下する。これが過学習ということらしい。
ということでEpochs=4にしてパラメータをいろいろ弄ったときにどうなるかを見てみる。

>>> model = models.Sequential()
>>> model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))
>>> model.add(layers.Dense(16, activation='relu'))
>>> model.add(layers.Dense(1, activation='sigmoid'))
>>> model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])
>>> model.fit(x_train, y_train, epochs=4, batch_size=512)
Epoch 1/4
  512/25000 [..............................] - ETA: 4s - loss: 0.6888 - accuracy 1536/25000 [>.............................] - ETA: 2s - loss: 0.6763 - accuracy 3072/25000 [==>...........................] - ETA: 1s - loss: 0.6539 - accuracy 4608/25000 [====>.........................] - ETA: 1s - loss: 0.6258 - accuracy 6144/25000 [======>.......................] - ETA: 1s - loss: 0.6000 - accuracy 7680/25000 [========>.....................] - ETA: 0s - loss: 0.5756 - accuracy 9216/25000 [==========>...................] - ETA: 0s - loss: 0.5538 - accuracy10752/25000 [===========>..................] - ETA: 0s - loss: 0.5425 - accuracy12288/25000 [=============>................] - ETA: 0s - loss: 0.5248 - accuracy13824/25000 [===============>..............] - ETA: 0s - loss: 0.5098 - accuracy15360/25000 [=================>............] - ETA: 0s - loss: 0.4961 - accuracy16896/25000 [===================>..........] - ETA: 0s - loss: 0.4854 - accuracy18432/25000 [=====================>........] - ETA: 0s - loss: 0.4739 - accuracy19968/25000 [======================>.......] - ETA: 0s - loss: 0.4653 - accuracy21504/25000 [========================>.....] - ETA: 0s - loss: 0.4570 - accuracy23040/25000 [==========================>...] - ETA: 0s - loss: 0.4488 - accuracy24576/25000 [============================>.] - ETA: 0s - loss: 0.4413 - accuracy25000/25000 [==============================] - 1s 47us/step - loss: 0.4394 - accuracy: 0.8197
Epoch 2/4
  512/25000 [..............................] - ETA: 1s - loss: 0.2519 - accuracy 2048/25000 [=>............................] - ETA: 1s - loss: 0.2655 - accuracy 3584/25000 [===>..........................] - ETA: 0s - loss: 0.2662 - accuracy 5120/25000 [=====>........................] - ETA: 0s - loss: 0.2647 - accuracy 6656/25000 [======>.......................] - ETA: 0s - loss: 0.2632 - accuracy 8192/25000 [========>.....................] - ETA: 0s - loss: 0.2604 - accuracy 9728/25000 [==========>...................] - ETA: 0s - loss: 0.2571 - accuracy11264/25000 [============>.................] - ETA: 0s - loss: 0.2628 - accuracy12800/25000 [==============>...............] - ETA: 0s - loss: 0.2600 - accuracy14336/25000 [================>.............] - ETA: 0s - loss: 0.2579 - accuracy15872/25000 [==================>...........] - ETA: 0s - loss: 0.2569 - accuracy17408/25000 [===================>..........] - ETA: 0s - loss: 0.2574 - accuracy18944/25000 [=====================>........] - ETA: 0s - loss: 0.2550 - accuracy20480/25000 [=======================>......] - ETA: 0s - loss: 0.2549 - accuracy22016/25000 [=========================>....] - ETA: 0s - loss: 0.2543 - accuracy23552/25000 [===========================>..] - ETA: 0s - loss: 0.2532 - accuracy25000/25000 [==============================] - 1s 43us/step - loss: 0.2525 - accuracy: 0.9104
Epoch 3/4
  512/25000 [..............................] - ETA: 1s - loss: 0.2105 - accuracy 2048/25000 [=>............................] - ETA: 0s - loss: 0.1823 - accuracy 3072/25000 [==>...........................] - ETA: 1s - loss: 0.1813 - accuracy 4608/25000 [====>.........................] - ETA: 0s - loss: 0.1817 - accuracy 6144/25000 [======>.......................] - ETA: 0s - loss: 0.1990 - accuracy 7680/25000 [========>.....................] - ETA: 0s - loss: 0.1989 - accuracy 9216/25000 [==========>...................] - ETA: 0s - loss: 0.1956 - accuracy10752/25000 [===========>..................] - ETA: 0s - loss: 0.1985 - accuracy12288/25000 [=============>................] - ETA: 0s - loss: 0.1991 - accuracy13824/25000 [===============>..............] - ETA: 0s - loss: 0.1961 - accuracy15360/25000 [=================>............] - ETA: 0s - loss: 0.1961 - accuracy16896/25000 [===================>..........] - ETA: 0s - loss: 0.1955 - accuracy18432/25000 [=====================>........] - ETA: 0s - loss: 0.1950 - accuracy19968/25000 [======================>.......] - ETA: 0s - loss: 0.1942 - accuracy21504/25000 [========================>.....] - ETA: 0s - loss: 0.1946 - accuracy23040/25000 [==========================>...] - ETA: 0s - loss: 0.1953 - accuracy24576/25000 [============================>.] - ETA: 0s - loss: 0.1961 - accuracy25000/25000 [==============================] - 1s 44us/step - loss: 0.1960 - accuracy: 0.9292
Epoch 4/4
  512/25000 [..............................] - ETA: 1s - loss: 0.1593 - accuracy 2048/25000 [=>............................] - ETA: 0s - loss: 0.1431 - accuracy 3584/25000 [===>..........................] - ETA: 0s - loss: 0.1434 - accuracy 5120/25000 [=====>........................] - ETA: 0s - loss: 0.1454 - accuracy 6656/25000 [======>.......................] - ETA: 0s - loss: 0.1475 - accuracy 8192/25000 [========>.....................] - ETA: 0s - loss: 0.1570 - accuracy 9728/25000 [==========>...................] - ETA: 0s - loss: 0.1575 - accuracy11264/25000 [============>.................] - ETA: 0s - loss: 0.1587 - accuracy12800/25000 [==============>...............] - ETA: 0s - loss: 0.1563 - accuracy14336/25000 [================>.............] - ETA: 0s - loss: 0.1597 - accuracy15872/25000 [==================>...........] - ETA: 0s - loss: 0.1609 - accuracy17408/25000 [===================>..........] - ETA: 0s - loss: 0.1620 - accuracy18944/25000 [=====================>........] - ETA: 0s - loss: 0.1635 - accuracy20480/25000 [=======================>......] - ETA: 0s - loss: 0.1636 - accuracy22016/25000 [=========================>....] - ETA: 0s - loss: 0.1634 - accuracy23552/25000 [===========================>..] - ETA: 0s - loss: 0.1639 - accuracy25000/25000 [==============================] - 1s 42us/step - loss: 0.1636 - accuracy: 0.9417
<keras.callbacks.callbacks.History object at 0x7f948c3d0b00>
>>> results = model.evaluate(x_test, y_test)
25000/25000 [==============================] - 1s 52us/step
>>> results
[0.29960916100025176, 0.8816400170326233]
>>> model.predict(x_test)
array([[0.18844399],
       [0.9997899 ],
       [0.8822979 ],
       ...,
       [0.14371884],
       [0.08577126],
       [0.62207   ]], dtype=float32)
>>> model = models.Sequential()
>>> model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))
>>> model.add(layers.Dense(1, activation='sigmoid'))
>>> model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])
>>> model.fit(x_train, y_train, epochs=4, batch_size=512)
Epoch 1/4
・・・・
<keras.callbacks.callbacks.History object at 0x7f9491b66160>
>>> results = model.evaluate(x_test, y_test)
25000/25000 [==============================] - 1s 42us/step
>>> results
[0.30649682688713076, 0.880840003490448]
>>> model = models.Sequential()
>>> model.add(layers.Dense(32, activation='relu', input_shape=(10000,)))
>>> model.add(layers.Dense(32, activation='relu'))
>>> model.add(layers.Dense(1, activation='sigmoid'))
>>> model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])
>>> model.fit(x_train, y_train, epochs=4, batch_size=512)
Epoch 1/4
・・・・・
<keras.callbacks.callbacks.History object at 0x7f948bf58978>
>>> results = model.evaluate(x_test, y_test)
25000/25000 [==============================] - 1s 48us/step
>>> results
[0.31329001052379607, 0.8779199719429016]
>>> model = models.Sequential()
>>> model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))
>>> model.add(layers.Dense(16, activation='relu'))
>>> model.add(layers.Dense(1, activation='sigmoid'))
>>> model.compile(optimizer='rmsprop', loss='mse', metrics=['accuracy'])
>>> model.fit(x_train, y_train, epochs=4, batch_size=512)
Epoch 1/4
・・・・・
<keras.callbacks.callbacks.History object at 0x7f948bdcbb38>
>>> results = model.evaluate(x_test, y_test)
25000/25000 [==============================] - 1s 41us/step
>>> results
[0.08861505253911019, 0.8802400231361389]

clusteringだけmatplotlibで

ヒートマップの描画ははじめてseabornを使ってみたのだが、噂通り細かい調節は難しい。
今回系統樹の枝の長さを入れたかったのだが、結局方法にたどり着けなかったので、系統樹の部分だけmatplotlibで描いてみる。
f:id:k-kuro:20200114145420p:plain
枝が自動で色がついていたり、まだ改良の余地はあるかな。

@app.route('/cluster', methods=['GET', 'POST'])
def cluster_index():
    user = g.user.name
    form=StatForm()
    dir = "./user/" + user + "/cluster"
    if os.path.exists(dir):
        shutil.rmtree(dir)
    if not os.path.exists(dir):
        os.makedirs(dir)
    input = dir + "/input.txt"
    output = dir + "/results.txt"

    if request.method == 'POST':
        if 'file' not in request.files:
            flash('No file part')
            return redirect(request.url)
        file = request.files['file']
        if file.filename == '':
            flash('No data input')
            return redirect(request.url)
        if file and allowed_file(file.filename):
            file.save(input)

    if os.path.exists(input):
        if form.fig_x.data:
        	fig_x=form.fig_x.data
        if form.fig_y.data:
        	fig_y=form.fig_y.data
        if form.fig_dpi.data:
        	fig_dpi=form.fig_dpi.data
        if form.c_method.data:
        	c_method=form.c_method.data
        if form.c_dist.data:
        	c_dist=form.c_dist.data
        if form.font_size.data:
        	f_size=form.font_size.data

        df = pd.read_csv(input, index_col=0)
        df_t = df.T
        header = df.columns
        from scipy.spatial.distance import pdist
        linkage_result = linkage(df_t, method=c_method, metric=c_dist)
        plt.figure(num=None, figsize=(fig_x,fig_y), dpi=fig_dpi, facecolor='w', edgecolor='k')
        dendrogram(linkage_result, leaf_font_size=f_size, labels=header)
        cluster_fig = dir + "/cluster_" + str(time.time()) + ".png"
        plt.savefig(cluster_fig)
        img_url = "../../static/" + cluster_fig

        return render_template('/tools/cluster.html', form=form, img_url=img_url)

    return render_template('/tools/cluster.html', form=form)