python - out of memory error when reading csv file in chunk -
i processing csv-file 2.5 gb big. 2.5 gb table looks this:
columns=[ka,kb_1,kb_2,timeofevent,timeinterval] 0:'3m' '2345' '2345' '2014-10-5',3000 1:'3m' '2958' '2152' '2015-3-22',5000 2:'ge' '2183' '2183' '2012-12-31',515 3:'3m' '2958' '2958' '2015-3-10',395 4:'ge' '2183' '2285' '2015-4-19',1925 5:'ge' '2598' '2598' '2015-3-17',1915 and want groupby ka , kb_1 result this:
columns=[ka,kb,errornum,errorrate,totalnum of records] '3m','2345',0,0%,1 '3m','2958',1,50%,2 'ge','2183',1,50%,2 'ge','2598',0,0%,1 (definition of error record: when kb_1 != kb_2, corresponding record treated abnormal record )
my computer, ubuntu 12.04, has 16 gb memory , free -m returns
total used free shared buffers cached mem: 112809 14476 98333 0 128 10823 -/+ buffers/cache: 3524 109285 swap: 0 0 0 my python file called bigdata.py
import pandas pd import numpy np import sys,traceback,os cksize=98333 # or 1024, either chunk size didn't work @ try: dfs = pd.dataframe() reader=pd.read_table('data/petajoined.csv', chunksize=cksize) chunk in reader:#when executed line,error occur! pass #temp=tb_createtopranktable(chunk) #dfs.append(temp) #df=tb_createtopranktable(dfs) except: traceback.print_exc(file=sys.stdout) ipdb> pd.__version__ '0.16.0' i use following command monitor memory usage:
top ps -c python -o %cpu,%mem,cmd since takes 2 seconds crash, can see mem usage had reached 90% time, , cpu usage reached 100%
when excecute python bigdata.py, following error generate:
/usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: userwarning: module dateutil imported /usr/local/lib/python2.7/dist-packages/dateutil/__init__.pyc, /usr/lib/python2.7/dist-packages being added sys.path pkg_resources import resource_stream /usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: userwarning: module pytz imported /usr/local/lib/python2.7/dist-packages/pytz/__init__.pyc, /usr/lib/python2.7/dist-packages being added sys.path pkg_resources import resource_stream traceback (most recent call last): file "bigdata.py", line 10, in <module> chunk in reader: file "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 691, in __iter__ yield self.read(self.chunksize) file "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 715, in read ret = self._engine.read(nrows) file "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 1164, in read data = self._reader.read(nrows) file "pandas/parser.pyx", line 758, in pandas.parser.textreader.read (pandas/parser.c:7411) file "pandas/parser.pyx", line 792, in pandas.parser.textreader._read_low_memory (pandas/parser.c:7819) file "pandas/parser.pyx", line 833, in pandas.parser.textreader._read_rows (pandas/parser.c:8268) file "pandas/parser.pyx", line 820, in pandas.parser.textreader._tokenize_rows (pandas/parser.c:8142) file "pandas/parser.pyx", line 1758, in pandas.parser.raise_parser_error (pandas/parser.c:20728) cparsererror: error tokenizing data. c error: out of memory segmentation fault (core dumped) or
/usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: userwarning: module dateutil imported /usr/local/lib/python2.7/dist-packages/dateutil/__init__.pyc, /usr/lib/python2.7/dist-packages being added sys.path pkg_resources import resource_stream /usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: userwarning: module pytz imported /usr/local/lib/python2.7/dist-packages/pytz/__init__.pyc, /usr/lib/python2.7/dist-packages being added sys.path pkg_resources import resource_stream traceback (most recent call last): file "bigdata.py", line 10, in <module> chunk in reader: file "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 691, in __iter__ yield self.read(self.chunksize) file "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 715, in read ret = self._engine.read(nrows) file "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 1164, in read data = self._reader.read(nrows) file "pandas/parser.pyx", line 758, in pandas.parser.textreader.read (pandas/parser.c:7411) file "pandas/parser.pyx", line 792, in pandas.parser.textreader._read_low_memory (pandas/parser.c:7819) file "pandas/parser.pyx", line 833, in pandas.parser.textreader._read_rows (pandas/parser.c:8268) file "pandas/parser.pyx", line 820, in pandas.parser.textreader._tokenize_rows (pandas/parser.c:8142) file "pandas/parser.pyx", line 1758, in pandas.parser.raise_parser_error (pandas/parser.c:20728) cparsererror: error tokenizing data. c error: out of memory *** glibc detected *** python: free(): invalid pointer: 0x00007f750d2a4c0e *** ====== backtrace: ======== /lib/x86_64-linux-gnu/libc.so.6(+0x7db26)[0x7f7511529b26] /usr/local/lib/python2.7/dist-packages/pandas/parser.so(+0x4d5a1)[0x7f750d29d5a1] /usr/local/lib/python2.7/dist-packages/pandas/parser.so(parser_cleanup+0x15)[0x7f750d29de45] /usr/local/lib/python2.7/dist-packages/pandas/parser.so(parser_free+0x9)[0x7f750d29e039] /usr/local/lib/python2.7/dist-packages/pandas/parser.so(+0xb43e)[0x7f750d25b43e] .... python(pydict_setitem+0x49)[0x577749] python(_pymodule_clear+0x149)[0x4cafb9] python(pyimport_cleanup+0x477)[0x4cb4f7] python(py_finalize+0x18e)[0x549f0e] python(py_main+0x3bc)[0x56b56c] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f75114cd76d] python[0x41bb11] ======= memory map: ======== 00400000-00670000 r-xp 00000000 08:01 26612 /usr/bin/python2.7 0086f000-00870000 r--p 0026f000 08:01 26612 /usr/b....... 008d9000-008eb000 rw-p 00000000 00:00 0 01ddb000-036f7000 rw-p 00000000 00:00 0 [heap] 7f748c179000-7f74cc17a000 rw-p 00000000 00:00 0 7f7504000000-7f7504021000 rw-p 00000000 00:00 0 7f7504021000-7f7508000000 ---p 00000000 00:00 0 7f750bf83000-7f750c285000 rw-p 00000000 00:00 0 7f750c285000-7f750c586000 rw-p 00000000 00:00 0 7f750c586000-7f750c707000 rw-p 00000000 00:00 0 7f750c707000-7f750c711000 r-xp 00000000 08:01 533205 /usr/local/lib/python2.7/dist-packages/pandas/_testing.so 7f750c711000-7f750c911000 ---p 0000a000 08:01 533205 /usr/local/lib/python2.7/dist-packages/pandas/_testing.so 7f750c911000-7f750c912000 r--p 0000a000 08:01 533205 /usr/local/lib/python2.7/dist-packages/pandas/_testing.so 7f750c912000-7f750c913000 rw-p 0000b000 08:01 533205 /usr/local/lib/python2.7/dist-packages/pandas/_testing.so 7f750c913000-7f750c914000 rw-p 00000000 00:00 0 7f750c914000-7f750c918000 r-xp 00000000 08:01 2331 /lib/x86_64-linux-gnu/libuuid.so.1.3.0 7f750c918000-7f750cb17000 ---p 00004000 08:01 2331 /lib/x86_64-linux-gnu/libuuid.so.1.3.0 7f750cb17000-7f750cb18000 r--p 00003000 08:01 2331 /lib/x86_64-linux-gnu/libuuid.so.1.3.0 7f750cb18000-7f750cb19000 rw-p 00004000 08:01 2331 /lib/x86_64-linux-gnu/libuuid.so.1.3.0 7f750cb19000-7f750cb34000 r-xp 00000000 08:01 533071 /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so 7f750cb34000-7f750cd33000 ---p 0001b000 08:01 533071 /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so 7f750cd33000-7f750cd34000 r--p 0001a000 08:01 533071 /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so 7f750cd34000-7f750cd38000 rw-p 0001b000 08:01 533071 /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so 7f750cd38000-7f750d039000 rw-p 00000000 00:00 0 7f750d039000-7f750d04e000 r-xp 00000000 08:01 533070 /usr/local/lib/python2.7/dist-packages/pandas/json.so 7f750d04e000-7f750d24e000 ---p 00015000 08:01 533070 /usr/local/lib/python2.7/dist-packages/pandas/json.so 7f750d24e000-7f750d24f000 r--p 00015000 08:01 533070 /usr/local/lib/python2.7/dist-packages/pandas/json.so 7f750d24f000-7f750d250000 rw-p 00016000 08:01 533070 /usr/local/lib/python2.7/dist-packages/pandas/json.so 7f750d250000-7f750d2a9000 r-xp 00000000 08:01 533270 /usr/local/lib/python2.7/dist-packages/pandas/parser.so 7f750d2a9000-7f750d4a8000 ---p 00059000 08:01 533270 /usr/local/lib/python2.7/dist-packages/pandas/parser.so 7f750d4a8000-7f750d4a9000 r--p 00058000 08:01 533270 /usr/local/lib/python2.7/dist-packages/pandas/parser.so 7f750d4a9000-7f750d4af000 rw-p 00059000 08:01 533270 /usr/local/lib/python2.7/dist-packages/pandas/parser.so 7f750d4af000-7f750d591000 r-xp 00000000 08:01 49584 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16 7f750d591000-7f750d790000 ---p 000e2000 08:01 49584 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16 7f750d790000-7f750d798000 r--p 000e1000 08:01 49584 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16 7f750d798000-7f750d79a000 rw-p 000e9000 08:01 49584 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16 7f750d79a000-7f750d7af000 rw-p 00000000 00:00 0 7f750d7af000-7f750d7f1000 r-xp 00000000 08:01 530477 /usr/lib/pyshared/python2.7/matplotlib/_path.so 7f750d7f1000-7f750d9f1000 ---p 00042000 08:01 530477 /usr/lib/pyshared/python2.7/matplotlib/_path.so 7f750d9f1000-7f750d9f3000 r--p 00042000 08:01 530477 /usr/lib/pyshared/python2.7/matplotlib/_path.so 7f750d9f3000-7f750d9f4000 rw-p 00044000 08:01 530477 /usr/lib/pyshared/python2.7/matplotlib/_path.so 7f750d9f4000-7f750da2d000 r-xp 00000000 08:01 533269 /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so 7f750da2d000-7f750dc2c000 ---p 00039000 08:01 533269 /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so 7f750dc2c000-7f750dc2d000 r--p 00038000 08:01 533269 /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so 7f750dc2d000-7f750dc31000 rw-p 00039000 08:01 533269 /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so 7f750dc31000-7f750dc7d000 r-xp 00000000 08:01 533447 /usr/local/lib/python2.7/dist-packages/pandas/_period.so 7f750dc7d000-7f750de7c000 ---p 0004c000 08:01 533447 /usr/local/lib/python2.7/dist-packages/pandas/_period.so 7f750de7c000-7f750de7d000 r--p 0004b000 08:01 533447 /usr/local/lib/python2.7/dist-packages/pandas/_period.so 7f750de7d000-7f750de86000 rw-p 0004c000 08:01 533447 /usr/local/lib/python2.7/dist-packages/pandas/_period.so 7f750de86000-7f750de87000 rw-p 00000000 00:00 0 7f750de87000-7f750debc000 r-xp 00000000 08:01 533203 /usr/local/lib/python2.7/dist-packages/pandas/index.so 7f750debc000-7f750e0bb000 ---p 00035000 08:01 533203 /usr/local/lib/python2.7/dist-packages/pandas/index.so 7f750e0bb000-7f750e0bc000 r--p 00034000 08:01 533203 /usr/local/lib/python2.7/dist-packages/pandas/index.so 7f750e0bc000-7f750e0c0000 rw-p 00035000 08:01 533203 /usr/local/lib/python2.7/dist-packages/pandas/index.so 7f750e0c0000-7f750e295000 r-xp 00000000 08:01 533278 /usr/local/lib/python2.7/dist-packages/pandas/algos.so 7f750e295000-7f750e494000 ---p 001d5000 08:01 533278 /usr/local/lib/python2.7/dist-packages/pandas/algos.so 7f750e494000-7f750e495000 r--p 001d4000 08:01 533278 /usr/local/lib/python2.7/dist-packages/pandas/algos.so 7f750e495000-7f750e4a9000 rw-p 001d5000 08:01 533278 /usr/local/lib/python2.7/dist-packages/pandas/algos.so 7f750e4a9000-7f750e4ac000 rw-p 00000000 00:00 0 7f750e4ac000-7f750e4b2000 r-xp 00000000 08:01 48831 /usr/lib/python2.7/lib-dynload/_csv.so 7f750e4b2000-7f750e6b1000 ---p 00006000 08:01 48831 /usr/lib/python2.7/lib-dynload/_csv.so 7f750e6b1000-7f750e6b2000 r--p 00005000 08:01 48831 /usr/lib/python2.7/lib-dynload/_csv.so 7f750e6b2000-7f750e6b4000 rw-p 00006000 08:01 48831 /usr/lib/python2.7/lib-dynload/_csv.so 7f750e6b4000-7f750e782000 r-xp 00000000 08:01 533449 /usr/local/lib/python2.7/dist-packages/pandas/lib.so 7f750e782000-7f750e981000 ---p 000ce000 08:01 533449 /usr/local/lib/python2.7/dist-packages/pandas/lib.so 7f750e981000-7f750e982000 r--p 000cd000 08:01 533449 /usr/local/lib/python2.7/dist-packages/pandas/lib.so 7f750e982000-7f750e990000 rw-p 000ce000 08:01 533449 /usr/local/lib/python2.7/dist-packages/pandas/lib.so 7f750e990000-7f750e992000 rw-p 00000000 00:00 0 7f750e992000-7f750ea8f000 r-xp 00000000 08:01 533271 /usr/local/lib/python2.7/dist-packages/pandas/tslib.so 7f750ea8f000-7f750ec8e000 ---p 000fd000 08:01 533271 /usr/local/lib/python2.7/dist-packages/pandas/tslib.so 7f750ec8e000-7f750ec8f000 r--p 000fc000 08:01 533271 /usr/local/lib/python2.7/dist-packages/pandas/tslib.so 7f750ec8f000-7f750eca1000 rw-p 000fd000 08:01 533271 /usr/local/lib/python2.7/dist-packages/pandas/tslib.so 7f750eca1000-7f750eca4000 rw-p 00000000 00:00 0 7f750eca4000-7f750ecc5000 r-xp 00000000 08:01 48837 /usr/lib/python2.7/lib-dynload/_ctypes.so 7f750ecc5000-7f750eec4000 ---p 00021000 08:01 48837 /usr/lib/python2.7/lib-dynload/_ctypes.so 7f750eec4000-7f750eec5000 r--p 00020000 08:01 48837 /usr/lib/python2.7/lib-dynload/_ctypes.so 7f750eec5000-7f750eec9000 rw-p 00021000 08:01 48837 /usr/lib/python2.7/lib-dynload/_ctypes.so 7f750eec9000-7f750eeca000 rw-p 00000000 00:00 0 7f750eeca000-7f750ef24000 r-xp 00000000 08:01 532046 /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so 7f750ef24000-7f750f123000 ---p 0005a000 08:01 532046 /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so 7f750f123000-7f750f124000 r--p 00059000 08:01 532046 /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so 7f750f124000-7f750f15c000 rw-p 0005a000 08:01 532046 /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so 7f750f15c000-7f750f15d000 rw-p 00000000 00:00 0 7f750f15d000-7f750f166000 r-xp 00000000 08:01 532085 /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so 7f750f166000-7f750f365000 ---p 00009000 08:01 532085 /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so 7f750f365000-7f750f366000 r--p 00008000 08:01 532085 /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so 7f750f366000-7f750f367000 rw-p 00009000 08:01 532085 /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so 7f750f367000-7f750f368000 r-xp 00000000 08:01 48818 /usr/lib/python2.7/lib-dynload/future_builtins.so 7f750f368000-7f750f567000 ---p 00001000 08:01 48818 /usr/lib/python2.7/lib-dynload/future_builtins.so 7f750f567000-7f750f568000 r--p 00000000 08:01 48818 /usr/lib/python2.7/lib-dynload/future_builtins.so 7f750f568000-7f750f569000 rw-p 00001000 08:01 48818 /usr/lib/python2.7/lib-dynload/future_builtins.so 7f750f569000-7f750f588000 r-xp 00000000 08:01 48815 /usr/lib/python2.7/lib-dynload/_io.so 7f750f588000-7f750f787000 ---p 0001f000 08:01 48815 /usr/lib/python2.7/lib-dynload/_io.so 7f750f787000-7f750f788000 r--p 0001e000 08:01 48815 /usr/lib/python2.7/lib-dynload/_io.so 7f750f788000-7f750f791000 rw-p 0001f000 08:01 48815 /usr/lib/python2.7/lib-dynload/_io.so 7f750f791000-7f750f907000 r-xp 00000000 08:01 532132 /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so 7f750f907000-7f750fb06000 ---p 00176000 08:01 532132 /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so 7f750fb06000-7f750fb07000 r--p 00175000 08:01 532132 /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so 7f750fb07000-7f750fb08000 rw-p 00176000 08:01 532132 /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so 7f750fb08000-7f750fba4000 rw-p 00000000 00:00 0 7f750fba4000-7f750fd01000 r-xp 00000000 08:01 532128 /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so 7f750fd01000-7f750ff00000 ---p 0015d000 08:01 532128 /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so 7f750ff00000-7f750ff01000 r--p 0015c000 08:01 532128 /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so 7f750ff01000-7f750ff02000 rw-p 0015d000 08:01 532128 /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so 7f750ff02000-7f750ff9d000 rw-p 00000000 00:00 0 7f750ff9d000-7f750ffa3000 r-xp 00000000 08:01 532067 /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so 7f750ffa3000-7f75101a2000 ---p 00006000 08:01 532067 /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so 7f75101a2000-7f75101a3000 r--p 00005000 08:01 532067 /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so 7f75101a3000-7f75101a4000 rw-p 00006000 08:01 532067 /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so 7f7510265000-7f751028f000 r-xp 00000000 08:01 532108 /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so 7f751028f000-7f751048e000 ---p 0002a000 08:01 532108 /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so 7f751048e000-7f751048f000 r--p 00029000 08:01 532108 /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so 7f751048f000-7f7510491000 rw-p 0002a000 08:01 532108 /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so 7f7510491000-7f75104d2000 rw-p 00000000 00:00 0 7f75104d2000-7f75104d5000 r-xp 00000000 08:01 48833 /usr/lib/python2.7/lib-dynload/_heapq.so 7f75104d5000-7f75106d4000 ---p 00003000 08:01 48833 /usr/lib/python2.7/lib-dynload/_heapq.so 7f75106d4000-7f75106d5000 r--p 00002000 08:01 48833 /usr/lib/python2.7/lib-dynload/_heapq.so 7f75106d5000-7f75106d7000 rw-p 00003000 08:01 48833 /usr/lib/python2.7/lib-dynload/_heapq.so 7f75106d7000-7f751073e000 r-xp 00000000 08:01 532118 /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so 7f751073e000-7f751093d000 ---p 00067000 08:01 532118 /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so 7f751093d000-7f751093e000 r--p 00066000 08:01 532118 /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so 7f751093e000-7f7510942000 rw-p 00067000 08:01 532118 /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so 7f7510942000-7f7510944000 rw-p 00000000 00:00 0 7f7510944000-7f7510958000 r-xp 00000000 08:01 48804 /usr/lib/python2.7/lib-dynload/datetime.so 7f7510958000-7f7510b57000 ---p 00014000 08:01 48804 /usr/lib/python2.7/lib-dynload/datetime.so 7f7510b57000-7f7510b58000 r--p 00013000 08:01 48804 /usr/lib/python2.7/lib-dynload/datetime.so 7f7510b58000-7f7510b5c000 rw-p 00014000 08:01 48804 /usr/lib/python2.7/lib-dynload/datetime.so 7f7510b5c000-7f7510caf000 r-xp 00000000 08:01 532106 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so 7f7510caf000-7f7510eae000 ---p 00153000 08:01 532106 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so 7f7510eae000-7f7510eb0000 r--p 00152000 08:01 532106 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so 7f7510eb0000-7f7510ebd000 rw-p 00154000 08:01 532106 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so 7f7510ebd000-7f7510ecf000 rw-p 00000000 00:00 0 7f7510ecf000-7f7510f08000 r-xp 00000000 08:01 533450 /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so 7f7510f08000-7f7511107000 ---p 00039000 08:01 533450 /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so 7f7511107000-7f7511108000 r--p 00038000 08:01 533450 /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so 7f7511108000-7f751110c000 rw-p 00039000 08:01 533450 /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so 7f751110c000-7f751110d000 rw-p 00000000 00:00 0 7f751110d000-7f7511296000 r--p 00000000 08:01 58562 /usr/lib/locale/locale-archive 7f7511296000-7f75112ab000 r-xp 00000000 08:01 2312 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f75112ab000-7f75114aa000 ---p 00015000 08:01 2312 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f75114aa000-7f75114ab000 r--p 00014000 08:01 2312 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f75114ab000-7f75114ac000 rw-p 00015000 08:01 2312 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f75114ac000-7f7511660000 r-xp 00000000 08:01 2327 /lib/x86_64-linux-gnu/libc-2.15.so 7f7511660000-7f751185f000 ---p 001b4000 08:01 2327 /lib/x86_64-linux-gnu/libc-2.15.so 7f751185f000-7f7511863000 r--p 001b3000 08:01 2327 /lib/x86_64-linux-gnu/libc-2.15.so 7f7511863000-7f7511865000 rw-p 001b7000 08:01 2327 /lib/x86_64-linux-gnu/libc-2.15.so 7f7511865000-7f751186a000 rw-p 00000000 00:00 0 7f751186a000-7f7511965000 r-xp 00000000 08:01 2400 /lib/x86_64-linux-gnu/libm-2.15.so 7f7511965000-7f7511b64000 ---p 000fb000 08:01 2400 /lib/x86_64-linux-gnu/libm-2.15.so 7f7511b64000-7f7511b65000 r--p 000fa000 08:01 2400 /lib/x86_64-linux-gnu/libm-2.15.so 7f7511b65000-7f7511b66000 rw-p 000fb000 08:01 2400 /lib/x86_64-linux-gnu/libm-2.15.so 7f7511b66000-7f7511b7c000 r-xp 00000000 08:01 2288 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f7511b7c000-7f7511d7b000 ---p 00016000 08:01 2288 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f7511d7b000-7f7511d7c000 r--p 00015000 08:01 2288 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f7511d7c000-7f7511d7d000 rw-p 00016000 08:01 2288 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f7511d7d000-7f7511f2f000 r-xp 00000000 08:01 2279 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f7511f2f000-7f751212e000 ---p 001b2000 08:01 2279 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f751212e000-7f7512149000 r--p 001b1000 08:01 2279 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f7512149000-7f7512154000 rw-p 001cc000 08:01 2279 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f7512154000-7f7512158000 rw-p 00000000 00:00 0 7f7512158000-7f75121ac000 r-xp 00000000 08:01 2393 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f75121ac000-7f75123ac000 ---p 00054000 08:01 2393 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f75123ac000-7f75123af000 r--p 00054000 08:01 2393 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f75123af000-7f75123b6000 rw-p 00057000 08:01 2393 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f75123b6000-7f75123b8000 r-xp 00000000 08:01 2283 /lib/x86_64-linux-gnu/libutil-2.15.so 7f75123b8000-7f75125b7000 ---p 00002000 08:01 2283 /lib/x86_64-linux-gnu/libutil-2.15.so 7f75125b7000-7f75125b8000 r--p 00001000 08:01 2283 /lib/x86_64-linux-gnu/libutil-2.15.so 7f75125b8000-7f75125b9000 rw-p 00002000 08:01 2283 /lib/x86_64-linux-gnu/libutil-2.15.so 7f75125b9000-7f75125bb000 r-xp 00000000 08:01 2406 /lib/x86_64-linux-gnu/ld-2.15.so 7f7512a2d000-7f7512b31000 rw-p 00000000 00:00 0 7f7512b62000-7f7512bea000 rw-p 00000000 00:00 0 7f7512bf7000-7f7512bf9000 rw-p 00000000 00:00 0 7f7512bf9000-7f7512bfa000 rwxp 00000000 00:00 0 7f7512bfa000-7f7512bfc000 rw-p 00000000 00:00 0 7f7512bfc000-7f7512bfd000 r--p 00022000 08:01 2260 /lib/x86_64-linux-gnu/ld-2.15.so 7f7512bfd000-7f7512bff000 rw-p 00023000 08:01 2260 /lib/x86_64-linux-gnu/ld-2.15.so 7ffcf454c000-7ffcf4585000 rw-p 00000000 00:00 0 [stack] 7ffcf459b000-7ffcf459d000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] aborted (core dumped) with below code, there no memory problem, can below code , mean doing group , data aggregation
with open("data/petajoined.csv", "r") content: line in content: #print line pass #do stuff line` content.close() anyone knows happening?
actually want reach result shown in pandas read csv out of memory
maybe there solution?
note use read csv chunk, still there memory error
then, changed chunk size have bigdata.py file in way
import pandas pd import numpy np import sys, traceback, os import etl2 # self processing flow reload(etl2) def iter_chunks(n,df): while true: try: yield df.get_chunk(n) except stopiteration: break cksize=5 try: dfs = pd.dataframe() reader=pd.read_table( 'data/petajoined.csv', chunksize = cksize, low_memory = false, iterator = true ) # choose appropriate chunk in iter_chunks(cksize,reader): temp=etl2.tb_createtopranktable(chunk) dfs.append(temp) df=tb_createtopranktable(dfs) # # chunk in reader: # pass # temp=tb_createtopranktable(chunk) # dfs.append(temp) # df=tb_createtopranktable(dfs) except: traceback.print_exc(file=sys.stdout) still, there segmentation error after running sometime
def tb_createtopranktable(df): try: key='name1' key2='name2' df2 = df.groupby([key,key2])['iserror'].agg({ 'errornum': 'sum','totalparcel': 'count' }) df2['errorrate'] = df2['errornum'] / df2['totalparcel'] return df2
based on snippet, when reading line-by-line.
i assume kb_2 error indicator,
groups={} open("data/petajoined.csv", "r") large_file: line in large_file: arr=line.split('\t') #assuming structure: ka,kb_1,kb_2,timeofevent,timeinterval k=arr[0]+','+arr[1] if not (k in groups.keys()) groups[k]={'record_count':0, 'error_sum': 0} groups[k]['record_count']=groups[k]['record_count']+1 groups[k]['error_sum']=groups[k]['error_sum']+float(arr[2]) k,v in groups.items: print ('{group}: {error_rate}'.format(group=k,error_rate=v['error_sum']/v['record_count'])) this code snippet stores groups in dictionary, , calculates error rate after reading entire file.
it encounter out-of-memory exception, if there many combinations of groups.
Comments
Post a Comment