How to read complex data in python?

I’m trying to read the data which is not structured well. It looks something like this

Generated by trjconv : P/L=1/400 t=   0.00000
11214
    1P1     aP1    1  80.48  35.36   4.25
    2P1     aP1    2  37.45   3.92   3.96
    3P2     aP2    3  18.53  -9.69   4.68
    4P2     aP2    4  55.39  74.34   4.60
    5P3     aP3    5  22.11  68.71   3.85
    6P3     aP3    6  -4.13  24.04   3.73
    7P4     aP4    7  40.16   6.39   4.73
    8P4     aP4    8  -5.40  35.73   4.85
    9P5     aP5    9  36.67  22.45   4.08
   10P5     aP5   10  -3.68 -10.66   4.18
Generated by trjconv : P/L=1/400 t=   1000.000
11214
    1P1     aP1    1  80.48  35.36   4.25
    2P1     aP1    2  37.45   3.92   3.96
    3P2     aP2    3  18.53  -9.69   4.68
    4P2     aP2    4  55.39  74.34   4.60
    5P3     aP3    5  22.11  68.71   3.85
    6P3     aP3    6  -4.13  24.04   3.73
    7P4     aP4    7  40.16   6.39   4.73
    8P4     aP4    8  -5.40  35.73   4.85
    9P5     aP5    9  36.67  22.45   4.08
   10P5     aP5   10  -3.68 -10.66   4.18
Generated by trjconv : P/L=1/400 t=   2000.000
11214
    1P1     aP1    1  80.48  35.36   4.25
    2P1     aP1    2  37.45   3.92   3.96
    3P2     aP2    3  18.53  -9.69   4.68
    4P2     aP2    4  55.39  74.34   4.60
    5P3     aP3    5  22.11  68.71   3.85
    6P3     aP3    6  -4.13  24.04   3.73
    7P4     aP4    7  40.16   6.39   4.73
    8P4     aP4    8  -5.40  35.73   4.85
    9P5     aP5    9  36.67  22.45   4.08
   10P5     aP5   10  -3.68 -10.66   4.18
Generated by trjconv : P/L=1/400 t=   3000.000
11214
    1P1     aP1    1  80.48  35.36   4.25
    2P1     aP1    2  37.45   3.92   3.96
    3P2     aP2    3  18.53  -9.69   4.68
    4P2     aP2    4  55.39  74.34   4.60
    5P3     aP3    5  22.11  68.71   3.85
    6P3     aP3    6  -4.13  24.04   3.73
    7P4     aP4    7  40.16   6.39   4.73
    8P4     aP4    8  -5.40  35.73   4.85
    9P5     aP5    9  36.67  22.45   4.08
   10P5     aP5   10  -3.68 -10.66   4.18

It consists of different frames with updated time. What I showed here is just a sample. The whole file is around 50GB. therefore it will be better to read it line by line or in chunks. But I could not figure out how to deal with the headers of each frame. Are there any ways to get rid of these headers? For now I used following method:

import numpy as np

#define a np.dtype for gro array/dataset (hard-coded for now)
gro_dt = np.dtype([('col1', 'S4'), ('col2', 'S4'), ('col3', int), 
                   ('col4', float), ('col5', float), ('col6', float)])

file = np.genfromtxt('sample.gro', skip_header = 2, dtype=gro_dt)

But it throws the following error when it comes to next header.

ValueError: Some errors were detected !
    Line #13 (got 7 columns instead of 6)
    Line #14 (got 1 columns instead of 6)
    Line #25 (got 7 columns instead of 6)
    Line #26 (got 1 columns instead of 6)
    Line #37 (got 7 columns instead of 6)
    Line #38 (got 1 columns instead of 6)

Answer

Write an adaptor that strips the periodic headers.

def adapt(f):
    for line in f:
        if line.startswith("Generated"):
            print(line, end='')
            # Consume the following line as well.
            # If your data is well behaved, you can 
            # assume the following line exists and should be
            # skipped, instead of using the try statement.
            try:
                print(next(f), end='')
            except StopIteration:
                pass
            continue
        yield line

with open('sample.gro') as f:
    file = np.genfromtxt(adapt(f), dtype=gro_dt)