I have a generator producing a list of strings. Is there a utility/adapter in Python that could make it look like a file?
For example,
>>> def str_fn(): ... for c in 'a', 'b', 'c': ... yield c * 3 ... >>> for s in str_fn(): ... print s ... aaa bbb ccc >>> stream = some_magic_adaptor(str_fn()) >>> while True: ... data = stream.read(4) ... if not data: ... break ... print data aaab bbcc c
Because data may be big and needs to be streamable (each fragment is a few kilobytes, the entire stream is tens of megabytes), I do not want to eagerly evaluate the whole generator before passing it to stream adaptor.
Advertisement
Answer
Here’s a solution that should read from your iterator in chunks.
class some_magic_adaptor: def __init__( self, it ): self.it = it self.next_chunk = "" def growChunk( self ): self.next_chunk = self.next_chunk + self.it.next() def read( self, n ): if self.next_chunk == None: return None try: while len(self.next_chunk)<n: self.growChunk() rv = self.next_chunk[:n] self.next_chunk = self.next_chunk[n:] return rv except StopIteration: rv = self.next_chunk self.next_chunk = None return rv def str_fn(): for c in 'a', 'b', 'c': yield c * 3 ff = some_magic_adaptor( str_fn() ) while True: data = ff.read(4) if not data: break print data