HDFSiterator¶
Usage
use HDFSiterator;
Iterators for distributed iteration over Hadoop Distributed Filesystem
Warning
This module is currently not working. See #12627 to track progress.
Iterators that can iterate over distributed data in an HDFS filesystem
in a distributed manner. See HDFS
.
-
iter
HDFSiter
(path: string, type rec, regex: string)¶ Iterate through an HDFS file (available in the default configured HDFS server) and yield records matching a regular expression.
Serial and leader-follower versions of this iterator are available.
Arguments: - path -- the path to the file within the HDFS server
- rec -- the type of the records to return
- regexp -- a regular expression with the same number of captures as
the number of fields in the record type
rec