Front page | perl.perl6.language |
Postings from June 2010
Re: The obligation of free stuff: Google Storage
Thread Previous
|
Thread Next
From:
Richard Hainsworth
Date:
June 10, 2010 00:15
Subject:
Re: The obligation of free stuff: Google Storage
Message ID:
4C109110.1070508@rusrating.ru
Ideally [at least, what I would like], managing a file on a remote
resource should be the same as managing one locally, eg.
my Amazon $fn = open("$path-to-input-file-location/$file-name", :r) or
die $!;
for $fn.readlines { };
$fn.close;
my Google $fn = open("$path-to-output-file-location/$file-name", :w) or
die $!;
for @lots-of-data -> $item { $fn.say: $item };
$fn.close;
If it is possible to make file on a cloud system as easy to manipulate
as a local file, that will aid the acceptibility of perl6.
Hence, the use of 'open', 'close', 'say', 'readlines', rather than
'connect', 'put', 'get'.
The issue - it seems to me - is that the connection between a computer
and its local filesystem has such a low probability of failure (viz.,
disk failure or disk full) that is can normally be ignored, but there is
relatively high probability of a connection failure between the computer
and a remote site and so it would be normal to have exception handling
in the code.
That means the designer of the API should indicate how these exceptions
should be handled so that a program doesnt hang in a loop that is
expecting data from a remote resource whose server has disconnected.
On 06/10/2010 03:24 AM, Aaron Sherman wrote:
> On Wed, Jun 9, 2010 at 10:04 AM, Aaron Sherman<ajs@ajs.com> wrote:
>
>> Has anyone begun to consider what kind of filesystem interface we want
>> for things like sftp, Amazon S3, Google Storage and other remote
>> storage possibilities? Is there any extant work out there, or should I
>> just start spit-balling?
>>
> In the absence of anything forthcoming and a totally arbitrary sense
> of urgency ;-) here's what I think I should do:
>
> IO::FileSystems (S32) gives us some basics and the Path role also
> provides some useful features.
>
> I will start there and build an IO::FileSystems::VFS roughly like:
>
> class IO::VFS is IO::FileSystems {
> ...
> # Session data if applicable
> has IO::VFS::Session $.session;
>
> # Many methods take a $context which, if supplied
> # will contain back-end specific data such as restart markers
> # or payment model information. I'll probably define
> # a role for the context parameter, but otherwise
> # leave it pretty loose as a back-end specific structure.
>
> # A simple operation that guarantees a round-trip to the filesystem
> method nop($context?) { ... }
>
> # list of sub-IO::VFS partitions/buckets/etc
> method targets($context?) { ... }
> method find_target($locator, $context?) { ... }
>
> # Means of acquiring file-level access through a VFS
> method find($locator, $enc = $.session.encoding, $context?) { ... }
> method glob(Str $matcher, $enc = $.session.encoding, $context?) { ... }
>
> # Like opening and writing to filehandle, but the operation is totally
> # opaque and might be a single call, senfile or anything else.
> # Note that this doesn't replace $obj.find($path).write(...)
> method put($locator, $data, $enc = $.session.encoding, $context?) { ... }
>
> # Atomic copy/rename, etc. are logically filesystem operations, even though
> # they might have counterparts at the file level. The distinction being that
> # at the filesystem level I never know nor care what the contents of the
> # file are, I just ask for an operation to be performed on a given path.
> method copy($from, $to, $enc = $.session.encoding, $context?) { ... }
> method rename($from, $to, $enc = $.session.encoding, $context?) { ... }
> method delete($locator, $enc = $.session.encoding, $context?) { ... }
>
> # service-level ACLs if any
> method acl($locator, $context?) { ... }
> }
>
> The general model I imagine would be something like:
>
> my IO::VFS::S3 $s3 .= new();
> $s3.session.connect($amazonlogininfo);
> my $bucket = $s3.find_target($bucket_name);
> $bucket.put("quote.txt", "Now is the time for all good men...\n");
> say "URI: ", $bucket.find("quote.txt").uri;
>
> or
>
> my IO::VFS::GoogleStorage $goog .= new();
> $goog.session.connect($googlelogininfo);
> my $bucket = $goog.find_target($bucket_name);
> $bucket.put("quote.txt", "Now is the time for all good men...\n");
> say "URI: ", $bucket.find("quote.txt").uri;
>
> or
>
> my IO::VFS::SFTP $sftp .= new();
> $sftp.session.connect(:host<storage>, :user<ajs>, :password<iforgotit>);
> my $filesystem = $sftp.find_target("/tmp");
> $filesystem.put("quote.txt", "Now is the time for all good men...\n");
> say "URI: ", $filesystem.find("quote.txt").uri; # using sftp:...
>
> Notice that everything after $obj.session.connect is identical except
> for my choice of variable names. In fact, you shouldn't have to worry
> about what storage back-end you're using as long as you have a valid
> VFS handle. Really path names are the only thing that might trip you
> up.
>
> Thoughts?
>
> I think that in order to do this, I'll need the following support
> libraries which may or may not exist (I'll be looking into these):
>
> IO::FileSystems
> Path
> HTTP (requires socket IO, MIME, base64, etc.)
> Various crypto libs
>
> I don't intend to provide a finished implementation of any of these
> where they don't already exist (I may not even end up with a final
> implementation of the VFS layer), but at least I'll get far enough
> along that others who want to work on this will have a starting point,
> and I'll want to at least have a test that fakes its way all the way
> down to creating a remote file on all three services, even if most of
> the operations involve passing on blobs of data generated by
> equivalent calls in other languages.
>
>
Thread Previous
|
Thread Next