What you think about virtual files?

Post comments about Shareaza code and discuss with other developers.
Forum rules
Home | Wiki | Rules

What you think about virtual files?

Postby ivan386 » 17 Nov 2009 18:49

I think this a great idea but not right realized.

Shareaza must share 2 files (1 real, 2 virtual (part of real)).

First file avalible by name and by hash.
Second (virtual) only by hash.

Real file download have new header.

X-Virtual: [URN's] [Range's in real file]
Last edited by ivan386 on 06 Dec 2009 19:57, edited 1 time in total.
data:application/exe,%B4%09%BA%0D%01%CD%21%B4%08%CD%21%CD%20Hello,World!$
ivan386
 
Posts: 189
Joined: 17 Jun 2009 14:08

Re: What you think about virtual files?

Postby old_death » 17 Nov 2009 19:16

I don't understand what positive effect this could bring... (and I don't understand the idea, BTW ;) )
User avatar
old_death
 
Posts: 1630
Joined: 13 Jun 2009 16:19

Re: What you think about virtual files?

Postby ivan386 » 17 Nov 2009 19:32

Virtual file is a part of file.

In MP3 file is a MP3 Data without ID3 tag. If you change Tag file hash will chenge. Hash of virtual file is stay same. Tag is not part of Virtual file.
data:application/exe,%B4%09%BA%0D%01%CD%21%B4%08%CD%21%CD%20Hello,World!$
ivan386
 
Posts: 189
Joined: 17 Jun 2009 14:08

Re: What you think about virtual files?

Postby ocexyz » 17 Nov 2009 20:10

ivan386 wrote:Virtual file is a part of file.

In MP3 file is a MP3 Data without ID3 tag. If you change Tag file hash will chenge. Hash of virtual file is stay same. Tag is not part of Virtual file.

But why? Or what for? I also don't get. But this is common with genuiue ideas: common ppl don't understant them. That is why they are so extraordinary... :mrgreen:
User avatar
ocexyz
 
Posts: 624
Joined: 15 Jun 2009 13:09

Re: What you think about virtual files?

Postby ivan386 » 17 Nov 2009 20:31

It alow get many sources for virtual file part.
data:application/exe,%B4%09%BA%0D%01%CD%21%B4%08%CD%21%CD%20Hello,World!$
ivan386
 
Posts: 189
Joined: 17 Jun 2009 14:08

Re: What you think about virtual files?

Postby ailurophobe » 17 Nov 2009 22:11

Right. So add file metadata that gives the hash and offset for the metadata stripped version of the file so you can use sources with versions of the same file with identical data but different metadata. Would mainly be useful for mp3. I think Shareaza already supports sharing both the full and the stripped version of mp3 files. No idea if it can share hash and offset data about alternate versions with same data. (With 'offset' I mean that it would be necessary to share data about the sizes of headers before Shareaza could use data from versions with different headers.)
ailurophobe
 
Posts: 702
Joined: 11 Nov 2009 05:25

Re: What you think about virtual files?

Postby kevogod » 18 Nov 2009 01:46

ailurophobe wrote:I think Shareaza already supports sharing both the full and the stripped version of mp3 files.

No, Shareaza only supports sharing one hash. The stripped version of the MP3 has its hash shared and the MP3 is uploaded to the downloader without any ID3 tags. Shareaza manages to maintain some sort of ID3 tag data by adding the audio metadata as ID3 tags after completion. However, this option was only turned on for one version.

Gnucleus shares tagless MP3s by default.
Image
kevogod
 
Posts: 267
Joined: 13 Jun 2009 16:13

Re: What you think about virtual files?

Postby ailurophobe » 18 Nov 2009 02:38

Right. All I remembered was that Shareaza can strip the tags and share the stripped version and that there was talk about also sharing the tagged data. I never was that interested in sharing music, so my recollections of the discussions are vague.
ailurophobe
 
Posts: 702
Joined: 11 Nov 2009 05:25

Re: What you think about virtual files?

Postby punkmaister » 18 Nov 2009 07:21

*removed by moderator*
Stop hijacking topics without adding something useful to the discussion.

mfg,
Old
punkmaister
 
Posts: 104
Joined: 10 Aug 2009 04:09

Re: What you think about virtual files?

Postby old_death » 18 Nov 2009 11:17

Well, if that's your idea, ivan386, it's a cool one. Shareaza already supports sharing either only the stripped file, or only the complete file. Therefore I think adding your virtual files wold be useful (but only with sha1, not the other hashes, as this would (at least at the beginning) be only for the G2 network).
User avatar
old_death
 
Posts: 1630
Joined: 13 Jun 2009 16:19

Re: What you think about virtual files?

Postby ailurophobe » 18 Nov 2009 18:32

The same logic might be usable for sharing other alternate hashes. Mainly I am thinking that after BT DHT support is working it should be possible to search G2 with the btih to get files with sha1 and ed2k hashes (I think this already happens, just rarely, since Shareaza is pretty rare seed in BT) and then get alternate btih hashes (for torrents that contain the same file) from the sources found using sha1 hash. Also storing alternate hashes in metadata would allow sharing files by btih over G2 while not seeding over BT. This doesn't seem to be happening currently since the btih is only linked to the torrent file. Again, not familiar with 2.5 changes, many of which were in BT handling.
ailurophobe
 
Posts: 702
Joined: 11 Nov 2009 05:25

Re: What you think about virtual files?

Postby old_death » 02 Dec 2009 12:52

See my comments on this over here: viewtopic.php?f=7&t=358

Virtual files can be used a) to share the metadata-stripped mp3 file hash & b) share the BITH batch torrent hash of torrents that contain one or several files found in the library.
User avatar
old_death
 
Posts: 1630
Joined: 13 Jun 2009 16:19

Re: What you think about virtual files?

Postby mojo85 » 02 Dec 2009 21:12

I don't think this will add any benefit unless downloading from a G1 client. It would add a new layer of complication for too little a gain. It only works as a limited scope "mp3" files, and those files are small enough that 12 original sources are enough as opposed to 100. It also defeats the reason for implementing a tagless hashing for mp3's ... the real metadata was constantly changing and fragmenting the original sources ... it may have started as 100 sources but as time progresses you get 75 under original hash1 25 under new hash2. A couple of months later this changes again ... and more peers are able to share but some under totally new hashes like hash(3,4,5,...n). This was the reason ... now we go back to constantly changing hashes.

The bittorrent idea sounds neat ... but I got an alternative. Instead of re-inventing the wheel (a virtual file to me sounds like a container file). Could we not utilize collections for this sake? It is our equivalent to a Torrent but it is much more powerful and useful. We could add BTIH urn support, and Tracker support. Each file would have it's own sha1/ eD2k/ and TTH hashes ... but on top of that the whole set of files in the collection could also have batch download support (sha1/ TTH/ and BTIH with Tracker), and sha1/ eD2k/ tth hashes for each file within it could be passed to the Downloader as a virtual file. We can modify TorrentWizard to spit out Collections ... this would move us in the right step and also create a semi standard for how future Torrents should behave with respect to batch file. At this point we can't wait for BT to write the rules for us, we have to set an example and they can follow if need be. But it would make Shareaza a better BT client.

If the above is implemented on the collections side of things, we have to suport it on the download side of things. I think the best approach is to use the virtual idea at this point. A batch download would appear as a Folder a drop down menu containing the files within the folder, and there individual progress. Another drop down menu under each file would show the peers we are downloading from. Now each file is technically a virtual file because in the partials it appears as one .sd file ... but because we are treating it as a virtual file ... it contains the information necessary to download from G2, G1, and eD2k (hashes, file offsets, file size, file name, etc...) ... bittorrent would only require the tracker and btih from the collection to download.

Now for progress bars, we can color them in individually for each downloaded part because we know it's offsets and so when downloading we can color based on those ranges with respect to the offsets ... but the parent batch download folder can be colored in based on overall downloaded percentage instead of downloaded pieces.

On uploading ... the BT upload runs as usual but now we can support partial file support for each child file under a batch download folder.
mojo85
 
Posts: 115
Joined: 27 Sep 2009 05:35

Re: What you think about virtual files?

Postby ocexyz » 02 Dec 2009 21:53

Good idea IMHO.
User avatar
ocexyz
 
Posts: 624
Joined: 15 Jun 2009 13:09

Re: What you think about virtual files?

Postby ivan386 » 02 Dec 2009 22:12

If you create torrent in TorrentWizard SHA1 ED2K and MD5 will be included for each file. But Shareaza AFAIK don't use them for multi-file torrent.
data:application/exe,%B4%09%BA%0D%01%CD%21%B4%08%CD%21%CD%20Hello,World!$
ivan386
 
Posts: 189
Joined: 17 Jun 2009 14:08

Re: What you think about virtual files?

Postby old_death » 02 Dec 2009 23:30

ivan386 wrote:If you create torrent in TorrentWizard SHA1 ED2K and MD5 will be included for each file. But Shareaza AFAIK don't use them for multi-file torrent.
Wish is a problem... AFAIK, it is also impossible to merge single files of a batch torrent with files on your HD.
I think, the files contained in a batch torrent should be threated as if they were single file downloads, with the exception that they are grouped together and that there is a hash for the entire group of files. I think this should be the most efficient way of doing it, as Shareaza has been written and optimized for single file downloads...
User avatar
old_death
 
Posts: 1630
Joined: 13 Jun 2009 16:19

Re: What you think about virtual files?

Postby rolandas » 18 Dec 2009 20:30

The major problem for virtual file hashes is their support. The term "virtual" is an abstract conception. It means that one day the virtual portion of file can become false "virtual", because the tagging changes from day to day, and many proprietary players use their own hackish tags, which no one understands. If you take into account that there's no absolute "tagger" nowadays, then you will get why it's hard to achieve this. Shareaza can not recognize all MPC tags, and that means we can not remove them, or all tags which come before it (if is present in footer), or after it (if is present in header).
There can be APE, MPC, various flavours of ID3 tags in any sequence. There are even LAME encoder tags with its version number embedded. The majority of taggers don't even touch that information. In the past I wanted to implement LAME tag detection, I believe the code is in the codebase, but it was not completed. It's a big headacke with the encoder tags... Some of them are comercial ones, and you never know how to detect them.
rolandas
 
Posts: 6
Joined: 13 Jun 2009 14:01

Re: What you think about virtual files?

Postby ivan386 » 23 Dec 2009 18:03

Wrong logic. Only need to find MP3 frames.

I use code that read technical info from MP3 frames (CLibraryBuilderInternals::ReadMP3Frames). It's alredy implemented in shareaza.

"Detect virtual Mp3"
data:application/exe,%B4%09%BA%0D%01%CD%21%B4%08%CD%21%CD%20Hello,World!$
ivan386
 
Posts: 189
Joined: 17 Jun 2009 14:08


Return to Development Discussion

Who is online

Users browsing this forum: No registered users and 1 guest