On Downloading Fic

[quietmoon]

I never used to download fic — honestly, I'd hear about it, and wonder why people went to such great lengths when they could just bookmark it or add it to their favourites or something. Twelve years in fandom later, I have very much learned my lesson. orz

So I was wondering if anyone had any advice on how to go about doing this? On cataloguing and stuff, I was thinking maybe making an excel spreadsheet to help with navigation and general safety-netting for my terrible memory, but I haven't gotten any further than that thought lol. Also, any recommendations on archiving from FFn.net and LJ? I'm browsing myself for any information I can find, but everything I've seen so far is from 2012 and I didn't know if any better methods have cropped up since then. Instapaper seems to be promising? I don't know.

I bought a very cute Ravage USB and I thought I could save it on there (as well as to the cloud). When I went to find it, though, I found it... and the detached tail. Seems my cats got to her before I did. I now have a very cute tailless 32GB Ravage.

Anyway, if you have any helpful links lying around, please do toss them my way. :3 I'd love to hear any stray thoughts on archiving fic in general, too. I haven't put much here because I don't really know what I have to add beyond 'websites crash, fics disappear, i have been burned but it has shown me the light' lol.

Comments

[enemytosleep]

Man I don’t really know, but kalloway has been on the archiving run of doom so they may have more advice on this? I did my transfers back in the day so no idea what has changed since then.

[quietmoon]

Thank you!

[pauraque]

I've actually never downloaded fic, so I'm afraid I have no advice. I guess I don't often re-read fics, in general, so that's probably why it's never occurred to me. You're certainly right that fics have a tendency to disappear, though, so if you do want to re-read them it makes a lot of sense not to rely on that not happening!

[quietmoon]

It's a mix of old favourite stories going completely missing, having a lack of any cohesive organisation on what I read when I want to reference something I only half-remember, and also reading more widely than just on one website. I've started delving into old lj comms and trying to keep that straight in my head on top of tumblr, ao3, and ffn.net is too much for me. ;__; Even with ao3's bookmark tool.

I have such a worry now that more stuff I love will forever disappear. I read a ton of dramione and when Hawthorne and Vine went down, it broke my heart a little. It doesn't help that I love to reread things, too.

[unavee]

I don't download fic to catalogue it, but I do download some to read on my ereader. I use calibre and it's worked pretty well for me. The software keeps track of the metadata, and you can sort and search and add custom columns. AO3 downloads seem to work well with it; the title, author, series, tags, publish date, summary automatically get filled in. I think there are plug-ins for even more utility, but I haven't delved into those.

[quietmoon]

Ooh, thank you! I'll check it out.

[shipperslist]

I bookmark longer fics to download on my Kindle and save the ones I like to reread on individual ”best of ship” folders. And yeah, I mostly save AO3 fics because that’s where I mostly read my stuff and the mobi-file works well. I do have a couple of favorites I’ve compiled as a giant PDF file myself but I feel like that’s a bit of a grey area so I don’t do that much.

Was this in any way helpful? Most likely not. 😬

[quietmoon]

No, it was, thank you! I like your way of setting aside your favourites; I feel like I'll only download those that I can see myself revisiting (which might still be half or so of what I read or something lol) since I don't use an ereader, and maybe organise them similarly :o

a question about mobi files

[oldtoadwoman]

Hi. Jumping in with a question. I recently downloaded a fic from AO3 to read on my phone and I noticed it had a lot of typos (which seemed particularly odd since I know the author had recently mentioned finishing up the final edits). When I looked at the fic on AO3... the typos were gone. I figured the author had just fixed them so I re-downloaded the mobi file and... same typos were back. Have you ever noticed the AO3 mobi file not matching the website? (I'm torn between the theory that the download files aren't updated instantly and it was still a pre-edited version and the theory that my phone randomly introduced glitches into the file.)

Re: a question about mobi files

[shipperslist]

I honestly don't know because I've never really paid attention? I mean, I tend to read the longfics on my Kindle anyway so I wouldn't know about the small irregularities. Sounds interesting.

Re: a question about mobi files

[oldtoadwoman]

I'm going with the theory that my phone glitched the download, but I guess I just need to read more fanfic and see if it does it again.

reading

[oldtoadwoman]

I only download fic if I want to read it on my phone and this is only a recent experiment. (I'm having a lot of problems with my attention span and more than once I've closed a tab with a story unfinished and didn't even remember until later that I hadn't finished reading it and by then it's usually too much hassle to go and find the story again. If I've got the story in an ereader on the phone I can keep the app locked so I don't accidentally close it and forget I was reading something.)

I don't do any kind of archiving other than AO3 bookmarks.

Re: reading

[quietmoon]

A bit unrelated but on your point of problems with attention span — a habit I've found I've slipped into is to skim read fic so that I know vaguely what it's about, I get a fuzzy sense of 'ah, characters I like, very nice' and then leaving it as Marked for Later. It's a struggle now to remember what I've read, what I've skimmed, what I imagined, and what I've marked to read later vs read 'properly' later. /sigh

Is reading downloaded fic on your phone working for you? A fair bit of the fic I read is on my phone, but I just use the AO3 mobile website when I do.

Re: reading

[oldtoadwoman]

If I'm only reading one thing at a time, reading it on my phone works for me (by locking the app so I don't accidentally close it). I prefer it to the website because I can't figure out how to tell the phone to not auto-rotate when I lie down. Whereas the ereader I have set to not auto-rotate so I can curl up on my side and still read.

The worst is when I start to read something I know I've read before but I don't know how it ends and searching for my name in the kudos shows I never left kudos and I am not sure if that's because the story...

A: is an unfinished story that's not marked as unfinished (boo!)

B: has a boring, forgettable ending and I didn't leave kudos because I was vaguely disappointed (meh)

C: is a story I was reading and enjoying, accidentally closed the tab, and never found again until now (yay!)

D: got so bad in later chapters that I quit reading on purpose, possibly because it hit an unlabeled quick of mine (hiss!)

I've definitely experienced all of these, from the "Yay! I found this lost story again!" to the "Oh! Oh, that's why I quit reading this!"

Re: reading

[quietmoon]

Yeah, I'd get p frustrated if my phone auto-rotated every time I shifted to lie on my side. :/

Oof, yes, I get you! I'm not picky with where I give kudos (generally any fic I liked enough to not exit out of/finish will get one from me) and sometimes I have moments where I'll bookmark things with "read halfway" or "have to reread properly!" but then other things I won't, and it's... just. a mess. I need to work out a proper system and stick to it.

The struggles of having a poor memory :')

Re: reading

[oldtoadwoman]

A couple of friends are very much in the camp of "If you read to the end, you must leave a comment," and... that kind of just taught me to bail on a story really close to the end. Like if the plot feels as resolved as it's likely to get and all that's left is the prologue or a sex scene, my brain will go, "If I stop reading now, I don't have to leave a comment." (Trying to leave an insincere positive comment on something that was "meh" stresses me out because I then project that insincerity onto every positive comment I've ever read. I've really grown to hate the if-you-don't-comment-you're-a-bad-fan guilt, mainly from the perspective of an insecure writer who is sure that everyone secretly hates my writing.)

I wish there were a way to just mark a story finished or not without having to bookmark it. I suppose I could just bookmark everything I read, but... I'm lazy and it feels too late to start a new system. I think if I were to start a brand-new system, I could bookmark stories and tag them as "started reading 5/09/2020" or "bookmarking to read later" and update them to "finished reading 5/13/2020" but... knowing me, I'd still mess up the system somehow.

Re: reading

[quietmoon]

Oh, dear. I think there's a middle ground to be found that respects the readers right to their own response, but also the writer who would certainly love feedback. I used to try and comment on everything I read — it just resulted in me not reading as much lol. When I finally decided that it was okay to leave kudos, that was enough, it made me feel way better, and I ended up feeling much more okay with leaving shorter comments too. Everyone has their own comfort zone, and I'm sorry you've been pushed out of yours. :( I hope you find it easier someday to put the pressure out of your mind!

hhhh of course I understand why a writer might do that (why are we writers so sensitive, ACK) but it's definitely not true! And I think you make a very good point; commenting should be encouraged and welcomed, but not forced or pressured into. Part of the magic of comments is that they did it because they wanted to.

Organising fic reading is frustrating bc u know you've missed so many before starting. *sigh*

[octahedrite]

I wrote a program to download my bookmarked fics from AO3 a year ago, but it's against AO3's TOS -- might result in an account/IP ban if you try using it to download >100 works. How comfortable are you with Python? It could be repurposed for your purposes, e.g. downloading fic from a list of links.

[quietmoon]

I know,, that Python is a coding language. That is the extent of my familiarity, I'm afraid. TwT

That's very cool though! I'm nervous risking a ban aklsdhfj - how does it work? I don't know if I could make it work having no experience or knowledge of Python...

[octahedrite]

Normally you would log into your AO3 account, go to your bookmarks page, click every work there, and then download the work in your preferred format, right? My program basically does all that clicking for you. It's a bot.

According to AO3's terms of service,

>Do you have a policy on bots or scraping? These are ways of extracting information from or indexing websites.

>Using bots or scraping is not against our Terms of Service unless it relates to our guidelines against spam or other activities. However, we do reserve the right to implement robots.txt or other protocols limiting what bots can do, or to notify you and ask you to discontinue if a bot or scraping program is causing problems for the site.

TL;DR: basically they want us to respect AO3's robots.txt, which bans bots from accessing the work and download pages.

My program ignores the robots.txt and accesses those pages anyway. It's fine in limited doses in my experience -- say, downloading 20 works at a time/per day -- I haven't tried anything beyond that.

It's been a while since I myself used it, so I'll check if anything's broken and upload it to GitLab with a big fat disclaimer over the weekend. I'll add some minimal rate-limiting just to be safe (like a 5-second delay between accessing/downloading works). (The program is complete btw, I've just been waffling about whether to upload it or not.)

What OS are you using? At minimum you would need Python 3 and pip to be installed to run this.

[quietmoon]

:o very convenient though omg! Thank you so much, I think I'm gonna try figure out how to use it huhu. I saw you uploaded it already (congrats~).

I use Windows 10. Over the coming weekend I will see if I can figure out what these things are /sweatdrop/ I had to google to double check if windows 10 was an OS, I'm,,, truly clueless dude. Is it the kind of thing where you download it and it runs for itself? or would I have to... asjlkdhf Do Things (i'm so sorry)

(This is wholly unrelated but I went to call you 'Octa' and realised I don't know what to call you even after all this time lol. Should I stick with 'Octahedrite'? ^_^)

[octahedrite]

octa's fine :)

You have to do things, I'm afraid -- read the 'Getting Started' section https://gitlab.com/antarcticite/ao3-bulk-downloader/#getting-started and let me know if I can clear anything up for you. Follow 'Install Python' from this article https://docs.microsoft.com/en-us/windows/python/beginners#install-python and then the rest of the steps from my README. 'Navigate to the folder...' means you have to unzip the code somewhere, copy that folder's path, and type 'cd YOURPATH' into the shell. Step 3 is 'python -m pip install -r requirements.txt', and 4 is 'python ao3-bulk-dl.py [your options]' on Windows 10, unless I'm mistaken (don't have a Windows machine to check rn). Let me know how it goes :)

[silveradept]

I haven't collected fic as a rule, but I suggest that if you want to organize and download them, that you think about how you want to arrange them, and then how you want to be able to search them, such that you'll probably, if your collection gets about so big, have a folder collection where they all are and a relevant database to search through them. Which might mean Calibre or other such program with those function of organizing and metadata built in.

[quietmoon]

This is great advice, thank you. Since I'll be archiving backwards for a while (so many years of reading to sort through) I want to make sure there's an organised system from the get go.

Calibre definitely merits a look. :)

[silveradept]

Having an organized system before helps a lot. Occasionally, you still end up in a situation where you reorganize, but hopefully it's not too terrible if you've set things up well beforehand.

[quietmoon]

I'm not looking forward to the inevitable hours spent organising everything, but as you say, it's worth it. ^_^

[prettyarbitrary]

I learned this the hard way too. I use Calibre to manage and catalogue my downloaded fics. You may still need/want to do some hand-editing of metadata and whatnot, but it certainly beats doing it all manually in a spreadsheet!

As for saving fic from LJ (or Dreamwidth or other LJ clones) I find that the code cooperates well with simply saving the page as an HTML or PDF document. FF.net is grumpier. My best results there have come from cut-and-paste, but it's been quite a while since I spent enough time over there to bother trying.

[quietmoon]

Aghh yeah, Calibre def seems the way to go. I was initially thinking of doing it all manually, and I'm sure I'd start off strong, but that isn't something i can see myself maintaining in the years to come with the accuracy I would wish for xD

FFn.net have made it so that you can't copy and paste anything from their site anymore (at least, I can't figure out how to). But I didn't think of saving as a PDf, I think that might be my best bet for there, thank you!

[prettyarbitrary]

I'd forgotten ff.net did that. Man, they're annoying. I hope saving as pdf works for them!